hyper-v cluster issue - CSV read only on one node

[KK20]

environment: 2x 2012R2 nodes (Dell R620's - full 2012R2 std install ), dedicated 10GB heartbeat via isolated switch, 10Gb to rest of network. Storage via HBA SAS to an MD3220 SAN (dual quad port on SAN , 2x HBA lines to each server one from each controller for multipath redundancy). 192Gb RAM about 70% in use if all VMs are on one node. The cluster hosts 8 VMs (not quite enough to justify 2x copies of datacenter yet). I also have a backup synology array connected via iSCSI (this is also included in the CSV)

 

Firstly, the cluster "works", it can failover and VMs migrate happily. I inherited a dell T610 and was going to add this as a third node 'cause "why not". Redundancy is good yes? I setup 2012R2 the same as I did with all my Dell machines. The specs are the same other than CPUs (VMs do have the "different CPU" enabled as the CPUs in my existing R620s are mildy different so I have done this from the very beginning), same RAM, same NIC, same HBA setup. New node joined the cluster and all seemed to be well. The LUNs were added fine from the MD3220, the iSCSI LUN was setup fine from the synology.

 

However. I ahve discovered that unless the new node is the "owner" of the CSV drives, it can "read only" CSVs. So if either of the R620 nodes are "owners" of a CSV the new node cannot write to the volume (read only) - this is for ANY of the CSV's, i.e. the ones mounted from MD3220 LUNs or the one from the iSCSI LUN. There is no corruption, no dataloss, nothing to suggest the CSV is dodgy (the VMs run quite happily on the new node as long as the new node is the OWNER of the CSV the VM is residing on). If the new node is the owner of (any of) the CSVs, the old nodes can read and write to the CSVs, it is just if the old nodes are owners the new node cannot write.

 

Im sure ive had a brain burp and forgotten something, any ideas on what ive forgotten to do?

Edited February 26, 2018 by KK20

[Steve21]

Did you modify the iscsi on the storage to allow the new host write access to the connection?

 

Steve

[KK20]

I didnt change anything on the iSCSI host box, i merely added the new node iSCSI target on the node, added the CHAP credentials the discovered the LUNs, then joined the cluster. All appeared fine (the target node appears in the iSCSI host screen the same as the "old" nodes). The odd thing is, the primary SAN is HBA, the backup is iSCSI and CSVs that are located on either are affected - if the CSV isnt owned by the new node then the new node cannot write to that CSV (regardless if it is a HBA or iSCSI "hosted" CSV). The old nodes dont care who owns the CSV - they can read write at any time.

 

I'm thinking an evict and reinstall from scratch in case ive missed some driver or whatnot; I could understand if I had forgotten (say) a HBA driver but for iSCSI *and* HBA hosted CSVs to be affected had me stumped.

[Steve21]

It depends on the storage normally, some default to an effective "view only" unless modified per host, other's don't

 

Do the ClusterStorage folders all show up ok with the permissions etc when it's not owner?

 

Also depending on what you're using cluster side VMM/FOM etc did you modify things like the Qorum as assignable to the other node?

 

Steve

[KK20]

I didnt change the witness disk quorum settings (the quorum is actually hosted on the MD3220 HBA) and I dont seem to be able to change the owner of the quorum (which makes sense as it isnt a CSV - it is listed as a witness quorum disk, the cluster sorted the disk quorum out when it was created). I havent touched any permissions to any folders on the CSV folders, i'll monitor the raw permissions when i change owners manually. Im just creating a fresh volume and LUN on the MD3220 from some free space. Id rather not mess about with live CSVs so ive taken affinity off the VMs for the new node and will play about with the "test" voume im creating.

 

Looking in MD3220, it sees the new node as a valid host, the host is in the correct storage host group with the LUNs (4 total LUNs plus a quorum LUN), it sees both HBA paths - all looks the same as the other nodes in the cluster. The iSCSI (synology) array is quite simplistic, it shows 3 targets active and the single LUN is set to read/write (it does not seem to have the ability to set read only per target, only for the LUN as a whole).

 

once the new lun/storage has been added to the cluster I will play with that CSV and monitor the permissions depending on owner etc. That might shed some light.

 

edit: the windows updates are the same on each server - I made sure I ran a cluster aware update before I added the new node. The new node was updated before being added too.

Edited February 26, 2018 by KK20

[KK20]

problem solved. When I created the R610 I used the same downloads I had used for the R620 (the ISOs said they supported both hence me not worrying). Rather than look at downloading NEW drivers and having a mismatch I thought I would use IDENTICAL drivers for compatibility. This was the wrong thing to do it seems. I downloaded the latest MDSS for the MD3220 and upgraded the current MDSS on the new node only. this appears to have added different MPIO drivers to the new node system as I can now access CSVs on the new node with *any* owner on the CSV - this also appears to have fixed my non-dell iSCSI CSV too!

 

This was most weird, I can only think that somehow the MPIO driver for the MD3220 was interferring somehow with the node CSVs - I thought the owner of the CSV should be updating metadata only (it wasnt a major role as far as im aware). I am also at a loss how the dell MPIO could affect the microsoft DSM iSCSI MPIO.... I should also point out that it PASSED cluster validation before joining (i forgot to mention that part - and as far as im aware, the cluster validation RUNS a failover on the VMs; the cluster validation could well swap the disk owner to the node it is failing over to - thus masking this particular issue!)

Edited February 26, 2018 by KK20