Tefters

Trying to find the root cause though is key to this all. HCI does have a wide audience, maybe not as much as classic server/san but I do see HCI taking over whether that be 2 node setup or more. Is this a freak occurrence, is it a Mellanox firmware update, is this a Windows update, those are questions that need answers. Sans also suffer from similar issues, if a SAN header fails the other is supposed to take over, supposed to being the crucial word here. I've had a storage network completely collapse due to a SAN header failure and the other one didn't register properly to take over. I've also been in a split brain scenario with HA databases, that is not fun! I've seen core switches both for normal network and VOIP horribly fail. I remember an issue 2 years back from cross line VOIP calls which is supposed to be impossible due to encrypted end to end signalling and it took me 11 weeks to convince my SIP provider who had to convince BT, Virgin Media and O2 to all jump on a call while I explain this and everyone tells me no it's impossible we are not seeing this and then an old boy from BT in 60's who consulta for them now joined the call after coming back from holiday in Lanzarote, heard me explain the issue for the third time and he instantly knew what i was talking about and traced it in 6 test calls. It was a faulty module on a Virgin Media core VOIP switch in Northampton and yes calls were crossing over due to this and when calling our school people were slipping into doctors surgery calls and all sorts. Anyways I digress but the main point is sh#t happens regardless and we need to investigate and eliminate to work out what happened and maybe fix or at least document it for future references.

@robyholmes What are your storage NIC's on your servers, make/model? Having just checked my cluster i'm running QLogic FastLinQ QL41262-DE 25GbE Adapters for storage NIC's so wonder if there is a Mellanox issue here in which case Nvidia have possibly F'd up somewhere as I believe they own Mellanox now.

I will add that I do not trust any company when it comes to patching a cluster, Microsoft, Citrix, VMware, I have been burnt by them all in the past so I turn off auto update on cluster hosts as the "aware" in cluster aware updating stand for "we are aware something might F up" and cause a HUGE headache, IT nowadays is "it should" not "it will". I manually patch my hosts quarterly during half terms that way I have adequate time to troubleshoot and if 1 goes wrong I don't hit the update button on the second. If I was a data centre engineer nowadays I would either have a mirrored test environment both HW and SW or have CAU off, patch 1 manually and then re-enable it for the rest and then turn it straight back off after it's done.

What hardware are we talking about here, make/model? You say 2 years old, was this new or 2nd hand? The storage IO errors seen in this post were only root caused down to a Windows patch but didn't pursue exactly which one as it was out of 2. It could have been storage or storage network issues, looked like storage on the surface but this was a 2nd hand setup with no time for further analysis unfortunately, getting the cluster up was the priority. The general consensus is something funny went into the January updates which has caused a bit of a stir with HCI clusters. Have not spotted anything out there on the web specifically stating about this but haven't been hunting either so... Who are the engineers you have been working with, are they or do they have direct lines into MS premium support?

I do all my servers. 1. Backup all server infrastructure to on-site backup server, including itself (excluding drive that holds the backup files itself). 2. Have this mirror off-site via a scale out repository to Wasabi with a 30 day immutable tag for ransomware/hacking protection. Have GFS setup with 7 daily, 4 weekly, 12 monthly, 3 yearly. 3 copies of data, 2 sets of media/locations, 1 location off-site and immutable for 30 days, simples!

City fibre terminated to wherever we wanted and then Exa came and dropped in their router and hooked it up. We then used our router after this and told them to just forward all traffic as we would manage the routing.

Just making sure at this point that all your backups are in a good state and that your nightly backup is still functioning? Also while I have a think, do you not have any vendor support?

Ok so I just watched this to confirm my thoughts on CSV's So what you currently have is a 1 node compute cluster with a CSV storage pool across 2 servers. The state you ideally need to get to is a 1 node cluster running with a storage pool on 1 node so you can then tackle the faulty server/storage and then implement it again once fixed. Hopefully that seems logical. I'm trying to piece together the best way I would go about this myself so give me a few

Reading this https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/remove-servers did you use the cleanup disks flag half way down the page?

I'm also then thinking do you need to remove the disks from the storage pool via PowerShell, something like https://serverfault.com/questions/983824/how-to-remove-a-perfectly-good-disk-s2d-storage-spaces-direct-so-it-can-be-used

So the node is completely removed but it's disks are still in the storage pool. My best guess is it's a cluster issue not a server/disk issue as those disks shouldn't be showing. In cluster aware manager under all the side tabs and tabs at the bottom of each page has the old server references all gone? Does PowerShell if you do a node query on show 1 node in the cluster?

Do the disks still show in the storage pool in cluster manager?

1. Remove Server 2. Check Storage Pool to see if disks removed 3. Remove disks manually via PowerShell if possible That where i'm at in terms of thinking at the moment.

I would strongly recommend AGAINST that idea, you do not want to affect the data on the working disks or the host for that matter! Not unless you are prepared for it all to possibly go Pete Tong and you left with an entire cluster rebuild and DR restore.

One thing I will say on Exa's behalf is that they will go above and beyond to help with the install/provisioning. Their overall service is fantastic and pricing very good, I wouldn't be with them for nearly 5 years if it wasn't! We are a Milton Keynes school that the CF network ran around but not directly past, civil works would of been in the thousands to get the line to our property, 20k to 30k if i remember rightly back in the day. Exa managed to either swallow this temporarily or angle some sort of deal with CF as they got the civil works + install down to £500, yes i said that right. What happened from here is the CF expanding into the whole of Wolverton a year later based on our install and Vodafone struck a partner deal with them to sell FTTP to the public. Wolverton went from a 5mb at best ADSL town paying £30 to £35 a month to a FTTP town paying £25 for initial 2 year deals with speeds of 200mb+ A small story that i'm sure is a little boring to some but brings a smile to my face knowing between ourselves and Exa we pathed the way for the town to get FTTP and our install didn't cost a kidney!

My next chain of thought here and note its only thought as I haven't done this in practice but its the route i would personally go down. If device manager shows the disks as being ok and so does everything else then chances are its just a MS storage pool f*** up ?! Based on this move all VM's onto the working server (if not already) and then remove the non-working server from the cluster and its disks from the storage pool and then try re-adding them. s2d cluster remove server from cluster pool - Search (bing.com)

Exa Networks through CityFibre for 4 nearly 5 years now, good prices but not as good as E2BN by the looks so might look into them.

Ow and cluster validation as well. Worth running that as it might spit out some information that's of use.

Also what are the errors in the event log and is device manager showing any anomalies for the host such as raid controller driver issues?

If you think it's occurred due to a recent update have you tried rolling back the update?

I did this add-in originally on the desktop version when first investigation as I thought that is what the end user was on about and that works but is a little clunky as you say and doesn't offer all the features this integrated version does. This integrated version is supposed to offer extra functionality and backed in natively as demonstrated here: https://www.instagram.com/reel/C1oPzQnqL2q/?igsh=dXBxYzRyMW5iN2pu (apologies for the Insta link, I want to hang myself for just providing a social media link...).

Someone just came to me today regarding PowerPoint an insert Forms feature I didn't know existed, rarely use PP to be honest... Anyways when trying to use it the side window shows either blank or a message saying "cannot get form list from forms site". My machine is domain joined but I have circumvented web filtering and still doesn't work so know its not that. Tried on my home machine remotely and it works fine. Little bit of GoogleFu shows plenty of other people having this issue with domain joined computers but no solution. Anyone got any thought on this or actually got it working? Wont let me upload image but here you are: PPForms.png

*urgh* so Teams V2 is still user appdata based? This is on my to do list but I was hoping they had done away with that so I could fix appdata bloat and pc rotation but nooooo M$ and their idiotic ways... Looks like I will seriously be considering forcing it through a web browser shortcut because the bloat and update method of v1 was crap and people appdata ballooned so much thanks to the install and downloading of updates to appdata that when people roamed machines profile sizes meant having to run delprof2 quite regularly.

Wasabi.

Built a new backup server through Dell last year, £8k for a 35tb HDD raid 5, raid 1 256gb NVMe boot volume, 24GB ram and 2x Xeon silver CPU. Using Veeam, onsite backups go to this and then offload to Wasabi in the cloud where I brought a 5 year 45tb bucket which is immutable for 30 days. Setup GFS and bish bash bosh! I then threw away 2 12 disk Nas boxes, old backup server well past it's expiry date and 2 WD portable Nas that we used rotation each half term for air gap backups and put into a safe. Best purchase of my budget last year all for around £15k for 5 years. As for Office365, I'm currently trying to persuade the school to let me buy into Barracuda but it's a slow negotiation to get people to understand backing up Office365.