Everything is green; everything is gone. Suprise SAN data loss.

[chazzy2501]

The Fujitsu 9 year old SAN (SX60) didn't like being powered down on Tuesday, upon being powered on the drives all came up green THEN amber and the vdisk didn't appear on the server.

 

This is an old SAN and I've allowed people to use it as an area to share large multimedia projects e.g, many iterations of multilayer camera RAWs and lightly compressed video. This was with the understanding that the data was too large to backup (to costly in time and storage) and all the data on this drive was too be considered transient. Final projects were to be stored elsewhere.

 

You can tell by this post that people heard my words but didn't listen, it would seem that the multimedia studies dept had kept important coursework on the drive with no other copies. This is going to be a bad time.

 

 

After an age, I got into the management interface of the san (terminal access was a pain). The SAN told me that 4 of the 5 drives were missing and the vdisk was offline. Well a raid 5 with 5 drives and 4 missing is F'd. As it was looking like a total loss I powered the SAN down, reseated the drives and upon a restart the same issue occured. (I was really hoping a power cycle would get it)

 

I figured that as the vdisk was offline that removing the drives whilst powered up the SAN would'nt object, removing the drives and reinserting them live! Well it seemed to work, as I inserted them they populated the drive list for the vdisk, when I re-inserted the final missing drive the vdisk brought it'self back on line! I quickly ran a verify and (by the next day) it was still all green.. yay! doged a bullet....

 

 

The fiber channel san is connected to a physical server, it wasn't seen or mounted so I rebooted the server. after 10 minutes the server still hadn't come back up, so I had to KVM to it. (something I've not used in a loooong time) it was running a chkdsk on the SAN and it didn't look good.

 

 

Anyhow after an hour the server booted, showed the vdisk and all seemed fine. The directory structure, the files, the ACLs all working... The only problem was every file was corrupt. All of them, every picture has a sliver of an image but fails to open, all videos were corrupted... The SAN is all green, the files all CHKDSK, they're all broken.

 

 

Not sure where to go from here, ideas?

Edited November 1, 2018 by chazzy2501

[rich_tech]

removing the disks like that was probably not the best idea with it switched on, I get you were out of options there, but usually inserting more than 1 disk at a time on a SAN is fraught as they try to rebuild any raids or vdisks, usually automatically as part of their operating procedure especially if they detect fails or offline disks, reinserting them could ave triggered them being wiped clean to re-add into the setup.

 

I would say check and see on the backend if its rebuilding anything raid wise at the moment, it may well have booted up from a basic image and still be checking the rest.

 

If your school is desperate though I would say back off now and dont use the server or run any recovery tools and look to specialists like Kroll Ontrack https://www.ontrack.com/uk/ if its pupil examination work that your after, though their not cheap.

[mavhc]

yeah, in this case you shouldn't have touched them at all, if there's files on there that are required, ask for the £x to buy new drives to clone them onto before experimenting. Oh well.

 

Point out the 100s of times and places you told them not to use it.

 

At least the 2nd time they do the project it will be better

[chazzy2501]

Well, the reason I was being so bullish was that there wasn't supposed to be anything important on it. I'm sure pulling the drives when the Vdisk is offline is allowed. (as no data is being written, or read.) To allow disk migration without taking the whole san offline. I was surprised that it brought the Vdisk back online automatically, AND that it was green, (I even did a verify that took a full day). I was expecting to use a force vdisk command to extract the data in a last ditch effort. but I couldn't use force previously as 4 of the 5 drives were (supposedly) absent.

 

Had I known that their was important data it'd of been backed up. I'd not even look at the san but chuck it.

 

anyhow I'm talking to ontrack and packing up the drives thanks @rich_tech.

 

(next time get a commission:)

[chazzy2501]

The big laugh! ontrack have evaluated the drives...this is the biggest load of crap ever, they sent me an email to tell me that they'd visually checked the drives and were ready to charge me £8000+VAT to do a diagnosis! ha ha ha

 

OK if you didn't get it, they took the drives out onto a table and said yeap, those are hard disks, give me £8000 to see if I can fix em'..

 

btw it's then £10000 to actually do the recovery

 

what a waste of time.