Planehazza Posted October 4, 2019 Posted October 4, 2019 Thankfully, this is my personal file store at home so data loss won't get me sacked I have a 3 x 4TB RAID 5 Array at home, using some old SAS drives. One of them just failed, which normally would be 'OK' - OK in the sense that it shouldn't fail the array... but it did. . Took it offline, went into the P410 RAID controller and yup, drive 5, the middle one, was 'MISSING'. Ordered a refurb from ebay, fitted it last night. Loads of warnings about the array still being failed and the data lost. Ok, lets not panic, the data is still on the disks, I'll be able to sort it... Put the array back online, boot the server up real quick (OS is on an SSD on a separate controller) and see if D drive has reappeared. It has! Performance is horrendous though, as the RAID controller is churning away, presumably rebuilding the data on the new disk. Once the noise stops, I try to open a folder - great, I see folders! Subfolders now, this is looking good! File, crap - corrupt or missing. I try a single tiny 20KB jpg and that opens. Mind starts racing about stripe size - I'm a RAID rookie, but my understanding is that if a file is smaller than the stripe size, it won't be split across the disks? Please tell me if this is correct or not. I shut it down again, go back to the controller and see a status about a parity initialising in progress. I've left this over night and it's currently on 19%. Please tell me I've done the right thing? First of all, I've not done the good thing (ever) of taking a back up of the array. If this data goes, it's gone forever. All my GoPro footage, WIP premiere videos, my Plex movie library, documents etc. All gone and it's my own fault. What are my chances of this parity thing actually getting my data back together and usable? Or is it more likely that all these writes have scuppered any chance I may have had?
3s-gtech Posted October 4, 2019 Posted October 4, 2019 Chances of another failure while the controller thrashes the disks rebuilding the array are greatly amplified when using RAID 5 (I haven't got the materials to hand now to understand why). That's not to say that it will fail - but you'll have to wait and see until it has finished. RAID 5 can tolerate a single drive failure without data loss, so be patient while it does its thing.
Planehazza Posted October 4, 2019 Author Posted October 4, 2019 Chances of another failure while the controller thrashes the disks rebuilding the array are greatly amplified when using RAID 5 (I haven't got the materials to hand now to understand why). That's not to say that it will fail - but you'll have to wait and see until it has finished. RAID 5 can tolerate a single drive failure without data loss, so be patient while it does its thing. Yeah that's what I thought, which is why I was so confused when the whole thing went down from a single failed disk. It's not life or death that I get the data back, but I'll be pretty gutted if it goes. All my holiday raw photos, GoPro footage of holidays and flying lessons, various documents, old photos/installers, WIP videos. I know I should have backed up, but I'm not sure where I would back up ~3TB of data without costing the earth.
XiJ Posted October 4, 2019 Posted October 4, 2019 Yeah that's what I thought, which is why I was so confused when the whole thing went down from a single failed disk. It's not life or death that I get the data back, but I'll be pretty gutted if it goes. All my holiday raw photos, GoPro footage of holidays and flying lessons, various documents, old photos/installers, WIP videos. I know I should have backed up, but I'm not sure where I would back up ~3TB of data without costing the earth. For photos and videos I use Smugmug. ~£80 a year. I think I can even refer you if you’re interested. Documents etc are usually one drive or google.
Planehazza Posted October 4, 2019 Author Posted October 4, 2019 For photos and videos I use Smugmug. ~£80 a year. I think I can even refer you if you’re interested. Documents etc are usually one drive or google. yeah I have 200GB on Google Drive, so my phone photos and the saved holiday albums are online. It's more the raw files from the camera that I'd meant to go back to edit etc that I'm going to miss.
glen_j Posted October 4, 2019 Posted October 4, 2019 A 8TB raid array will take a long time to rebuild using sata disks. best bet is to leave it to rebuild and see what you have once it settles down. Although the chances of having a second drive fail are higher It should rebuild
PotNoodleTech Posted October 4, 2019 Posted October 4, 2019 Raid 5 is most dangerous when used with a) large disks and b) mechanical disks. You've got very large mechanical disks there. You'll have to just leave it on whilst it's rebuilding parity onto the new drive. it's gonna take a long time with 4TB disks probably a couple to a few days.
Planehazza Posted October 4, 2019 Author Posted October 4, 2019 A 8TB raid array will take a long time to rebuild using sata disks. best bet is to leave it to rebuild and see what you have once it settles down. Although the chances of having a second drive fail are higher It should rebuild They're 4TB Constellation ES.3 SAS drives. Is parity initialisation 'HP Speak' for rebuilding then?
glen_j Posted October 4, 2019 Posted October 4, 2019 I believe it is. Basically using the parity data to recalculate the missing data.
Planehazza Posted October 4, 2019 Author Posted October 4, 2019 I believe it is. Basically using the parity data to recalculate the missing data. I wonder if the P410 is just a very basic controller then. I still don't know why my little HP microserver with the HW RAID controller took the whole array down after a single disk failure...
localzuk Posted October 4, 2019 Posted October 4, 2019 If your disks have an URE of 10^14, then you have a 53% chance of recovery. If it is 10^15, then you have a 94% chance of recovery. 3
Planehazza Posted October 4, 2019 Author Posted October 4, 2019 If your disks have an URE of 10^14, then you have a 53% chance of recovery. If it is 10^15, then you have a 94% chance of recovery. New terminology to me, but I'll check tonight. Not that it matters in the grand scheme; if it's 10^14 there's nothing I can do but hope and keep my fingers crossed. I'd pray but I'm an atheist 1
PotNoodleTech Posted October 4, 2019 Posted October 4, 2019 New terminology to me, but I'll check tonight. Not that it matters in the grand scheme; if it's 10^14 there's nothing I can do but hope and keep my fingers crossed. I'd pray but I'm an atheist You can still pray....just pray to the force ghost of Obi Wan. He'll sort you out
Planehazza Posted October 4, 2019 Author Posted October 4, 2019 You can still pray....just pray to the force ghost of Obi Wan. He'll sort you out "This isn't the data you're looking for". Well, feck... 1
chazzy2501 Posted October 4, 2019 Posted October 4, 2019 I had an older SAN here with raid5 went offline, bad drive.. no biggie. New drive and rebuild all good I thought... 2 days later people said their files wouldn't work. I browse the store and all the files ALL were broken, the directories and fil names were all there but everything was broken, all files corrupt. That was no fun. EDIT: The moral of the story is that EVERYTHING is RAID6 now, I don't care if it slower.
Planehazza Posted October 4, 2019 Author Posted October 4, 2019 I had an older SAN here with raid5 went offline, bad drive.. no biggie. New drive and rebuild all good I thought... 2 days later people said their files wouldn't work. I browse the store and all the files ALL were broken, the directories and fil names were all there but everything was broken, all files corrupt. That was no fun. EDIT: The moral of the story is that EVERYTHING is RAID6 now, I don't care if it slower. Yup, if I could afford £800 for a new bank of 4TB SAS DISKs I'd get rid of RAID5 entirely. One of the sites we look after is RAID 5 still, and they've had a recent failure. Thankfully no loss yet. I'll be begging them to cough up for new disks and downtime soon.
featured_spectre Posted October 4, 2019 Posted October 4, 2019 You can still pray....just pray to the force ghost of Obi Wan. He'll sort you out Indeed
Planehazza Posted October 5, 2019 Author Posted October 5, 2019 No luck. Initialisation hit 100% and failed. Data is still showing folders but all are corrupt and unreadable. I'm determined not to loose this array, but not sure what do next as they're SAS drives. Are there any affordable adapters, even if it's SAS to USB3 etc? I don't want to remove the config of the RAID controller on the server itself and risk exacerbating the issue. I fear it's too far gone now anyway; it's more the old data I've forgotten about that I'm upset about the most. Years and years of stuff
3s-gtech Posted October 5, 2019 Posted October 5, 2019 Worth doing a backup to external, then attempting file recovery from there. That’ll let you know what you’ve still got intact - I suspect with the raw footage file sizes that most of those will be toast. You should be able to get something back from those though, with some bits lost.
Planehazza Posted October 5, 2019 Author Posted October 5, 2019 Worth doing a backup to external, then attempting file recovery from there. That’ll let you know what you’ve still got intact - I suspect with the raw footage file sizes that most of those will be toast. You should be able to get something back from those though, with some bits lost. Any tips on getting SAS drives hooked up to anything other than a live business server? Are there any reliable USB3 adaptors etc?
3s-gtech Posted October 5, 2019 Posted October 5, 2019 No, due to the RAID you’re going to have to use the host for thisdue to compatibility and configure data. As for extra disks to hook up, yeah just use a USB adapter but make it USB 3+ or it’ll be painfully slow.
Planehazza Posted October 7, 2019 Author Posted October 7, 2019 No, due to the RAID you’re going to have to use the host for thisdue to compatibility and configure data. As for extra disks to hook up, yeah just use a USB adapter but make it USB 3+ or it’ll be painfully slow. Does that mean I'll need to clear the config on the controller to allow the 3 drives to appear separately in Windows/other bootable OS to take a backup?
3s-gtech Posted October 7, 2019 Posted October 7, 2019 I just mean a file level backup - pulling off everything that's good or bad using a simple copy operation. Once that's done, then attempt file recovery on the unit. The file recovery operation could damage more stuff while it's trying to rebuild - that's why I'd do a backup first.
Planehazza Posted October 7, 2019 Author Posted October 7, 2019 I just mean a file level backup - pulling off everything that's good or bad using a simple copy operation. Once that's done, then attempt file recovery on the unit. The file recovery operation could damage more stuff while it's trying to rebuild - that's why I'd do a backup first. Ah got you. There's nothing to recover, sadly. Many folders appear, but most of them are corrupt/unreadable when you try to access them. I might bring the drive in to work and take an image of the whole 8TB drive at Windows level. After that, how you recommend I go about getting the disks to show up as separate disks? Take them out and connect them elsewhere?
Planehazza Posted October 8, 2019 Author Posted October 8, 2019 Just thinking; would a cheap ish DELL/HP PCI-E RAID controller work to just have all the separate disks show up on my desktop PC? Get them showing up as separate drives, back them up, then attempt to use software to recover a RAID config and pull out any files I can?
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now