Irazmus Posted March 5, 2007 Posted March 5, 2007 At home I run an Ubuntu (6.06) file, media, web, VMWare server with 4 RAID arrays. This system has been running happily since I built it last September, but this evening while testing an Asterisk VM I noticed that the contents of the largest and most important array (md0) were no longer visible from my workstation (XP Pro). Strange, thought I, it was fine this afternoon. I can see no problems with Samba as my other shares are working fine, so I SSH to the server and ls -la the array's mount point (/srv/videos), which returnstotal 0 [Random expletive], thought I, my files are gone, however a quick df -H confirms the array still has data Filesystem Size Used Avail Use% Mounted on /dev/md0 946G 745G 153G 83% /srv/videos So I check the array >cat /proc/mdstat Personalities : [raid1] [raid5] md3 : active raid1 hda3[0] 27993152 blocks [2/1] [u_] md2 : active raid1 hda1[0] hdb1[1] 48064 blocks [2/2] [uU] md1 : active raid1 hde1[0] sde1[1] 195357632 blocks [2/2] [uU] md0 : active raid5 sda1[0] sdd1[3] sdc1[2] sdb1[1] 937705728 blocks level 5, 64k chunk, algorithm 2 [4/4] [uUUU] unused devices: No problems there (Except the failed disk in md3 I already knew, and didn't care, about). Next comes >mdadm --query /dev/md0 /dev/md0: 894.27GiB raid5 4 devices, 0 spares. Use mdadm --detail for more detail. /dev/md0: No md super block found, not an md component. Followed by >mdadm --detail /dev/md0 /dev/md0: Version : 00.90.03 Creation Time : Sat Sep 23 22:20:25 2006 Raid Level : raid5 Array Size : 937705728 (894.27 GiB 960.21 GB) Device Size : 312568576 (298.09 GiB 320.07 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Mon Mar 5 17:30:37 2007 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K UUID : 93fe44fd:f2cffc76:5ddc9418:d343a912 Events : 0.6029041 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 3 active sync /dev/sdd1 I'm now more than a little concerned so I check the disks >mdadm --query /dev/sda1 /dev/sda1: is not an md array /dev/sda1: device 0 in 4 device active raid5 md0. Use mdadm --examine for more detail. >mdadm --query /dev/sdb1 /dev/sdb1: is not an md array /dev/sdb1: device 1 in 4 device active raid5 md0. Use mdadm --examine for more detail. >mdadm --query /dev/sdc1 /dev/sdc1: is not an md array /dev/sdc1: device 2 in 4 device active raid5 md0. Use mdadm --examine for more detail. >mdadm --query /dev/sdd1 /dev/sdd1: is not an md array /dev/sdd1: No md super block found, not an md component. >mdadm --examine /dev/sdd1 mdadm: No super block found on /dev/sdd1 (Expected magic a92b4efc, got 00000000) I now think I've found the problem, but daren't try anything lest I make things worse. Can anyone help me get my RAID back in working order without losing the video collection I've spent the last 3 years putting together? And before anyone asks, no, I have not tried turning it off and on again, as it's also my web server and I'd like to keep it running if possible, however I will do if rebooting is likely to help.
Geoff Posted March 5, 2007 Posted March 5, 2007 Can you do for me: lsraid -a /dev/md0 and fdisk -l /dev/sdd and cat /etc/raidtab
Irazmus Posted March 5, 2007 Author Posted March 5, 2007 sudo lsraid -a /dev/sdd lsraid: Device "/dev/sdd1" does not have a valid raid superblock lsraid: Device "/dev/sdd1" does not have a valid raid superblock lsraid: Device "/dev/sdd1" does not have a valid raid superblock lsraid: Device "/dev/sdd1" does not have a valid raid superblock [dev 9, 0] /dev/md0 93FE44FD.F2CFFC76.5DDC9418.D343A912 online [dev 8, 1] /dev/sda1 93FE44FD.F2CFFC76.5DDC9418.D343A912 good [dev 8, 17] /dev/sdb1 93FE44FD.F2CFFC76.5DDC9418.D343A912 good [dev 8, 33] /dev/sdc1 93FE44FD.F2CFFC76.5DDC9418.D343A912 good [dev ?, ?] (unknown) 00000000.00000000.00000000.00000000 missing sudo fdisk -l /dev/sdd Disk /dev/sdd: 320.0 GB, 320072933376 bytes 255 heads, 63 sectors/track, 38913 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdd doesn't contain a valid partition table sudo cat /etc/raidtab cat: /etc/raidtab: No such file or directory
CyberNerd Posted March 6, 2007 Posted March 6, 2007 you should be able to see evidence of a failed disk in /var/log/messages or dmesg. To set the disk as faulty and remove it: mdadm --set-faulty /dev/md0 /dev/sdd mdadm --remove /dev/md0 /dev/sdd
Irazmus Posted March 6, 2007 Author Posted March 6, 2007 Cheers guys, I've ordered another drive and will swap them over hopefully tomorrow. Just to check the rebuild procedure mdadm --set-faulty /dev/md0 /dev/sdd mdadm --remove /dev/md0 /dev/sdd Shutdown and replace sdd sfdisk -d /dev/sda | sfdisk /dev/sdd mdadm --add /dev/md0 /dev/sdd1 Out of interest, is it normal for a Raid 5 array to become inaccessible when a drive fails? I was under the impression they would carry on regardless, albeit slower than normal, or is that only realistic with hardware Riad?
CyberNerd Posted March 6, 2007 Posted March 6, 2007 you might want to double check against this: http://www200.pair.com/mecham/raid/raid1-page3.html it says something about zero'ing the superblock before adding teh drive to teh array. I don't think its normal for the array to become inaccessible. I had a failed raid1 on ubuntu once and it carried on working... as does your md3
CyberNerd Posted March 6, 2007 Posted March 6, 2007 ... what happens if you try and remount the array?
Irazmus Posted March 7, 2007 Author Posted March 7, 2007 Trying to remount gives mount: wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or other error In some cases useful info is found in syslog - try dmesg | tail or so And dmesg | tail gives [45235552.470000] EXT3-fs error (device md0): ext3_check_descriptors: Block bitmap for group 6016 not in group (block 200802304)! [45235552.480000] Remounting filesystem read-only [45235552.490000] EXT3-fs: group descriptors corrupted ! -------- So this is better? mdadm --set-faulty /dev/md0 /dev/sdd mdadm --remove /dev/md0 /dev/sdd Shutdown and replace sdd sfdisk -d /dev/sda | sfdisk /dev/sdd mdadm --zero-superblock /dev/sdd1 mdadm --add /dev/md0 /dev/sdd1
Geoff Posted March 7, 2007 Posted March 7, 2007 So what does mdadm --query --detail /dev/md0 result in now you've replaced the drive?
Irazmus Posted March 7, 2007 Author Posted March 7, 2007 I havn't replaced the drive yet, still waiting for the replacement to arrive
Irazmus Posted March 7, 2007 Author Posted March 7, 2007 OK, drive replaced and the array is recovering Thanks for all the help and advice, it was invaluable. I think I owe you both a drink at the next conference ^^ sudo mdadm --query --detail /dev/md0 /dev/md0: Version : 00.90.03 Creation Time : Sat Sep 23 22:20:25 2006 Raid Level : raid5 Array Size : 937705728 (894.27 GiB 960.21 GB) Device Size : 312568576 (298.09 GiB 320.07 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Wed Mar 7 18:14:54 2007 State : clean, degraded, recovering Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 64K Rebuild Status : 1% complete UUID : 93fe44fd:f2cffc76:5ddc9418:d343a912 Events : 0.6029066 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 0 0 - removed 4 8 49 3 spare rebuilding /dev/sdd1
Geoff Posted March 7, 2007 Posted March 7, 2007 Yeah, no problem. Generally I find that if anything goes screwy with Linux software raid (provided it's been setup and running happy a while) the first thing to do is swap the hardware. Then only if that doesn't fix it to start questioning the configuration.
contink Posted March 8, 2007 Posted March 8, 2007 As an interested bystander I'm still curious as to why a RAID 5 array would fail to work though given that the whole point is that it can suffer a single drive failure and stay working normally..
Geoff Posted March 8, 2007 Posted March 8, 2007 The drive had not entirely failed. It physically worked and the drive electronics responded however the data was gone. In such situations Linux tends to err on the side of caution. Thus requiring a little manual intervenion. Chances are the drive could of been marked manually as failed and the array would of come back on-line. However running a raid5 array with a drive down is a really bad idea and should be avoided at all costs.
contink Posted March 9, 2007 Posted March 9, 2007 The drive had not entirely failed. It physically worked and the drive electronics responded however the data was gone. In such situations Linux tends to err on the side of caution. Thus requiring a little manual intervenion. Chances are the drive could of been marked manually as failed and the array would of come back on-line. However running a raid5 array with a drive down is a really bad idea and should be avoided at all costs. Ah... ok... that makes more sense.. Thanks for the explanation and I'd definitely agree with the last sentiment... waiting on something like this is no option.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now