+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 16
*nix Thread, Help with Ubuntu RAID in Technical; At home I run an Ubuntu (6.06) file, media, web, VMWare server with 4 RAID arrays. This system has been ...
  1. #1
    Irazmus's Avatar
    Join Date
    Feb 2006
    Location
    Suffolk
    Posts
    313
    Thank Post
    13
    Thanked 19 Times in 14 Posts
    Rep Power
    22

    Help with Ubuntu RAID

    At home I run an Ubuntu (6.06) file, media, web, VMWare server with 4 RAID arrays. This system has been running happily since I built it last September, but this evening while testing an Asterisk VM I noticed that the contents of the largest and most important array (md0) were no longer visible from my workstation (XP Pro). Strange, thought I, it was fine this afternoon.
    I can see no problems with Samba as my other shares are working fine, so I SSH to the server and ls -la the array's mount point (/srv/videos), which returns
    Code:
    total 0
    [Random expletive], thought I, my files are gone, however a quick df -H confirms the array still has data
    Code:
    Filesystem             Size   Used  Avail Use% Mounted on
    /dev/md0               946G   745G   153G  83% /srv/videos
    So I check the array
    Code:
    >cat /proc/mdstat
    Personalities : [raid1] [raid5]
    md3 : active raid1 hda3[0]
          27993152 blocks [2/1] [U_]
    
    md2 : active raid1 hda1[0] hdb1[1]
          48064 blocks [2/2] [UU]
    
    md1 : active raid1 hde1[0] sde1[1]
          195357632 blocks [2/2] [UU]
    
    md0 : active raid5 sda1[0] sdd1[3] sdc1[2] sdb1[1]
          937705728 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
    
    unused devices: <none>
    No problems there (Except the failed disk in md3 I already knew, and didn't care, about).
    Next comes
    Code:
    >mdadm --query /dev/md0
    /dev/md0: 894.27GiB raid5 4 devices, 0 spares. Use mdadm --detail for more detail.
    /dev/md0: No md super block found, not an md component.
    Followed by
    Code:
    >mdadm --detail /dev/md0
    /dev/md0:
            Version : 00.90.03
      Creation Time : Sat Sep 23 22:20:25 2006
         Raid Level : raid5
         Array Size : 937705728 (894.27 GiB 960.21 GB)
        Device Size : 312568576 (298.09 GiB 320.07 GB)
       Raid Devices : 4
      Total Devices : 4
    Preferred Minor : 0
        Persistence : Superblock is persistent
    
        Update Time : Mon Mar  5 17:30:37 2007
              State : clean
     Active Devices : 4
    Working Devices : 4
     Failed Devices : 0
      Spare Devices : 0
    
             Layout : left-symmetric
         Chunk Size : 64K
    
               UUID : 93fe44fd:f2cffc76:5ddc9418:d343a912
             Events : 0.6029041
    
        Number   Major   Minor   RaidDevice State
           0       8        1        0      active sync   /dev/sda1
           1       8       17        1      active sync   /dev/sdb1
           2       8       33        2      active sync   /dev/sdc1
           3       8       49        3      active sync   /dev/sdd1
    I'm now more than a little concerned so I check the disks
    Code:
    >mdadm --query /dev/sda1
    /dev/sda1: is not an md array
    /dev/sda1: device 0 in 4 device active raid5 md0.  Use mdadm --examine for more detail.
    >mdadm --query /dev/sdb1
    /dev/sdb1: is not an md array
    /dev/sdb1: device 1 in 4 device active raid5 md0.  Use mdadm --examine for more detail.
    >mdadm --query /dev/sdc1
    /dev/sdc1: is not an md array
    /dev/sdc1: device 2 in 4 device active raid5 md0.  Use mdadm --examine for more detail.
    >mdadm --query /dev/sdd1
    /dev/sdd1: is not an md array
    /dev/sdd1: No md super block found, not an md component.
    >mdadm --examine /dev/sdd1
    mdadm: No super block found on /dev/sdd1 (Expected magic a92b4efc, got 00000000)
    I now think I've found the problem, but daren't try anything lest I make things worse. Can anyone help me get my RAID back in working order without losing the video collection I've spent the last 3 years putting together?

    And before anyone asks, no, I have not tried turning it off and on again, as it's also my web server and I'd like to keep it running if possible, however I will do if rebooting is likely to help.

  2. #2

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,802
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Help with Ubuntu RAID

    Can you do for me:

    Code:
    lsraid -a /dev/md0
    and

    Code:
    fdisk -l /dev/sdd
    and

    Code:
    cat /etc/raidtab

  3. #3
    Irazmus's Avatar
    Join Date
    Feb 2006
    Location
    Suffolk
    Posts
    313
    Thank Post
    13
    Thanked 19 Times in 14 Posts
    Rep Power
    22

    Re: Help with Ubuntu RAID

    sudo lsraid -a /dev/sdd
    Code:
    lsraid: Device "/dev/sdd1" does not have a valid raid superblock
    lsraid: Device "/dev/sdd1" does not have a valid raid superblock
    lsraid: Device "/dev/sdd1" does not have a valid raid superblock
    lsraid: Device "/dev/sdd1" does not have a valid raid superblock
    [dev   9,   0] /dev/md0         93FE44FD.F2CFFC76.5DDC9418.D343A912 online
    [dev   8,   1] /dev/sda1        93FE44FD.F2CFFC76.5DDC9418.D343A912 good
    [dev   8,  17] /dev/sdb1        93FE44FD.F2CFFC76.5DDC9418.D343A912 good
    [dev   8,  33] /dev/sdc1        93FE44FD.F2CFFC76.5DDC9418.D343A912 good
    [dev   ?,   ?] (unknown)        00000000.00000000.00000000.00000000 missing
    sudo fdisk -l /dev/sdd
    Code:
    Disk /dev/sdd: 320.0 GB, 320072933376 bytes
    255 heads, 63 sectors/track, 38913 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    
    Disk /dev/sdd doesn't contain a valid partition table
    sudo cat /etc/raidtab
    Code:
    cat: /etc/raidtab: No such file or directory

  4. #4

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,802
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Help with Ubuntu RAID

    ok, it looks to me like /dev/sdd has failed.

  5. #5


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339

    Re: Help with Ubuntu RAID

    you should be able to see evidence of a failed disk in /var/log/messages or dmesg. To set the disk as faulty and remove it:
    Code:
     
    mdadm --set-faulty /dev/md0 /dev/sdd
    mdadm --remove /dev/md0 /dev/sdd

  6. #6
    Irazmus's Avatar
    Join Date
    Feb 2006
    Location
    Suffolk
    Posts
    313
    Thank Post
    13
    Thanked 19 Times in 14 Posts
    Rep Power
    22

    Re: Help with Ubuntu RAID

    Cheers guys, I've ordered another drive and will swap them over hopefully tomorrow.

    Just to check the rebuild procedure
    Code:
    mdadm --set-faulty /dev/md0 /dev/sdd
    mdadm --remove /dev/md0 /dev/sdd
    Shutdown and replace sdd
    Code:
    sfdisk -d /dev/sda | sfdisk /dev/sdd
    mdadm --add /dev/md0 /dev/sdd1
    Out of interest, is it normal for a Raid 5 array to become inaccessible when a drive fails? I was under the impression they would carry on regardless, albeit slower than normal, or is that only realistic with hardware Riad?

  7. #7


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339

    Re: Help with Ubuntu RAID

    you might want to double check against this:
    http://www200.pair.com/mecham/raid/raid1-page3.html
    it says something about zero'ing the superblock before adding teh drive to teh array.

    I don't think its normal for the array to become inaccessible. I had a failed raid1 on ubuntu once and it carried on working... as does your md3

  8. #8


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339

    Re: Help with Ubuntu RAID

    ... what happens if you try and remount the array?

  9. #9
    Irazmus's Avatar
    Join Date
    Feb 2006
    Location
    Suffolk
    Posts
    313
    Thank Post
    13
    Thanked 19 Times in 14 Posts
    Rep Power
    22

    Re: Help with Ubuntu RAID

    Trying to remount gives
    Code:
    mount: wrong fs type, bad option, bad superblock on /dev/md0,
           missing codepage or other error
           In some cases useful info is found in syslog - try
           dmesg | tail  or so
    And dmesg | tail gives
    Code:
    [45235552.470000] EXT3-fs error (device md0): ext3_check_descriptors: Block bitmap for group 6016 not in group (block 200802304)!
    [45235552.480000] Remounting filesystem read-only
    [45235552.490000] EXT3-fs: group descriptors corrupted !
    --------

    So this is better?
    Code:
    mdadm --set-faulty /dev/md0 /dev/sdd
    mdadm --remove /dev/md0 /dev/sdd
    Shutdown and replace sdd
    Code:
    sfdisk -d /dev/sda | sfdisk /dev/sdd
    mdadm --zero-superblock /dev/sdd1
    mdadm --add /dev/md0 /dev/sdd1

  10. #10

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,802
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Help with Ubuntu RAID

    So what does
    Code:
    mdadm --query --detail /dev/md0
    result in now you've replaced the drive?

  11. #11
    Irazmus's Avatar
    Join Date
    Feb 2006
    Location
    Suffolk
    Posts
    313
    Thank Post
    13
    Thanked 19 Times in 14 Posts
    Rep Power
    22

    Re: Help with Ubuntu RAID

    I havn't replaced the drive yet, still waiting for the replacement to arrive

  12. #12
    Irazmus's Avatar
    Join Date
    Feb 2006
    Location
    Suffolk
    Posts
    313
    Thank Post
    13
    Thanked 19 Times in 14 Posts
    Rep Power
    22

    Re: Help with Ubuntu RAID

    OK, drive replaced and the array is recovering
    Thanks for all the help and advice, it was invaluable. I think I owe you both a drink at the next conference ^^

    Code:
    sudo mdadm --query --detail /dev/md0
    /dev/md0:
            Version : 00.90.03
      Creation Time : Sat Sep 23 22:20:25 2006
         Raid Level : raid5
         Array Size : 937705728 (894.27 GiB 960.21 GB)
        Device Size : 312568576 (298.09 GiB 320.07 GB)
       Raid Devices : 4
      Total Devices : 4
    Preferred Minor : 0
        Persistence : Superblock is persistent
    
        Update Time : Wed Mar  7 18:14:54 2007
              State : clean, degraded, recovering
     Active Devices : 3
    Working Devices : 4
     Failed Devices : 0
      Spare Devices : 1
    
             Layout : left-symmetric
         Chunk Size : 64K
    
     Rebuild Status : 1% complete
    
               UUID : 93fe44fd:f2cffc76:5ddc9418:d343a912
             Events : 0.6029066
    
        Number   Major   Minor   RaidDevice State
           0       8        1        0      active sync   /dev/sda1
           1       8       17        1      active sync   /dev/sdb1
           2       8       33        2      active sync   /dev/sdc1
           3       0        0        -      removed
    
           4       8       49        3      spare rebuilding   /dev/sdd1

  13. #13

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,802
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Help with Ubuntu RAID

    Yeah, no problem. Generally I find that if anything goes screwy with Linux software raid (provided it's been setup and running happy a while) the first thing to do is swap the hardware. Then only if that doesn't fix it to start questioning the configuration.

  14. #14
    contink's Avatar
    Join Date
    Jul 2006
    Location
    South Yorkshire
    Posts
    3,791
    Thank Post
    303
    Thanked 327 Times in 233 Posts
    Rep Power
    118

    Re: Help with Ubuntu RAID

    As an interested bystander I'm still curious as to why a RAID 5 array would fail to work though given that the whole point is that it can suffer a single drive failure and stay working normally..

  15. #15

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,802
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224

    Re: Help with Ubuntu RAID

    The drive had not entirely failed. It physically worked and the drive electronics responded however the data was gone. In such situations Linux tends to err on the side of caution. Thus requiring a little manual intervenion. Chances are the drive could of been marked manually as failed and the array would of come back on-line. However running a raid5 array with a drive down is a really bad idea and should be avoided at all costs.

SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Raid 10
    By Kyle in forum Windows
    Replies: 1
    Last Post: 7th January 2008, 01:06 PM
  2. software raid 5 for ubuntu
    By alonebfg in forum *nix
    Replies: 4
    Last Post: 15th November 2007, 11:55 AM
  3. raid problem
    By mark in forum Windows
    Replies: 4
    Last Post: 3rd September 2007, 02:22 PM
  4. RAID F
    By pete in forum Jokes/Interweb Things
    Replies: 0
    Last Post: 17th January 2006, 02:37 PM
  5. DIY NAS RAID 5
    By Dos_Box in forum Hardware
    Replies: 10
    Last Post: 13th July 2005, 10:16 AM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •