RAID failure

[joe90bass]

I think this a hardware issue rather than O/S!

 

I've taken our old Evesham Silveredge server (was running 2003 and Exchange 2003 but been offline for 18 months) and put Debian on it, it installed great. I changed my mind and reinstalled with Ubuntu 10.04 and I've had no end of trouble with RAID. The server has onboard RAID (0 or 1) and 2 160GB WD SATA disks. I'm a bit puzzled as the disks are in RAID 1, yet when I install Ubuntu asks which disk to install to, surely it should only see 1 as it's RAID 1? I've gone into the onboard RAID setup and ensured it is RAID 1 and initialised the RAID again, and run a consistency check, all is well!

 

After installing Ubuntu and rebooting, I get immediately get a RAID degraded warning and the system won't boot. I've replaced the 'failed disk' and tried again, same error. I've tried reinstalling to the other disk during the install and get the same error. I haven't yet disabled the RAID and tried installing that way, I'll try that tomorrow (I've brought the server home as this is really bugging me!). I left the Debian disk in school so can't retry that until Monday (had none of the above issues with it though)...

 

The idea is to use this server for Squid and Dansguardian......

[SYNACK]

The issue here is that the server in question uses software emulated RAID, even though the motherboard 'supports' RAID it does so by using special drivers and a very small bit of logic on the motherboard rather than a propper full RAID system that implements everything in the hardware. Many desktop grade boards have this feature.

 

The reason that this does not work as expected is because the appropriate driver is not loaded during the Ubuntu setup routine and so it sees them as two individual disks. I beleive that if the driver does exist it is possible to integrate the driver into the install medium so that it shows up right but this is not likely to be well supported by the 'community'. I know when I was attempting something simmilar on a desktop board all I got was snide remarks about buying windows hardware. The easiest and most supported solution will be to turn off RAID on the motherboard and install using software RAID like LVM (I think). This is a purely software system and will have a very slight performance toll but will serve better in the long run.

[joe90bass]

Thanks, I'll give software RAID a go. I should have put in the post what board it is! It's an Intel SE7320, a bottom end server board

[dhicks]

I beleive that if the driver does exist it is possible to integrate the driver into the install medium so that it shows up right but this is not likely to be well supported by the 'community'. I know when I was attempting something simmilar on a desktop board all I got was snide remarks about buying windows hardware. The easiest and most supported solution will be to turn off RAID on the motherboard and install using software RAID like LVM (I think).

 

The keyword you want to search Google for is "mdadm" (or just look up the man page "man mdadm", tells you pretty much all you need to know), Linux software RAID. LVM is something a bit different: lets you easily chop up a large storage volume into smaller chunks and do snapshots and so on. You can install LVM on top of an mdadm software RAID volume. I think the latest releases of both Debian and Ubuntu will actually boot off a software RAID array - look out for the option when you get to the disk partitioning stage (might need the alternate install CD for Ubuntu).

 

--

David Hicks

[joe90bass]

Thanks both. I've got installed and have selected LVM. Can I setup the software RAID once it's installed? Haven't had a chance to Google this yet, but will do tomorrow. I've also ordered a Ubuntu server book so plenty of reading and playing around next week when it turns up!

[powdarrmonkey]

The keyword you want to search Google for is "mdadm" (or just look up the man page "man mdadm", tells you pretty much all you need to know), Linux software RAID. LVM is something a bit different: lets you easily chop up a large storage volume into smaller chunks and do snapshots and so on. You can install LVM on top of an mdadm software RAID volume.

 

LVM is also for joining small partitions together into one large volume (hint: the acronym is "logical volume management"), a process called extending.

 

If you're going to use software RAID anyway, you can forget mdadm altogether and use LVM mirroring for redundancy (which can be faster, see http://www.joshbryan.com/blog/2008/01/02/lvm2-mirrors-vs-md-raid-1/) or LVM extending for striping:

 

LVM Mirroring - How to mirror a logical volume - Lvm, Mirroring, Physical Volume, Lvm Howto, Volume Group

 

HOWEVER this requires additional disks, because the parity log is not striped into the data, it's separate. The safest, but slower, arrangement is LVM on top of md software raid.

 

Of course, you'll also want to print out and archive 6.3.*Recovering from LVM Mirror Failure

 

Thanks both. I've got installed and have selected LVM. Can I setup the software RAID once it's installed? Haven't had a chance to Google this yet, but will do tomorrow. I've also ordered a Ubuntu server book so plenty of reading and playing around next week when it turns up!

 

Yes: partition the new disk, add the physical volumes to your group and convert the logical volume OR use mdadm to create true software RAID with only your two disks.

Edited July 24, 2010 by powdarrmonkey
make it less misleading

[dhicks]

LVM is also for joining small partitions together into one large volume

 

I'm confused! I've only ever used it for the opposite - taking one large volume (generally a software RAID 5 array) and chopping it up in to a bunch of smaller volumes, generally for use as virtual machine drives. You reckon it might give better performance if I could skip the RAID bit altogether? Can LVM manage RAID 5? If a system goes down due to a failure of the main OS drive, I know I can boot from something like SystemRescueCD, which will automatically recognise (and start) md RAID arrays, which makes them nice and easy to deal with - can I do the same for LVM volumes?

 

--

David Hicks

[powdarrmonkey]

I'm confused! I've only ever used it for the opposite - taking one large volume (generally a software RAID 5 array) and chopping it up in to a bunch of smaller volumes, generally for use as virtual machine drives.

 

Here, this might help visualise it better:

 

http://4.bp.blogspot.com/_rKe_nbWInhE/Swsese1VFFI/AAAAAAAADko/KgF1iW9AFm4/s1600/lvm.jpg

shamelessly cribbed from Linux Document: LVM - Logical Volume Manager - AIX 5.3

 

You reckon it might give better performance if I could skip the RAID bit altogether?

 

Plain LVM mirroring can provide better performance than plain software RAID mirroring, but it is (probably) less resilient, since it acts on volumes not whole disks.

 

Can LVM manage RAID 5?

 

No. LVM - all data on all PVs, extra PV for the logs. RAID-5: all data spread across all disks, parity information treated the same way.

 

If a system goes down due to a failure of the main OS drive, I know I can boot from something like SystemRescueCD, which will automatically recognise (and start) md RAID arrays, which makes them nice and easy to deal with - can I do the same for LVM volumes?

 

That depends on the kernel modules included on the rescue disk - any sane disk should include them, like it should also include md modules, but don't make assumptions.

 

Full LVM guide: LVM HOWTO

[dhicks]

Plain LVM mirroring can provide better performance than plain software RAID mirroring, but it is (probably) less resilient, since it acts on volumes not whole disks.

 

Right: so I get that LVM can join volumes together, but I don't quite understand where that would be useful. I get, and have used, LVM the other way around - splitting up a large RAID volume into chunks for VMs to use as disks - but if the best way to combine multiple physical disks into a single logical volume is with an md RAID array, where does LVM's joining-volumes facility come in useful?

 

--

David Hicks

[SYNACK]

where does LVM's joining-volumes facility come in useful?

 

JBOD (just a bunch of disks) type RAID would be one area allowing you to span over multiple disks if they are all of different sizes. You could probably then chuck some kind of software RAID layer on top of that but I'm not sure why you would want to.

[powdarrmonkey]

Right: so I get that LVM can join volumes together, but I don't quite understand where that would be useful.

 

Without RAID:

- making it easier to move volume groups between machines, because their place in /dev stops mattering

- extending volumes across disks when space is scarce

- starting with small volumes because you don't fancy forecasting growth, and extending later

- mixing disk sizes into single volumes

- splitting, moving, extending and partitioning volumes without taking the machine offline

 

I get, and have used, LVM the other way around - splitting up a large RAID volume into chunks for VMs to use as disks -

 

This doesn't strictly require LVM, you could do that with good old partitioning (unless that's a requirement of your hypervisor).

 

but if the best way to combine multiple physical disks into a single logical volume is with an md RAID array, where does LVM's joining-volumes facility come in useful?

 

With RAID:

- extending volumes you've been conservative with

- combining onboard and add-in RAID cards (there's no point having software RAID on top of two hardware RAID arrays, after all)

 

and so on.

[dhicks]

This doesn't strictly require LVM, you could do that with good old partitioning (unless that's a requirement of your hypervisor).

 

True, although using LVM lets you use snapshotting as a backup method. Right, RAID arrays: if I have an md RAID array and a disk fails, what happens if I don't have an identical disk to replace it with? I have an array of 12 2TB disks that I'm planning to set up as a RAID 5 array. What happens if one of those disks fails and I can't find an exact replacement, maybe I can only get a disk from a different manufacturer that gives a disk size of slightly less than the original? Would I do best to set up a bunch of LVM volumes first, maybe 1TB, then build a RAID array on top of those?

 

--

David Hicks

[joe90bass]

Thanks to the comments above, and my newly arrived "Ubuntu 10.04 LTS server guide" book I now have software RAID 1 running on my Ubuntu (verifired by running mdstat). So there wasn't really a RAID failure, but as Synack said a sneaky software implementation of what looked like Hardware RAID!

Now for squid and dansguardian!