Hate Linux - VLE Dead - Boot Errors/superblock

[Steve21]

So after a lovely weekend trying to kick a linux box back into shape it's still not playing ball after a UPS failure

 

When trying to boot it's coming up with errors about missing directories/superblock errors.

 

Booted into a repair disc to have a look, SDA/SDB are showing as online and can't be accessed with e2fsck (In a soft RAID so assuming this is correct)

 

All MDs are showing as correct and clean, apart from MD127 but no matter what superblock I try to restore backup from none are accepted.

 

The one bit I don't get and whether this is completely me looking in the wrong direction is the original device it's trying to boot from DEV/VG1 doesn't actually seem to exist at all... So not sure what I'm doing here haha

 

 

Thanks,

Steve

[MicrodigitUK]

That looks like one dead Frog Server. Is the system still in support with Frog? Actually I take it isn't seeing as you are trying to fix it yourself.

 

Good luck Steve, sorry I don't have any words of wisdom.

 

At least if you can't restore it and the school wont pay for Frog support, you can put Frog3 into its grave.

[Steve21]

That looks like one dead Frog Server. Is the system still in support with Frog? Actually I take it isn't seeing as you are trying to fix it yourself.

 

Good luck Steve, sorry I don't have any words of wisdom.

 

At least if you can't restore it and the school wont pay for Frog support, you can put Frog3 into its grave.

 

Haha nope it's not, as we're binning it later in the year so didn't see the point in forking out now, but sadfully lady luck didn't like that plan! £750 a day they want! (And that's no guarantees)

 

Just migrating the 200k~ files in the background off of it is gonna be fun :'(

 

Steve

[MicrodigitUK]

Haha nope it's not, as we're binning it later in the year so didn't see the point in forking out now, but sadfully lady luck didn't like that plan! £750 a day they want! (And that's no guarantees)

 

Just migrating the 200k~ files in the background off of it is gonna be fun :'(

 

Steve

 

I have some neat tricks using some of the unpublished Frog3 File API's that might help you with the migration. However that will depend on you having a working Frog server to access the API's.

 

P.S. All of the files stored on Frog3 system are actually blobs in the database and not stored on the actual file system.

 

Have fun. You know where I am if you need me.

[Steve21]

Morning Bump for my hate of Linux Bribery work?

 

Steve

[kmount]

What does 'cat /proc/mdstat' show?

[Steve21]

@kmount as picture Sorry had to run down other side of building.

 

 

Or lets try that the right way up haha

 

 

Thanks,

Steve

[kmount]

Ouch, that's proper angry.

 

Though in context it may not be because for example md122 and md124 are the same size, as is md123 and m126 so it may just be a bit confused.

 

Do you or does anyone have a frog box they can get a cat /proc/mdstat from to see what it *should* look like?

 

I presume pulling a disk doesn't help when booting? (try pulling one and booting, if no joy reseat and pull the other and boot)

[Steve21]

Ouch, that's proper angry.

 

Though in context it may not be because for example md122 and md124 are the same size, as is md123 and m126 so it may just be a bit confused.

 

Do you or does anyone have a frog box they can get a cat /proc/mdstat from to see what it *should* look like?

 

I presume pulling a disk doesn't help when booting? (try pulling one and booting, if no joy reseat and pull the other and boot)

 

Only thing I could guess is being mirrored raid will it show both sides as such, but aye :s Damn linux haha

 

Tried with one out already, but haven't tried other so will do that in a minute.

 

Thanks,

Steve

[cpjitservices]

try fsck.

 

fsck(8): check/repair file system - Linux man page

[Steve21]

try fsck.

 

fsck(8): check/repair file system - Linux man page

 

Tried the basics in Pic 3/4 up top, all showing as clean apart from one with a superblock error. However the backup superblocks showing as invalid/not applicable.

 

Thanks,

Steve

[Sephiroth]

I presume pulling a disk doesn't help when booting? (try pulling one and booting, if no joy reseat and pull the other and boot)

 

Problem with that is that it isn't one disk that has failed. md122 and md123 have lost the segment on sdb and md124 and md126 have lost the segment on sda.

 

What I would try is getting a fresh disk and 'dd' the living bits from the old RAID set onto it and try booting from that.

[dhicks]

I don't think your disks have failed, but your MD RAID metadata has got mangled. Looking at the partitions on your harddrives (given in your first post), you have 4 partitions on each drive that are intended to each be part of a separate RAID 1 device. From the output given by mdstat above, some of those partitions have become disjoined from each other - mdadm thinks it sees more separate RAID devices, when in fact it's seeing the two halves of a RAID 1 device as two separate arrays. My guess is that you're meant to have:

 

sda1 sdb1 RAID 1 mirror

sda2 sdb2 RAID 1 mirror

sda5 sdb5 RAID 1 mirror

sda6 sdb6 RAID 1 mirror

 

For some reason, sda1 / sdb1 and sda2 / sdb2 have come apart. Is there anything in /etc/mdadm.conf? My guess is that the partitions were laid out as above, then mdadm is intended to auto-assemble the RAID devices on boot, with /etc/mdadm.conf not containing any details of the partition layout. You could try adding that data, telling mdadm to stick the relevant partitions back together.

[kmount]

What's in mdadm --detail /dev/md122 and /dev/md124?

[dhicks]

What's in mdadm --detail /dev/md122 and /dev/md124?

 

Good point. Also, what about the contents of /etc/fstab - that should (hopefully) give some details of what filesystems the server was expecting to mount and what they were called, either by device name or UUID.

[dhicks]

what about the contents of /etc/fstab

 

I've just realised, of course, that I'm asking for the contents of a file that's on a mangled file system, so that's not going to help... What is the server booting from - is /dev/sda4 possibly the boot disk? Is it mountable?

[Steve21]

@kmount

 

@dhicks

@Sephiroth As in just merge the two RAIDs into one disk and try that?

 

Thanks,

Steve

[Steve21]

See I still don't get where that VG1 is even from as it doesn't seem to exist as such :s Wonder if it's just that causing the problem, as the main boot seems to be that MD one? Unless my linux understanding is failing me

 

Steve

[dhicks]

See I still don't get where that VG1 is even from

 

You server isn't booting from its main storage, therefore you are booting it from a CD-ROM or similar, therefore device names and so forth are being assigned by the OS on the boot CD. If the contents of that fstab file are anything to go by (and they might well not be, I'm quite sure where they would have been written from), /dev/md0 is your boot partition, /dev/md3 your root file system and /dev/md1 something to do with Squid (I'm guessing Frog uses it for something). I'm guessing md122 is what your rescue CD OS has decided to call what Frog calls /dev/md0 - if you can get /dev/md122 into a usable state your server might boot.

[kmount]

Out of interest ...

 

If you mkdir /blah can you mount /dev/md-anything into /blah and have it complete without crying?