Server crash - possible corrupt RAID? - can anyone help?

[ronnoco]

Hi all,

 

At some point over the weekend our MIS server which is running Server 2003 R2 crashed with a BSOD reporting a dump of physical memory. It did have the BSOD when I last shut it down but on power up last time it came back up fine so I didn't worry about it.

 

This time, it started by getting stuck at the screen showing the RAID arrangement. I could see all the drives looked ok but it would go no further. Powered down, reset all cables, memory, etc and on next boot it got to the Windows Server animated splash screen with the moving bar but after 10 minutes, no further. Tried last known good config and safe mode with no success. Get stuck at loading acpitabl.dat. Researched, could be various things but some say RAID or CMOS battery. Tried changing battery.

 

I take a backup of the entire server every night using NTBackup and do a weekly ASR. Used Recovery Console to try to restore the ASR but it seems to freeze at the 'examining XXXXMB disk space' part. It does the same if you go through the process of trying to re-install Windows or Recovery Console. Ideally I wanted to use RC to run chkdsk.

 

I'm suspecting either a corrupt RAID or possible faulty hard drive(s). I've noticed that only the first LED flashes lots out of the 4 drives (running RAID5 + spare) The rest are mostly solid.

 

I tried to test the memory to eliminate but test would have taken 6 hours so I’ve gone home now leaving it on the 'examining XXXXMB disk space' part.

 

If it's still like that in the morning I would guess I’m going to have to rebuild the RAID and try and recover from backup.

 

Has anyone got any advise/thoughts/suggestions?

 

If I select the option to rebuild the RAID, what does this actually do by default? Does it format the drives or does it try and recover any bad stripes, etc.

 

Thanks very much in advance for any help that can be offered.

Edited May 24, 2010 by ronnoco

[SYNACK]

Have you run a chkdsk on the OS volume, it could well be just a faulty file system. The rebuild option will rebuild the array from parity information if one of the drives has failed, I don't know if it would try anything or what it would do if it did not have a known faulty drive.

 

ACPI is usually power managment for hardware so it could be anything. I think checking the memory could be a good thing after doing a chkdsk to see if that fixes it.

[JJonas]

At the moment we cant get it to a command prompt to run ckhdsk. It either stops at the loading acpitabl.dat (if trying to get to safe mode with command prompt) or 'examining XXXXMB disk space' (if trying to load recovery console from the Server 2003 CD.)

 

Is there a good "Utility CD" that may have tools on it that would help. Something like UBCD or better?

 

The motherboard in this server is an Intel S5000VSA if that gives anyone a clue as to why it is hanging at acpitabl?

[glennda]

Try using a bootable cd such as Bart's pe it will boot to a ram drive and you can then try and run chkdsk on the volumes. If pe fails to load i would assume that it is a ram error as the pe drive is in the ram. I would also try running memtest.

 

Also what make is the server? alot of the major vendors have hardware diagnostic stuff that can be run from a cd.

[JJonas]

Also what make is the server? alot of the major vendors have hardware diagnostic stuff that can be run from a cd.

 

Its a server from Evesham but they dont support it anymore. We bought it just before they went bust.

 

are we going to run chkdsk on a raid array i would have thought we would have to install the Raid driver first.

[glennda]

I

are we going to run chkdsk on a raid array i would have thought we would have to install the Raid driver first.

 

I am not 100% sure if you will need to install the drivers as i have never done it on a raid controlled server due to using the hp stuff. If you keep the drivers on a memory stick you should be able to install them if they are needed.

[JJonas]

We used a copy of UBCD 4 and selected boot to drive C.

We have an error

 

Windows could not start because the following file is missing or corrupt system32\drivers\pci.sys

[danrhodes]

Can you get hold of a copy of pci.sys from your install disk copy it to usb disk, boot up knoppix and copy it from the usb disk to hda0\system32\drivers\

[tmcd35]

Have you tried findin pci.sys in the /i386 folder on the 2003R2 boot CD and restoring it to System32/drivers?

 

"Pci.sys Is Missing or Corrupt" Error Message When You Start Windows

 

Also, when you try booting in Safmode and it stops, whatever line it stops at is usually the last thing that was successfully loaded. So if it says acapitabl.dat, then it is whatever it is trying to load next that it is having problems with. If you can work out what that is...

 

Also, you could try a repair install of Windows. May knock out a driver or two but will keep all data, services and programs in tact.

[JJonas]

Can you get hold of a copy of pci.sys from your install disk copy it to usb disk, boot up knoppix and copy it from the usb disk to hda0\system32\drivers\

 

Thanks for the info. We have a copy of pci.sys and an older copy of knoppix(now downloading latest version) but we dont know how we can see our Raid array through knoppix (or any bootable cd) do you have to load the drivers? if so how do you do that under knoppix/linux?

[danrhodes]

Thanks for the info. We have a copy of pci.sys and an older copy of knoppix(now downloading latest version) but we dont know how we can see our Raid array through knoppix (or any bootable cd) do you have to load the drivers? if so how do you do that under knoppix/linux?

 

Try loading the latest version of knoppix, it may pickup the RAID if its configured from your BIOS at H/W Level?

 

D

[azrael78]

Just to be clear - you booted from the UBCD4Windows and chose to boot from C:?

 

Surely the correct option would have been to simply boot from the UBCD4Win CD.

 

However, it does indeed sound like a corrupt OS volume.

Before trying to do chkdsk, I suggest you get into the RAID controller software and check it out, perform any available non-destructive diagnostic tests on the disks (if any are available).

 

If the RAID volume is corrupt - this should point it out plainly, if it is okay, then your next step is to go for chkdsk.

Another way to possibly get to a command prompt is to boot from a Server 2008/2008R2 DVD and select the 'Repair' option when setup loads - you may or may not require the RAID drivers, as it may be included in Server 2008/2008R2 already but if not, you put the drivers on a USB stick and it should happily load them.

 

HTH,

 

Az

[danrhodes]

Just to be clear - you booted from the UBCD4Windows and chose to boot from C:?

 

Surely the correct option would have been to simply boot from the UBCD4Win CD.

 

However, it does indeed sound like a corrupt OS volume.

Before trying to do chkdsk, I suggest you get into the RAID controller software and check it out, perform any available non-destructive diagnostic tests on the disks (if any are available).

 

If the RAID volume is corrupt - this should point it out plainly, if it is okay, then your next step is to go for chkdsk.

Another way to possibly get to a command prompt is to boot from a Server 2008/2008R2 DVD and select the 'Repair' option when setup loads - you may or may not require the RAID drivers, as it may be included in Server 2008/2008R2 already but if not, you put the drivers on a USB stick and it should happily load them.

 

HTH,

 

Az

 

Good point Azrael, you may have some diagnostics tools inbuild into your RAID Software. Now why didn't I think of that!?

[azrael78]

Good point Azrael, you may have some diagnostics tools inbuild into your RAID Software. Now why didn't I think of that!?

 

It's early in the morning and it's hot?

 

Az

[JJonas]

The only options in the Raid controller are a consistency check that tells me that BGI is in progress (whatever that is)and then exits when you press a key and rebuild array.

 

We have just booted a copy Mini XP from Hirens and loaded the raid drivers into it but it hangs at the windows screen. So we are thinking it is seriously stuffed. We are now wondering if we should go do the rebuid array route?

[danrhodes]

It's early, but it's cold here in Castleford! Infact I would go as far to say that I think its goint to rain shortly!

 

D

[danrhodes]

The only options in the Raid controller are a consistency check that tells me that BGI is in progress (whatever that is)and then exits when you press a key and rebuild array.

 

We have just booted a copy Mini XP from Hirens and loaded the raid drivers into it but it hangs at the windows screen. So we are thinking it is seriously stuffed. We are now wondering if we should go do the rebuid array route?

 

BGI (Background Initialization) is a process to correct parity on the virtual (RAID) disks. BGI is an automated check for media errors in which parity is created and written in the process. It sound's like it knows there is a problem and is trying to recover itself.

 

Not sure wether a successfull recovery would solve your issue though if it completes without issue?

 

D

[ronnoco]

Thanks for all the support. Unfortunately, it was not possible to see the hard drive array from anything - even with the correct drivers loaded it just hung so it does look like a faulty RAID. We have deleted and re-created the array and are now initialising. After this, will re-install the OS then hopefully recover the OS using a backup...after that, just need to restore the rest of the data!

 

I am expecting grey hair by the end of this week!

[danrhodes]

Thanks for all the support. Unfortunately, it was not possible to see the hard drive array from anything - even with the correct drivers loaded it just hung so it does look like a faulty RAID. We have deleted and re-created the array and are now initialising. After this, will re-install the OS then hopefully recover the OS using a backup...after that, just need to restore the rest of the data!

 

I am expecting grey hair by the end of this week!

 

Good luck! Did you check your backups before you took this drastic action?

 

D

[azrael78]

I think you did the right thing, if it was doing BGI and with so many boot disks failing it did seem like the array itself was damaged.

 

This is where you pray to the Backup Gods - to be sure all goes well

 

Az

[JJonas]

Just to update this post its fixed now.

 

We reinstalled on a different server (one purchased from FN-GM off the Edugeek classified ads page) and its now running our MIS.

 

 

On the old server - One of the drives on the RAID array was giving very slow read times More than 8 minutes to do a scan as compared to about 50 seconds for the others. I have removed that drive and now I am reconfiguring as a RAID 5 and it appears to be working.

 

And to top off what started as a bad week this just arrived on our desk

 

So thanks to everyone that helped and thanks to FN-GM for the server.