contink Posted June 22, 2009 Posted June 22, 2009 (edited) I've just run a simple reboot on a server that had a number of windows updates pending on it. Only problem is that after various failed reboot messages (it just jumped to network boot attempts) I've gone into the RAID bios and found that it hasn't got the virtual drive listed at all now.. It's just gone. I've gone over everything I've done so far and nothing would account for this... All I can think of is some update in the winupdate auto queue but even that shouldn't have installed any new drivers... At a total loss. I'm guessing we're talking re-install here.. but anyone got any tips before I go to work on it? Server info: - Win2k3 SP2 - Intel Embedded Server Bios: A.06.02151557I - Four drives all showing as ready... no hardware errors, etc... Edited June 22, 2009 by contink
contink Posted June 22, 2009 Author Posted June 22, 2009 ... and then Edugeek goes offline for 5 minutes... You ****ing know how to make a guys day I can tell ya
SYNACK Posted June 22, 2009 Posted June 22, 2009 Grrrrrh, stupid server lost my post, it took time to type too
matt40k Posted June 22, 2009 Posted June 22, 2009 Have you restored from the previous system restore point? What raid is it? Is it the embedded controller? Is it software? 1
SYNACK Posted June 22, 2009 Posted June 22, 2009 Right lets try again, I'll try to use smaller words so as to not crash the server again... First up what kind of RAID controller and what RAID level? Some controllers store the config at the beginning of each drive in the set and so removing, rebooting and reseating then rebooting again in this case may recover the settings. This can cause issues with lesser controllers when one drive gets snotty, if RAID1+0 you could ty just having two of the drives in or three if RAID5 to attempt to recover the settings and data off the good drives. If the config is stored on the disks but has been overwritten then some controllers offer the option to configure the virtual disk without initializing the disk. This may or may not work if you set it up the same again by rewriting the config information to the start of the drives without touching the data. All of these methods come with no warrenty as they all may waste the data so I hope you also have backups. 1
srochford Posted June 22, 2009 Posted June 22, 2009 I've seen drives disappear completely before and completely disconnecting the power, waiting a bit and then reconnecting has magically made it come back. Too late now, but it's always a good idea to restart the machine before you run updates - that way, if it doesn't restart you know it's not an update problem but a machine problem and you don't waste time looking at the updates to see what went wrong 1
contink Posted June 22, 2009 Author Posted June 22, 2009 Right lets try again, I'll try to use smaller words so as to not crash the server again... So it was your fault! First up what kind of RAID controller and what RAID level? Intel Embedded, RAID 1+0.. Some controllers store the config at the beginning of each drive in the set and so removing, rebooting and reseating then rebooting again in this case may recover the settings. That's probably going to be about all I can hope for. This can cause issues with lesser controllers when one drive gets snotty, if RAID1+0 you could ty just having two of the drives in or three if RAID5 to attempt to recover the settings and data off the good drives. TBH I have no idea if any of the drives are "bad" so's to speak.. It's worth a shot though before the suggestion above. If the config is stored on the disks but has been overwritten then some controllers offer the option to configure the virtual disk without initializing the disk. This may or may not work if you set it up the same again by rewriting the config information to the start of the drives without touching the data. I suspect that's unlikely to be an option but we can give that a shot too... Right now it's saying that there is no virtual disk at all... All of these methods come with no warrenty as they all may waste the data so I hope you also have backups. Luckily I was in the process of moving some of the data to this machine but due to a few changes in the login script it seems I missed that data got reverted back... it seems my subconcious was looking after me. As for backups... They appear to have vaporised... I know I had a complete set but its gone walkabout.. At this point I'm just counting my lucky stars that the SIMs data wasn't on this one... For the life of me though I just can't figure out WTF caused this..
contink Posted June 22, 2009 Author Posted June 22, 2009 I've seen drives disappear completely before and completely disconnecting the power, waiting a bit and then reconnecting has magically made it come back. Well, it's worth a shot as I haven't done any of the other things suggested... Too late now, but it's always a good idea to restart the machine before you run updates - that way, if it doesn't restart you know it's not an update problem but a machine problem and you don't waste time looking at the updates to see what went wrong Unfortunately the whole operating system and data partitions are all on the RAID system so there's no diagnosis to be had.
contink Posted June 22, 2009 Author Posted June 22, 2009 If the config is stored on the disks but has been overwritten then some controllers offer the option to configure the virtual disk without initializing the disk. This may or may not work if you set it up the same again by rewriting the config information to the start of the drives without touching the data. After everything else, gone with this option... fingers crossed... Nope... it's just not happening... Ok... Thanks everyone... appreciate the help but reinstallation is required... This time with the system on a single disk.
matt40k Posted June 22, 2009 Posted June 22, 2009 Try plugging one disk in, see if it boots, wait it's RAID-10. Arse. Todays lesson, never use embedded\software RAID unless it's RAID-1.
SYNACK Posted June 22, 2009 Posted June 22, 2009 I had a look through the documentation for Intel RAID controllers and the config data is stored on the start of the disks, according to the Intel site they are rebranded LSI Logic MegaRAID devices. I have had the misfourtune of dealing with several of these which have decided to chew up RAID 'protected' data without reporting errors and requireing not only a full reset and reconfigure but also a full slow initialize to get them to sync up the data again without failing. They also have a nasty habit of crashing if you don't present them with perfectly clean drives to start with. I would hope that intel has got a hold of and medaled with the firmware signifigantly but from a personal experience view point I am not suprised that it has had issues. I hope that you are able to resolve it easily without too many hassles though.
contink Posted June 22, 2009 Author Posted June 22, 2009 Appreciate the tips guys... The irony of this is that I was just thinking that I was done for the day and could head home early to do some other work I'd gotten lined up in the home offie. I think that may have been mistake number 1 Ah well.... reinstallation and review of backup procedure now I think.
contink Posted June 22, 2009 Author Posted June 22, 2009 (edited) Well it seems the problems are more wide ranging.. When I boot up now I can't even get it past the Deployment disk linuz boot screen... It gets to ready and then just hangs. BIOS beeps are 2 [pause] 1... No corresponding error to be found (that I can see for that). This is with AHCI enabled instead of RAID... It does see the drives and if I turn off AHCI it can't see the DVD drive which would be a problem. BIOS beeps with RAID enabled start with 4 then 2 more IIRC... Unfortunately when I rang SCL Online at 4:30 their tech with the server knowledge had already gone home. So this rounds off my day I think... No sense in trying flog a dead un'... Edited June 22, 2009 by contink
contink Posted June 23, 2009 Author Posted June 23, 2009 Thought I'd provide an update on this after poor Pete from SCL Online suffered a very long rather random 2+ hour support call with me. To his credit he didn't scream or sob once The short version is that we have absolutely no iea what caused the original RAID configuration to just up and walk but in the process of tracking issues down we discovered a few "intersting" nuggets of info' that might have proven useful. The main one was that Intel had released a number of BIOS updates in quick succession in and around January 2009. It also looks like they screwed with the version numbers and didn't get things organised properly as a .094 version of a bios was listed on the file but .091 was appearing in the BIOS settings after update... This went on for a while. Pete found a lot of changes relating to RAID and ATA handling which raised a few red flags so it's entirely possible that Intel know something is fubar in their Integrated RAID for this board and have been crawling over each other to try and fix it... As Pete put it... For anyone else this would have been bad luck, for me it was just another day... ... Thanks! So, with BIOS updated all the way now and a new install on the system I now have two RAID 1 arrays so if the worst happens I can always recover with a standard hard drive until we can suss out what went, if it should happen again. I think it's safe to say that Pete is currently praying nothing further will happen... He's not alone.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now