Mystery Diagnosis

[bambroso]

I am at wits end with a server that has baffled my efforts to diagnose the following symptoms...

This all started about 2 months ago or so.. Nothing changed at the time so I can't blame a config change...

 

OS is Server 2003 R2 32bit (SP2) on Dell Power Edge 2850

Mirrored OS drives (2x73GB 15K) and 4 RAID5 Disks (300gb 10K RPM) all on PERC4e.

6GB RAM (using BIOS SPARE BANK ENABLED)

 

When backing up the server using Backup Exec 10D the server would just stop responding and eventually restart. Removed agent and reisntalled (both manually and push) and no change. If I attemtp to copy large files from other network locations it stops responding. When attempting to install the current patch (malicious software patch) it stops. It would seem to be OS issue but coupled with these issues we have had several hardware related anomolies.

Two failed drives (that were fixed using REBUILD in Dell Open Manage)

A PROC_INIT error failure that was fixed by a reboot.

A recent blue screen RAM PARITY CHECK / MEMORY PARITY ERROR. These all seem like RANDOM hardware failures aside from the disk failure which happended twice and was the same disk each time.

And the server kept complaining about a PCIe Riser problem (EB113) that Dell fixed with a reseat of the Riser.

 

Things we have done thus far:

Updated BIOS to current

Updated Disk FW to current

Updated PERC FW to current

RAM memory test

Ran all Diagnostics using DELL OPENMANAGE DIAGS.

Performed Repair on Windows..

 

The server boots fine and runs great offering up SOL 2005 and a web app that usses JBOSS serving up digitized newspapers that are fairly large (upwards of 25 -30 BM's). Its only when you try to stress the disks that things go awry. After the Windows Repair we try and install service pack 2 and not only does it cause the server to become unresponsive, it attempts to extract its files on a drive other than the %system% drive...

 

Where's Dr. House when you need him.. Any ideas or where to start...

 

Thanks,

 

\Bob

[contink]

I doubt very much this is related but recently I noticed my home server started keeling over and rebooting around the same time every day.

 

I managed to tie it down to a Sophos thorough virus scan that was running at the time. No idea why it was forcing a reboot but obviously it was hitting something it didn't like.

 

Took Sophos off completely and now it's working fine.... Like I said though, very unlikely to be related.

[speckytecky]

I'm thinking you are talking about a Dell server that is out of guarantee? Mine was as well last time it played up. Nevertheless Dell Server support came up trumps and even remotley accessed the beast and solved the problem. Ten out of 10 to Dell Server support. Worth a try in your case?

[apeo]

I would try what speckytechy suggested, otherwise (as painful as it is) do a clean rebuild and restore from backups prior to the problem (although the problem does seem to be hardware).

[dhicks]

We had a Dell server act in a similar way - it would be going fine until you tried to do anything involving lots of disk access, at which point it got confused and gradually conked out. The problem turned out to be caused by the RAID controller having its cache policy set to write-through instead of write-back (i.e. disk writes wound up taking a long time, and the controller was getting overloaded with write requests). The PERC controller had automatically set cache policy to write-through when some error had occurred (power outage or disk failure). Might be worth a quick check.

 

--

David Hicks

[joe90bass]

I had an intermittent fault here on an Evesham server (SATA not SCSI) turned out to be faulty SATA leads! Had me banging my had for a while, we had disks swapped out, and even a SATA RAID card!

 

If you've checked everything else and still banging your head prob worth checking, as leads are the last thing we often think of being faulty!

[bambroso]

bit of an update..

Uninstalled SOPHOS (as per reply)

Checked PERC config for cache method (writebk) per reply

 

 

After many hours of doing this and that with Dell they decided to replace almost everything.. Motherboard, Backplane, Riser, RAID Card, and the one hard drive that had failed twice. the tech who came out then tried to get the RAID arrays back online. The RAID 5 array showed up in the PERC as ready but the two mirrored dries showed up as installed but they could not get them to a ready state.. Unisys tech said he had never seen this before (gulp)!

Finally got server to boot though we are still having similar issues. We at least have been able to install SP2 and some other updates and get pretty good backups of some of the data. When I say some, I mean we can get relatively small amounts of data off to our Backup Exec server but anything large causes it to become unresponsive again. One of the partitions has 240 GB on it. We got 70 of it and then nothing. Interesting note, when we attached a USB drive and manually copied files it copied all 240 GB (took 4 hours) and never crashed. the major problem at this point is consistent at least in that it always ends in a Blue screen with error :

STOP 0x000000F4 (0x0000003,0x895b6a70,0x895B6BD4,0x8094C6E6)

A rare error I hear... I backed off the RAM amount to 2 GB just to see if that had anything to do with it (from 4GB) but no change..

I'll keep plugging away!

\Bob

[Michael]

Silly question, but why do you have 6GB of memory using a 32bit OS? In practice, a 32bit OS will only use 3GB max. You reduced it to 2GB and I would leave it at 2GB.

 

I did some searching on 0x000000F4 and I found this: http://search.microsoft.com/results.aspx?q=0x000000F4&qsc0=0&SearchBtn0=Search&FORM=QBME1&l=1&mkt=en-US&PageType=99

 

Seeing as you have changed a few components, the problem as far as I can see points to the hard drive(s) being at fault. This is the general consistency reading those support articles and from reading your descrption of the problem.

Hard drives can and do create lots of strange and inconsistent problems before completely dying. I feel this is why it's creating you problems to fully resolve the problems you're having.

[bambroso]

Silly question, but why do you have 6GB of memory using a 32bit OS? In practice, a 32bit OS will only use 3GB max. You reduced it to 2GB and I would leave it at 2GB.

 

I did some searching on 0x000000F4 and I found this: http://search.microsoft.com/results.aspx?q=0x000000F4&qsc0=0&SearchBtn0=Search&FORM=QBME1&l=1&mkt=en-US&PageType=99

 

Seeing as you have changed a few components, the problem as far as I can see points to the hard drive(s) being at fault. This is the general consistency reading those support articles and from reading your descrption of the problem.

Hard drives can and do create lots of strange and inconsistent problems before completely dying. I feel this is why it's creating you problems to fully resolve the problems you're having.

 

Thanks for the reply...

 

We have redundant memory set in the BIOS so the OS only sees 4GB (even though there is 6GB installed)... The OS will support 3 if you use the

 

I will look at the search index you provided but they all seem to talk about configurations that do not match ours (Japanese Edition of R2, using kanji characters, PATA drives, Resizing Windows,etc..)

 

Whats weird is that I was ablt to copy off 120 GB onto a USB drive with no problems.. I try to do it over the network and it no likey..

 

Keep plugging and thanks for the info..

 

\Bob

[bambroso]

great news.. We upgraded Backup Exec 10D to service pack 3 this morning and we have successfully backed up over 200GB of data.. The only exception is we do not get clean backups unless we specifically set the open file option to use the Microsoft Volume Shadow Copy Service.. If we leave it to "automatic" we get a complete backup with exceptions:

AOFO: Initialization failure on: "\\SERVER\D:". Advanced Open File Option used: Symantec Volume Snapshot Provider (VSP).

Snapshot provider error (0xE0008523): VERITAS Volume Snapshot Provider (VSP) cannot snap the specified volume

 

I looked at the Symantec site for any thing but info was scant.

We are upgrading to 11D over the Christmas break so we'll see. At least the dam thing is stable (so far)..

 

Thanks for all the help..

 

\Bob

[joe90bass]

Excellent news! Backup problems are the stuff of nightmares

[Michael]

Glad to hear you're making good progress. It is very mysterious I totally agree. I think once you have a full backup, I would probably start again from scratch as solving further problems will no doubt take much more of your time.

 

You appear to have made much better progress than Dell have by swapping out hardware components...

[PiqueABoo]

In practice, a 32bit OS will only use 3GB max.

 

32-bit Windows Enterprise OS supports 64GB unless you're trying to run it on CPUs that predate the Pentium Pro i.e. don't support PAE. Standard definitely supports 4GB, you just can't play with it all because of reservations for hardware I/O et al, but you'll typically get more than 3GB on a decent server.

 

[Just in case, I am not talking about anything to do with the /3GB switch which is a different concept/issue]

 

Backup Exec 10D to service pack 3

 

IME that's the only reasonably reliable version of 10D (there's also an an elusive SP4 which adds a couple of arcane fixes and is probably also OK). Not so sure about 11D yet, the initial release was definitely flaky, the current one seems OK but I wouldn't rush to install it if 10D is doing everything you need.