ABSOLUTE DISASTER - File Server running with 40% CPU for interrupts

[sonofsanta]

Christ I hate taking a holiday sometimes.

 

First week off since Xmas, looking forward to an easy morning going over e-mails and easing myself back in, and it turns out that at the end of the holiday, one of the file servers had a HDD go funny. Easy enough, it's in RAID 5, not a disaster.

 

Except now, while it tries to rebuild it (I assume)*, it is absolutely gorging on CPU. Running Process Explorer shows around 35% on Interrupts, with no clue as to what is sending all the IRQs. Students (who have their profiles housed on this server) have basically got a 50/50 chance of getting on, and loading up IE (for example) takes around 5 minutes.

 

Something is very wrong, and at the moment, I'm completely lost and a little bit scared. We have backups so worst case scenario, I have to stay behind tonight and re-install Server 2k3, but I'd really rather not.

 

This is a HP Proliant G5, the other HP's are using a Smart Array E200i Controller on System Board and are rebuilding fine (swapped a HDD out from a less vital server, so I can see that one rebuilding fine. With the faulty hard drive, oddly...)

 

Any clues at all? Any way of telling where the IRQs are coming from? Anyone had this before? Right now I'm grateful for any advice at all, just ask me questions and I'll give you what answers I can. As U2 could have sung, monday bloody monday... *sigh*

 

*can't actually tell if it's rebuilding because server is running so slow, it won't load the HP diagnostic malarkey to tell me.

[p858snake]

rebuild it outside of the operating system....

 

In my limited knowledge of RAID i would be surpised if you can do it in the OS since data is being written and changed on the drives.

[sonofsanta]

Blimey - should've come here to moan quicker!

 

Just writing this post has apparently dropped the interrupts down to about 5% CPU making the server actually usable. Although looking at the time, this could be because break is about to start so the server is being hammered a lot less...

 

Still, would love to know where all the interrupts are coming from. Some days being the boss is just no fun.

 

EDIT: obviously spoke too soon. Back up to 40-50% now -_-

[sonofsanta]

rebuild it outside of the operating system....

 

In my limited knowledge of RAID i would be surpised if you can do it in the OS since data is being written and changed on the drives.

 

Would love to try but as it stands, 50% of students getting logged on is still better than the 0% if it's offline. May try this over lunch still, they can live without flash games for one day.

 

Fun stats: on my BDC which is also rebuilding: 6 reads/sec, 8 writes/sec. On the file server with problems, 35 reads/sec and 349 writes/sec. So yes - OS rebuild is not ideal

[CtrlAltDel]

That RAID 5 array is using a lot of CPU time because of all the calculations for the Parity data. RAID 5 can hit the CPU a fair bit with a lot of sustained I/O writes and a rebuild will even more so. Unfortunatly, this kind of performance hit is unavoidable while rebuilding the array. Like has been suggested, and offline rebuild is the only other option but perhaps the greater of two evils if you need your server online...

The only way to truely minimise the overhead from a RAID5 is to spend lots of money on a card with full hardware processing of the parity data. Most, if not all integrated RAID controllers will tax the CPU for this duty.

Edited June 7, 2010 by CtrlAltDel

[sonofsanta]

After much fruitless and painful mucking about, I've just unplugged the network cable over lunch and the interrupts have plummeted. So I shall leave it churning away for now (as no-one could log on anyway, so no loss) and hopefully, once it's rebuilt, it will all be happy again.

 

Incidentally, to any passing mods who may read this - as I was posting all of this on a server with the hardened security etc. - when the tabbed forums won't load because JS is disabled, the message recommends switching to a no-tabs layout. Unfortunately the dropdown used to change layout relies on an onChange JavaScript event to do anything. Which won't tend to work when JS is disabled may just need a submit button adding for the form inside a noscript tag down there.

 

I might actually go have a cup of tea and some biscuits now, and try and have that quiet catch up on emails I was hoping for...

[sted]

it possibly depends on which g5 raid card it is (some have small cache some much more) and how the cache is set i think default is biased towards read might be worth using the hp tools and biasing it towards write for the time being

[sonofsanta]

So by unplugging the network cable, the interrupts dropped low enough to allow the RAID array to rebuild happily, took it a while but it got there.

 

Having come back in this morning though, whilst the RAID array is happy, the interrupts are still at a constant 40% and causing no end of problems.

 

So a quick question this one: does anyone know of a good tool for Server 2k3 that will help me trace the source of the interrupts? Process Monitor isn't telling me much about them at all, sadly. Once I know that I can start moving forward with this again.

[sted]

is it worth running a malware scanner on it?

[p858snake]

It would have been caused by the RAID controller trying to repair your broken raid whilst the network was still trying to use the resources of the server.

[sonofsanta]

It would have been caused by the RAID controller trying to repair your broken raid whilst the network was still trying to use the resources of the server.

 

I figured that was why the interrupts dropped while the array was rebuilding, but the array is now rebuilt and the interrupts are still riding high. KrView tells me that it's ntoskrnl, and it definitely seems to be disk activity causing the problems (file system runtime library and disk cache functions come top of the zoom results). I still don't understand why disk activity on a file server is such a problem, though. Replicating data across now so I can down the server and have a proper mess around with it, disable hardware etc. and do more to actually pin the problem down... still not a fun day today.

[ind1ekid]

I had a single sas drive die in a raid 5 on a proliant 380g5 not too long back, the spare kicked in straight away and once I got a new drive in to replace the dead on it soon rebuilt the raid quickly and quietly in the OS, P400 raid card. Lucky me ey?

[CtrlAltDel]

Its the first time i've heard of this behaviour carrying on after the RAID has successfully rebuilt.

Processor usage that high could indicate a problem with bus mastering/DMA transfers from an item of hardware. Why it would decide to start that after a disk failure I don't know.

As you say, having time to disable devices and update drivers will be the key.

Have you had time to fully power down the server and turn the power off for a few seconds at the plug? I've had experience of hardware becoming stuck in a loop due to a glitch in the driver and this has been a good fix on many an occasion.

[sonofsanta]

As you say, having time to disable devices and update drivers will be the key.

Yeah, I did... didn't work out, as I'll get to in a moment...

Have you had time to fully power down the server and turn the power off for a few seconds at the plug? I've had experience of hardware becoming stuck in a loop due to a glitch in the driver and this has been a good fix on many an occasion.

This is my usual fix for most problems - shutdown, pull the power cables, wait for 30 seconds and click the power button to discharge the power fully and then try, but it didn't have any effect.

 

As for disabling devices - at break time yesterday I thought I would take the opportunity to disable the new SCSI card that had gone in for the new tape drive, in case it had stemmed from that. Disabled in Device Manager, yes please to the restart, goes through the motions and theeeeeeeeeen it reboots just as it is about to show the login screen. At which point everything was royally, erm, "made love to" and it forced my hand with regards to breaking out the Server 2k3 discs and starting over.

 

So the net result is that it's been playing beautifully all day, my creeping apology has been made at staff briefing, and now I may never know what was actually the problem as all the hardware and all the drivers are back in place. There was surprisingly little information about the web for the issue though, at least that helped, which leads me to hope it was such a bizarre situation and so completey abnormal that I should be safe from it ever occurring again. If it does, though, I'm reaching straight for the install discs.

 

I've handed thanks out in this thread to people who chimed in; sometimes knowing this place is here as a fallback is all that keeps me going. Cheers!