Sun 7410 Getting Hammered - Suggestions?

[Duke]

Hey all, looking for any possible advice here. I've raised a case with Cutter who are looking into it.

 

We’ve got a start menu share on our S7410 that has student and staff mapped/shared (via DFS) start menus on it. Everything has been fine for ages, but yesterday at just after 10:00 CIFS, Network and CPU started getting absolutely hammered, and CIFS is now showing a hurricane on the BUI.

 

The access is only to the staff start menu and is coming from staff PCs. It drops off overnight when no one is logged on then starts up again in the morning. The type of operation going on is NtTransact and isn’t hitting the disks. Network traffic is an even split of in and out.

 

The only thing that changed around the time this started was that I added a new shortcut for software to the staff start menu. It’s just a shortcut to a html file but it did have a slightly unusual character in it (an e with an accent). Any idea if this could be the problem? Nothing else has changed to my knowledge. I’ve now removed that shortcut completely but so far it hasn’t made a difference. It might not change until the DFS caching runs out (30 mins).

 

http://www.hardenhuish.wilts.sch.uk/avatar/s7410_startmenu_highuse.png

 

Any suggestions?

Chris

[bossman]

@Duke:

 

What AV are you using?

[Duke]

Sophos, installed on all PCs but there's no AV on the 7410 itself. Sophos has caused us problems before, but nothing like this. It crossed my mind, I'll investigate further...

[apaton]

Time for Analytics!

 

Break down CPU in to process., identify which process is taking your CPU.

Break down CIFS to client access, file access.

 

See who? is doing what ? and when ?

 

 

Andy

[Duke]

I'm working on this now and analytics is saving me. More and more this is looking like a Sophos problem, even though Sophos itself isn't reporting any issues. Gonna work until I get kicked out, will post an update once I find one.

 

So far (and I'm halfway through testing), it looks like you can log onto a PC fine as a member of staff, but once you open the start menu usage on the SAN pick up loads. Rather than dropping off once the menu is closed, it then constantly stays at that level. It's like Sophos is doing an on-access scan of the start menu when you mouse over a folder (which is fine), but then keeps scanning those files rather than stopping!

 

EDIT: Disabling on-access scanning hasn't made any difference. Next step is a complete uninstall.

 

Chris

Edited November 25, 2009 by Duke

[cookie_monster]

I'm working on this now and analytics is saving me. More and more this is looking like a Sophos problem, even though Sophos itself isn't reporting any issues. Gonna work until I get kicked out, will post an update once I find one.

 

So far (and I'm halfway through testing), it looks like you can log onto a PC fine as a member of staff, but once you open the start menu usage on the SAN pick up loads. Rather than dropping off once the menu is closed, it then constantly stays at that level. It's like Sophos is doing an on-access scan of the start menu when you mouse over a folder (which is fine), but then keeps scanning those files rather than stopping!

 

EDIT: Disabling on-access scanning hasn't made any difference. Next step is a complete uninstall.

 

Chris

 

 

Can you add an exclusion in Sophos or just uninstall it on a PC to test.

[Duke]

Can you add an exclusion in Sophos or just uninstall it on a PC to test.

 

Yep, am working on it now. Should be able to give an update in 5 mins...

 

EDIT: Well, it's not Sophos. Uninstalled it on the test PC and still got the same problem. The weird thing is that it's not a bunch of files getting hammered, it's just the root programs folder on the start menu, e.g. /export/menus/Staff/Start Menu/Programs. I can see the individual files and folders pop up on the BUI analytics when I mouse over them, but they just read once then they're done. /export/menus/Staff/Start Menu/Programs sits there getting hammered after it's been opened once, even if the start menu is closed. Time to start killing services and processes...

 

EDIT 2: Alarms are going on site in a minute so I gotta run. No joy so far, even after killing off pretty much everything. I got SAN usage to dip a little when I killed some of the processes, but after giving it a few seconds it was back up to the normal level.

 

Chris

Edited November 25, 2009 by Duke

[pete]

Broken link somewhere in the root and windows is try to resolve it?

 

If this were a folder other than a start menu, I'd be looking for a corrupt Thumbs.db

[Duke]

Broken link somewhere in the root and windows is try to resolve it?

 

If this were a folder other than a start menu, I'd be looking for a corrupt Thumbs.db

 

Nothing obvious and certainly nothing that's changed recently. We do have some broken links (although not in the root) because some software is only installed in certain places. However, this has always been the case so is nothing new...

 

EDIT: Beer time, I'll let you guys know tomorrow if I get it sorted.

 

Chris

Edited November 25, 2009 by Duke

[cookie_monster]

It seems strange that one Windows client could hammer the SAN across the network, i'm assuming when your testing that it's still killing the SAN.

[Duke]

The SAN itself is actually running okay performance-wise and no one's noticed anything, probably because the traffic isn't hitting the disks, just network and CPU. I don't know exactly what NtTransact is but it doesn't seem to be doing any major reads or writes.

 

It looks like all staff PCs with a member of staff logged onto them (thus getting the staff start menu, students don't seem to be affected) are hitting it. Andy from Cutter's dropping by later for something else, might see if he has any ideas.

 

Chris

[DMcCoy]

Any mac clients? Although I have seen smb requests do the same to samba on OS X on many occasions.

[Duke]

Any mac clients? Although I have seen smb requests do the same to samba on OS X on many occasions.

 

Nope, all XP SP3.

 

I've just removed all the shortcuts that were in the root of the 'programs' folder but it's made no difference so far. Fun and games...

[pete]

The SAN itself is actually running okay performance-wise and no one's noticed anything, probably because the traffic isn't hitting the disks, just network and CPU. I don't know exactly what NtTransact is but it doesn't seem to be doing any major reads or writes.

 

It looks like all staff PCs with a member of staff logged onto them (thus getting the staff start menu, students don't seem to be affected) are hitting it. Andy from Cutter's dropping by later for something else, might see if he has any ideas.

 

Chris

Give a test staff account a brand new blank start menu with just a link to notepad in, see what happens?

[DMcCoy]

Nope, all XP SP3.

 

I've just removed all the shortcuts that were in the root of the 'programs' folder but it's made no difference so far. Fun and games...

 

Has the storage box been rebooted? I found once the smbd process crashes it will spawn replacement ones constantly.

[Duke]

Has the storage box been rebooted? I found once the smbd process crashes it will spawn replacement ones constantly.

 

Andy had a look while he was here and he thinks it might be a runaway/zombie process. Restarting CIFS would probably sort it if it is, but I've got to shut the 7410 down tomorrow as they're cutting off the power for a while so we shall see then!

 

Thanks for all the help guys, I'll update as and when I know for sure.

Chris

[john]

oooh Hurricanes I ain't seen that or rain yet on my SAN, need some shiny VMs and Sun Rays to tax mine a bit I think

[cookie_monster]

I'm runing loads on my 7110 and it's only seen a couple of showers so far. As for wind i'm still in the "Calm; smoke rises verticall" stage

[Duke]

Well a reboot fixed it! Looks like it must have been a runaway process that just happened to hit a particular file on a particular share, weird!

 

Good news is that the 7410 is fine now, handled this error with zero downtime and no affected performance and no harm done. The disks were never hit so there were no problems on the array, pretty impressive to see that level of CIFS IOPS though!

 

Thanks for all the help and suggestions everyone,

Chris

[Soulfish]

I'm runing loads on my 7110 and it's only seen a couple of showers so far. As for wind i'm still in the "Calm; smoke rises verticall" stage

 

Our 7110 has only managed to get to Cloudy so far

[Duke]

Yeah, on a normal day I've never even managed to get it to rain.

 

Forgot to mention - sched is what was taking up all the CPU time, so if anyone ever gets that problem a reboot may help and it may not be clients causing the problem which is what it initially looks like.

[teejay]

Very impressive really, that happening on a Windows or possibly even a Linux box would have brought it to its knees. Just shows how good Solaris is :-)

[Duke]

So... who wants a laugh?

 

http://www.hardenhuish.wilts.sch.uk/avatar/s7410_startmenu_highusefixed.png

 

The small spikes at the start of the CIFS graph show normal daily usage and the nightly backups. Those last three big spikes topping out at 117604 IOPS show what the last three days had been like!

 

Network usage was actually higher during the backups because the error I had didn't involve transferring a huge amount of data, but CPU peaked out at 99%. The box was entirely usable throughout all of this...

 

Chris

Edited November 27, 2009 by Duke