Sun 7410 Getting Hammered - Suggestions?

[Duke]

Has the storage box been rebooted? I found once the smbd process crashes it will spawn replacement ones constantly.

Andy had a look while he was here and he thinks it might be a runaway/zombie process. Restarting CIFS would probably sort it if it is, but I've got to shut the 7410 down tomorrow as they're cutting off the power for a while so we shall see then!

Thanks for all the help guys, I'll update as and when I know for sure.
Chris

[john]

oooh Hurricanes I ain't seen that or rain yet on my SAN, need some shiny VMs and Sun Rays to tax mine a bit I think

[cookie_monster]

I'm runing loads on my 7110 and it's only seen a couple of showers so far. As for wind i'm still in the "Calm; smoke rises verticall" stage

[Duke]

Well a reboot fixed it! Looks like it must have been a runaway process that just happened to hit a particular file on a particular share, weird!

Good news is that the 7410 is fine now, handled this error with zero downtime and no affected performance and no harm done. The disks were never hit so there were no problems on the array, pretty impressive to see that level of CIFS IOPS though!

Thanks for all the help and suggestions everyone,
Chris

[Soulfish]

I'm runing loads on my 7110 and it's only seen a couple of showers so far. As for wind i'm still in the "Calm; smoke rises verticall" stage

Our 7110 has only managed to get to Cloudy so far

[Duke]

Yeah, on a normal day I've never even managed to get it to rain.

Forgot to mention - sched is what was taking up all the CPU time, so if anyone ever gets that problem a reboot may help and it may not be clients causing the problem which is what it initially looks like.

[teejay]

Very impressive really, that happening on a Windows or possibly even a Linux box would have brought it to its knees. Just shows how good Solaris is :-)

[Duke]

So... who wants a laugh?



The small spikes at the start of the CIFS graph show normal daily usage and the nightly backups. Those last three big spikes topping out at 117604 IOPS show what the last three days had been like!

Network usage was actually higher during the backups because the error I had didn't involve transferring a huge amount of data, but CPU peaked out at 99%. The box was entirely usable throughout all of this...

Chris