File Servers Slow

[glennda]

I have 3 file servers which seem to be running slowly - although on the console they are super responsive. I've been through everything i can think off to try and diagnose the faults to no avail.

 

Two of the servers are not that old (about 2 years) HP DL380 G5 Dual Quad Xeons with 6gb ram.

The Third is HP Dl380 G4 Dual Dual Xeon with 4GB Ram. But i don't think its a hardware problem (enless switching)

 

First of all i removed the HP network teams which i had recently implemented to try and speed access up and thought as this had changed it could have been this - not this.

 

Also checked the switch for any packet errors - nothing.

 

i've been through the raid diagnostics and can't see anything wrong. I've checked the scheduled defrag was running (schduled for each sunday) yep all working.

 

Also checked there doesn't seem to be that many users active during the periods of slowness (around 100). The Pages/Second tends to sit during heavy loads averaging around 6/700 pages/s but sometimes is maxing at 5500.

 

I've just disabled FSRM quotas to check that its not that being slow when files are being written to the disk.

 

Its not our AV as its on access scanning is disabled and is scheduled to run at 5pm to check for virus's and all client machines scan the files anyway.

 

Any other ideas for me to check before i go throw these servers out the window? I'm thinking it could be a strange config as our predecessors seem to have done some strange things on other servers.

[bodminman]

Any recent s/w additions? Also, I know it maybe teaching granny to suck eggs etc but are the drivers/firmware for things upto date?

[dhicks]

Also checked there doesn't seem to be that many users active during the periods of slowness (around 100).

 

Do the periods of slowness occur at particular times of the day? When the slow-downs happen, are different people trying to access lots of different (larger) files - is the cache getting saturated? Are you using hardware or software RAID - does the RAID controller have its own RAM or does it use the system RAM? How is the cache configured - write-through or write-back?

[glennda]

No recent software updates - all drivers/firmware should be up to date as i updated them last october. although i've noticed a newer firmware for Dl380 g5 but the G4 is all up to date.

 

the G5's use a Hardware raid with the HP P400 with 512mb ram the read write cache is set to 50/50 which was all setup before i arrived do you think i should try changing it?

[Michael]

I have 3 file servers which seem to be running slowly

 

Instinct tells me it could well be a networking related problem. What switches are you using? If you download a large file, such as Windows 7 SP1 x64 (903MB) and copy it from your server to a workstation and vica versa, how long is it taking?

[bodminman]

Instinct tells me it could well be a networking related problem.

 

Yeah. Someone messing about creating a loop somewhere or damage?

 

Is this only happening when users are logged on or does the issue occur after the school has finished when the kit is still on but hardly anyone logged on?

 

Is it worth having your PC with multiple DOS screens open pinging the servers to see what happens when the sh1t starts going into the fan? We done this a few months back and as soon as the 'fan' started to get messy we started unplugging fibres at the core to isolate the peripheral cabs. Once we located the cab we very soon located the room etc and the location of the offending 'loop'.

[glennda]

I've just written a whole massive response and chrome crashed!

 

I was thinking it was a network related problem as our core is not very good (see attached image for all to be explained! basic) - although while sitting on one of the other servers which is fine (say our exchange box for example) plugged directly into the core is still slow to load files from the machine sitting next to it.

 

I don't think there is a loop anywhere as i've had a few of them before (where there are 20 sockets for students to connect laptops to and they loop them) and it doesn't seem to be that as normally nothing works then rather then just 3 file servers. Plus i've got 15 + other servers (not all file servers but a couple are) that are just fine.

 

I'm currently getting quotes to replace the core but its looking at loads due to the fact i need will 48 sfp ports and then 28 copper come the summer!

 

[oxide54]

toby i suspect it is the hp team. / nics /drivers,

 

the drivers /software is still there after the team is removed btw.

 

are they broadcom ?

 

i'd be tempted to virtualize with esxi and use the local storage for the vm, just to not have to use "real" windows drivers.

 

bit tricky though with loadsa data though.

[dhicks]

the G5's use a Hardware raid with the HP P400 with 512mb ram the read write cache is set to 50/50

 

Sorry, I have no idea what "50/50" means in this context. An HP p400 hardware RAID controller with 512MB of RAM certainly sounds like it should be able to handle a decent amount of throughput, although you might want to check what caching scheme it's using - I know I've seen at least one server's disk I/O periodically slow to a crawl due to the cache using write-through rather than write-back caching. This turned out to be due to a RAID card cache battery backup failure - you might want to check your RAID card's backup battery is present and working (or appropriate settings are in place if your card doesn't have a battery).

Edited March 15, 2011 by dhicks

[glennda]

bit tricky though with loadsa data though.

 

I thought about virtualizing but if i did i would rather put it on the san but due to the amount of data involved i don't think i can.

 

The HP software has always been there i just setup the team on the nics rather then having single.

[glennda]

Sorry, I have no idea what "50/50" means in this context. An HP p400 hardware RAID controller with 512MB of RAM certainly sounds like it should be able to handle a decent amount of throughput, although you might want to check what chaching scheme it's using - I know I've seen at least one server's disk I/O periodically slow to a crawl due to the cache using write-through rather than write-back caching. This turned out to be due to a RAID card cache battery backup failure - you might want to check your RAID card's backup battery is present and working (or appropriate settings are in place if your card doesn't have a battery).

 

Sorry doesn't make much sense does it. The read cache is 50% of the ram and the write cache is the other 50%.

 

The management software says the battery status is ok - in fact the only error its moaning about is the fact the ILO port is disconnected.

[SYNACK]

The 50/50 refers to the percentage of the cache used for read/write, you can change it depending on your usage pattern. Those controllers are very able and quick so I don't think that those are your bottleneck. What version of Windows server are they running? Later versions of Windows server can take longer to initially open up a connection as the server and client wade their way through SMB2 to figure out if they trust each other or not. Once the connections are open and authenticated though they should be fast.

[glennda]

The 50/50 refers to the percentage of the cache used for read/write, you can change it depending on your usage pattern. Those controllers are very able and quick so I don't think that those are your bottleneck. What version of Windows server are they running? Later versions of Windows server can take longer to initially open up a connection as the server and client wade their way through SMB2 to figure out if they trust each other or not. Once the connections are open and authenticated though they should be fast.

 

 

Its running windows 2k3 at the moment. Also there is 3 logical drives 1 at raid 1 + 0 and 2 at raid 5.

[dhicks]

Also there is 3 logical drives 1 at raid 1 + 0 and 2 at raid 5.

 

Those logical drives are provided by the RAID card? How are they utilised by Windows - is the RAID 10 (a four-drive RAID array?) drive the OS drive and the two RAID 5 drives separate shares or storage areas?

[glennda]

Those logical drives are provided by the RAID card? How are they utilised by Windows - is the RAID 10 (a four-drive RAID array?) drive the OS drive and the two RAID 5 drives separate shares or storage areas?

 

We have for the raid 1+0 2 x 73gb disks giving around 68gb for os then the raid 5 arrays are 3 146gb. they are all 10k sas drives with no errors according to the diagnostic softwares

[Michael]

I don't think it's a RAID or disk issue, simply because it's fast on the server itself.

 

I would agree with others - proceed to update drivers on your servers and make sure they're set to Auto with all default settings.

 

Have you tried resetting your switches? And how old are your switches?

[dhicks]

Is your RAID card's caching scheme set to write-through, write-back or something else?

 

We have for the raid 1+0 2 x 73gb disks giving around 68gb for os

 

I may be misunderstanding something here, but isn't a RAID 1+0 array two RAID-1 arrays striped together, i.e. a minimum of four drives?

 

then the raid 5 arrays are 3 146gb.

 

How does Windows utilise those two arrays - does it use them each as a separate share, or does it combine them into a single share a dynamic drive, a DFS, or something else?

[dhicks]

I have 3 file servers which seem to be running slowly - although on the console they are super responsive.

 

Sorry, just re-read that bit - I originally read that as "RDP access to the machine is working fine", but you actually mean "running disk operations locally on the servers, disk I/O seems fine". As others have discussed, then, it's likly to be a networking issue rather than local hardware.

[glennda]

Yeah I've updated all the firmware to the latest versions anyway.

 

What i thought it could have been is the FSRM although i'm not ruling out the network issues.

 

I'm going to sit down with my boss later and explain that i need about 10k to replace the core switching which i have wanted to do for ages!

[glennda]

I don't think it's a RAID or disk issue, simply because it's fast on the server itself.

 

I would agree with others - proceed to update drivers on your servers and make sure they're set to Auto with all default settings.

 

Have you tried resetting your switches? And how old are your switches?

 

 

I've power-cycled the core about a week ago (its been going on longer then that) - its anything up to about 4 years old but nobody knows 100% as none of us where here when it was purchased.

 

I'm currently getting prices on replacements anyway as i want to replace it but i don't want to spend 10k if its something with the servers configs

[Michael]

It's really difficult to say just how big your network is (from your diagram), but on newer HP/Cisco switches they have upto 4 transceiver slots which can be used for Ethernet or Fibre connections.

 

I would also buy 48 port switches wherever possible as this can save a lot of money and simplify your network. I would also stick to the same brand as this also will simplify your network.

[glennda]

I've got 48 port switches on the Edge (most are 18 month old hp 2610's few older netgears) there is a couple of 24 port's around but only where there is not a need 48 ports. currently using about 38 fiber connections (2 are for the wireless controllers) but come the summer this will be increased as we have having a whole load of building work done which requires another 6 switches in various parts of the school meaning we will be up to 44 then a few for further expansion after that.

 

Currently getting pricing from HP on an 5400zl series chassis although we may end up with a managed stack of E4800G's

[Michael]

I would probably go with a group of E4800G's at the core and then replace any other Netgear switches you have in your network. The HP 2610 is a good switch, so I doubt this is causing any problems.

[glennda]

Yeah I'm its in the plan to replace the netgears this summer - i'm just waiting on price for the e4800g's then i have to beg to get it added into this years money - rather then next years!

[glennda]

I've just sat down with my boss to talk about it, he doesn't beleive me and things i need to prove its the switch(es) causing the problem (the fact i've ruled everything else out doesn't count).

 

Does anybody know of any open-source software which i can use to monitor things on the switch? as the web interface on the switches i have is awful!