File Servers Slow

I have 3 file servers which seem to be running slowly - although on the console they are super responsive. I've been through everything i can think off to try and diagnose the faults to no avail.

Two of the servers are not that old (about 2 years) HP DL380 G5 Dual Quad Xeons with 6gb ram.
The Third is HP Dl380 G4 Dual Dual Xeon with 4GB Ram. But i don't think its a hardware problem (enless switching)

First of all i removed the HP network teams which i had recently implemented to try and speed access up and thought as this had changed it could have been this - not this.

Also checked the switch for any packet errors - nothing.

i've been through the raid diagnostics and can't see anything wrong. I've checked the scheduled defrag was running (schduled for each sunday) yep all working.

Also checked there doesn't seem to be that many users active during the periods of slowness (around 100). The Pages/Second tends to sit during heavy loads averaging around 6/700 pages/s but sometimes is maxing at 5500.

I've just disabled FSRM quotas to check that its not that being slow when files are being written to the disk.

Its not our AV as its on access scanning is disabled and is scheduled to run at 5pm to check for virus's and all client machines scan the files anyway.

Any other ideas for me to check before i go throw these servers out the window? I'm thinking it could be a strange config as our predecessors seem to have done some strange things on other servers.

Any recent s/w additions? Also, I know it maybe teaching granny to suck eggs etc but are the drivers/firmware for things upto date?

Quote:

---

Originally Posted by glennda

Also checked there doesn't seem to be that many users active during the periods of slowness (around 100).

---

Do the periods of slowness occur at particular times of the day? When the slow-downs happen, are different people trying to access lots of different (larger) files - is the cache getting saturated? Are you using hardware or software RAID - does the RAID controller have its own RAM or does it use the system RAM? How is the cache configured - write-through or write-back?

No recent software updates - all drivers/firmware should be up to date as i updated them last october. although i've noticed a newer firmware for Dl380 g5 but the G4 is all up to date.

the G5's use a Hardware raid with the HP P400 with 512mb ram the read write cache is set to 50/50 which was all setup before i arrived do you think i should try changing it?

Quote:

---

I have 3 file servers which seem to be running slowly

---

Instinct tells me it could well be a networking related problem. What switches are you using? If you download a large file, such as Windows 7 SP1 x64 (903MB) and copy it from your server to a workstation and vica versa, how long is it taking?

Quote:

---

Originally Posted by Michael

Instinct tells me it could well be a networking related problem.

---

Yeah. Someone messing about creating a loop somewhere or damage?

Is this only happening when users are logged on or does the issue occur after the school has finished when the kit is still on but hardly anyone logged on?

Is it worth having your PC with multiple DOS screens open pinging the servers to see what happens when the sh1t starts going into the fan? We done this a few months back and as soon as the 'fan' started to get messy we started unplugging fibres at the core to isolate the peripheral cabs. Once we located the cab we very soon located the room etc and the location of the offending 'loop'.

I've just written a whole massive response and chrome crashed!

I was thinking it was a network related problem as our core is not very good (see attached image for all to be explained! basic) - although while sitting on one of the other servers which is fine (say our exchange box for example) plugged directly into the core is still slow to load files from the machine sitting next to it.

I don't think there is a loop anywhere as i've had a few of them before (where there are 20 sockets for students to connect laptops to and they loop them) and it doesn't seem to be that as normally nothing works then rather then just 3 file servers. Plus i've got 15 + other servers (not all file servers but a couple are) that are just fine.

I'm currently getting quotes to replace the core but its looking at loads due to the fact i need will 48 sfp ports and then 28 copper come the summer!

Attachment 9702

toby i suspect it is the hp team. / nics /drivers,

the drivers /software is still there after the team is removed btw.

are they broadcom ?

i'd be tempted to virtualize with esxi and use the local storage for the vm, just to not have to use "real" windows drivers.

bit tricky though with loadsa data though.

Quote:

---

Originally Posted by glennda

the G5's use a Hardware raid with the HP P400 with 512mb ram the read write cache is set to 50/50

---

Sorry, I have no idea what "50/50" means in this context. An HP p400 hardware RAID controller with 512MB of RAM certainly sounds like it should be able to handle a decent amount of throughput, although you might want to check what caching scheme it's using - I know I've seen at least one server's disk I/O periodically slow to a crawl due to the cache using write-through rather than write-back caching. This turned out to be due to a RAID card cache battery backup failure - you might want to check your RAID card's backup battery is present and working (or appropriate settings are in place if your card doesn't have a battery).

Quote:

---

Originally Posted by oxide54

bit tricky though with loadsa data though.

---

I thought about virtualizing but if i did i would rather put it on the san but due to the amount of data involved i don't think i can.

The HP software has always been there i just setup the team on the nics rather then having single.

Quote:

---

Originally Posted by dhicks

Sorry, I have no idea what "50/50" means in this context. An HP p400 hardware RAID controller with 512MB of RAM certainly sounds like it should be able to handle a decent amount of throughput, although you might want to check what chaching scheme it's using - I know I've seen at least one server's disk I/O periodically slow to a crawl due to the cache using write-through rather than write-back caching. This turned out to be due to a RAID card cache battery backup failure - you might want to check your RAID card's backup battery is present and working (or appropriate settings are in place if your card doesn't have a battery).

---

Sorry doesn't make much sense does it. The read cache is 50% of the ram and the write cache is the other 50%.

The management software says the battery status is ok - in fact the only error its moaning about is the fact the ILO port is disconnected.

The 50/50 refers to the percentage of the cache used for read/write, you can change it depending on your usage pattern. Those controllers are very able and quick so I don't think that those are your bottleneck. What version of Windows server are they running? Later versions of Windows server can take longer to initially open up a connection as the server and client wade their way through SMB2 to figure out if they trust each other or not. Once the connections are open and authenticated though they should be fast.

Quote:

---

Originally Posted by SYNACK

The 50/50 refers to the percentage of the cache used for read/write, you can change it depending on your usage pattern. Those controllers are very able and quick so I don't think that those are your bottleneck. What version of Windows server are they running? Later versions of Windows server can take longer to initially open up a connection as the server and client wade their way through SMB2 to figure out if they trust each other or not. Once the connections are open and authenticated though they should be fast.

---

Its running windows 2k3 at the moment. Also there is 3 logical drives 1 at raid 1 + 0 and 2 at raid 5.

Quote:

---

Originally Posted by glennda

Also there is 3 logical drives 1 at raid 1 + 0 and 2 at raid 5.

---

Those logical drives are provided by the RAID card? How are they utilised by Windows - is the RAID 10 (a four-drive RAID array?) drive the OS drive and the two RAID 5 drives separate shares or storage areas?

Quote:

---

Originally Posted by dhicks

Those logical drives are provided by the RAID card? How are they utilised by Windows - is the RAID 10 (a four-drive RAID array?) drive the OS drive and the two RAID 5 drives separate shares or storage areas?

---

We have for the raid 1+0 2 x 73gb disks giving around 68gb for os then the raid 5 arrays are 3 146gb. they are all 10k sas drives with no errors according to the diagnostic softwares