Windows Server 2008 R2 File Server Performance

[dhicks]

Hello All,

 

I seem to be having an issue getting the best performance out of our Windows Server 2008 R2 file server. This is running as a VM under XenServer 6.2, and is currently assigned 4 virtual cores, 8GB of RAM and its own dedicated disk array passed through as a removeable device from XenServer. This disk array consists of 4 4TB drives on a hardware RAID controller running as a RAID 5 array. It has a virtual network interface assigned to it consisting of 3 physical gigabit NICs bonded together. All things considered, it should have plenty of resources available, and from what I can judge from Windows resource monitor the hardware isn't overly-taxed - CPU isn't going much above 10%, RAM usage around 2GB, network utilisation around 500Mb/s. If I've read the documentation right, a disk queue length of anything less than number of spindles + 2 for a RAID array is okay, and our disk queue length seems to hover around the "1" mark - plenty of capacity to spare, there. I've recently made sure the XenServer tools are installed on the VM, which changed the virtual NIC's driver and made it gigabit-capable.

 

This file server is now handling all shared file areas as well as all users' My Documents and Desktop folders and, mostly, seems to be running okay. However, we still seem to be having performance issues in some particular cases - by the looks of it, when there are simply lots of people using the system at a time. During busy times (i.e. when a class all logs on at the same time), some users will seem to be "locked out" during the login process, as if the file server is simply too busy to process their request right at the moment and is queing requests.

 

The only readout that I an see on the performance monitor that seems to be constantly at the top of its limit is the "TCP Connections" readout - it seems to hover near its "100" top limit. Are we running out of available TCP connections to the file server? Is this even a limitable figure? I get that the graph auto-scales to fit the figure that it's displaying, but it rather suspiciously never seems to go above 100, even though it seems to hit 100 at times. Is something limiting the number of TCP connections? If so, is that a limit that can be set in Windows, or is that a limit set from the XenServer side of things? I've investigated settings for Windows file servers and added Microsoft's suggested registry settings, however no mention seems to be made of a TCP connection limit. Is that limit actually a result of something else being limited - number of threads or similar?

 

Are there some diagnostic / benchmark tools I can run on the file server to judge how well it is performing overall? Will benchmarking to get an IOPS figure of some kind be any help, or will another benchmark be better in this case?

[PotNoodleTech]

Does the hardware Raid controller have battery backup and cache (usually look like RAM sticks). Without Battery backup to enable write caching, and the RAM cache, RAID 5 on SATA drives is going to be slow, especially when writing data.

 

A great little program for identifying disk speed problems is "CrystalDiskMark" Pop this on your VM and do two tests for me.

 

1. "All" 100MB Test with 3 Test runs. This will identify if you have raid controller cache or not.

 

2. "All" 1000MB Test with 3 Test runs. This wil give a general performance indicator.

 

Post your screenies for us to have a gander at.

[dhicks]

Does the hardware Raid controller have battery backup and cache (usually look like RAM sticks).

 

Yes, the RAID card is an LSI 9271-8i with the battery backup module attached and 1GB of cache RAM. I've ensured the array settings in the RAID card's BIOS have caching turned on and that the cache mode is set to "always write-back" to ensure the cache controller doesn't decide to switch back to write-through caching on a reboot or something. And with an average disk queue length of 0.5 on a 4-spindle array, physical disk performance looks pretty good, so I don't think it's a cache / controller problem.

 

Interestingly, the server's TCP Connections graph just broke the 100 limit - now up to 130 TCP conecctions, so it turns out that 100 wasn't a hard limit after all. Disk queue length seemed to go up to around 5 as a whole class logged on this morning, with the ICT teacher saying that some operations seemed "a bit slow". It seems that some users get "locked out" fora few minutes when logging on, as if the server is simply not able to process their requests just yet. All pupils' My Documents and Desktop folders are redirected to the file server, so on logon I guess it's having to do a fair few read / directory list operations, although I shouldn't have thought there'd be that much whole-file reading or writing neccesary.

 

A great little program for identifying disk speed problems is "CrystalDiskMark" Pop this on your VM and do two tests for me.

1. "All" 100MB Test with 3 Test runs. This will identify if you have raid controller cache or not.

2. "All" 1000MB Test with 3 Test runs. This wil give a general performance indicator.

 

Good plan, I'll give that a try - I might have to wait until this evening to run test on a live server, though.

[PotNoodleTech]

Yes, the RAID card is an LSI 9271-8i with the battery backup module attached and 1GB of cache RAM. I've ensured the array settings in the RAID card's BIOS have caching turned on and that the cache mode is set to "always write-back" to ensure the cache controller doesn't decide to switch back to write-through caching on a reboot or something. And with an average disk queue length of 0.5 on a 4-spindle array, physical disk performance looks pretty good, so I don't think it's a cache / controller problem.

 

Ok that all sounds good.

 

Interestingly, the server's TCP Connections graph just broke the 100 limit - now up to 130 TCP conecctions, so it turns out that 100 wasn't a hard limit after all. Disk queue length seemed to go up to around 5 as a whole class logged on this morning, with the ICT teacher saying that some operations seemed "a bit slow". It seems that some users get "locked out" fora few minutes when logging on, as if the server is simply not able to process their requests just yet. All pupils' My Documents and Desktop folders are redirected to the file server, so on logon I guess it's having to do a fair few read / directory list operations, although I shouldn't have thought there'd be that much whole-file reading or writing neccesary.

 

Yeh we're running at 190 connections here so there definatly should be no limit on that (2003 server on Vmware tho so a bit different to yours). We use redirected docs desktop and start menu and roaming profiles so no shortage of operations needed.

 

Good plan, I'll give that a try - I might have to wait until this evening to run test on a live server, though.

 

Yes if you're getting disconnects under some sort of load then don't run the benchmark "in hours". It's unusual to get "locked out" when busy as you suggest, I've only ever seen things slow down here, not stop.

[dhicks]

Yeh we're running at 190 connections here so there definatly should be no limit on that (2003 server on Vmware tho so a bit different to yours). We use redirected docs desktop and start menu and roaming profiles so no shortage of operations needed.

 

Load-wise you sound not too dissimilar to us - what kind of disk queue length, disc performance and network peformance are you seeing? Our disk queue length seems to average around 1, disc performance is around 70MB/sec and network around 500Mb/sec.

 

CrystalDiskInfo doesn't seem to find any disks on our VM - probably something to do with the drivers. I'll investigate further, and probably try IOMeter.

 

It's unusual to get "locked out" when busy as you suggest, I've only ever seen things slow down here, not stop.

 

I'm starting to think it's maybe a client configuration issue. The laptops in the ICT room seem to take a while to do some file operations, like when you go to "File, Open..." in a program there'll be a pause of a minute or so until the machine responds, rather as if the machine thinks it's searching an old path or something. No actual complaints from the reast of the school, so maybe it's time to reimage the ICT room and see if that solves the problem. I'd still like to make sure our file server is giving top performance, though.

[PotNoodleTech]

Is it in MS office you're having the problem? Have problems with Office 2003 and 2010 sometimes hanging when using the "File..Open" box and using the drop down box at the top to navigate - if you click on the left hand options "Documents" or "Computer" or whatever it works a treat.

 

That is an issue with MS Office trying to browse the network neighbourhood which you can't seem to disable on Windows 7. (didnt happen on XP).

[dhicks]

Have problems with Office 2003 and 2010 sometimes hanging when using the "File..Open" box and using the drop down box at the top to navigate - if you click on the left hand options "Documents" or "Computer" or whatever it works a treat.

 

I got the impression there's simply a delay between the user hitting "File, Open..." and the "open" dialog actually appearing. I'm going to try reimaging the room's worth of machines - I've been moving file areas around quite a bit recently, so the machines could well just be confused.

[dhicks]

CrystalDiskInfo doesn't seem to find any disks on our VM

 

Right, sorry, am being a wolly - completly different program. After running CrystalDiskMark, I have the following results:

 

 

Do those look reasonable?

[dhicks]

Do those look reasonable?

 

I don't think the above benchmark is really telling me an awful lot. A sequential read/write rate of 700MB/sec for smaller files is good (up there with SSD speeds), but then we'd expect that as we're actually reading/writing the cache RAM of the RAID controller. Speeds for larger chunks of data drop to the physical speed of the underlying disks (Hitachi 4TB 7,200RPM SATA drives). I guess what I'm trying to figure out is a way to stop large files (say I copy a large video file off an SD card onto a file share or my desktop) affecting other users - someone copying a 4GB file can just wait, it shouldn't start locking out people from logging in to their desktops.

 

Is there a way to get Windows to rate-limit file transfers - any file transfers over a certain size get bandwidth throttled?

 

Is there a way to get Windows to utilise the spare RAM as a disk cache, like Linux does? If I understand correctly, this is a feature of Server 2012's Cluster Shared Volumes - time to upgrade the file server to Server 2012 (it's currently running Server 2008 R2)?

 

Can I get XenServer to do the caching instead? The disk array is passed to the Windows VM directly as a removeable device, there's no virtual disk image on there. If I'd set it up differently - assigning the disk array as a XenServer Storage Repository, creating a disk image and assigning that to the VM - would XenServer then use any spare RAM fopr caching?

Edited December 4, 2013 by dhicks

[gaz350b]

My guess would be sata And the rpm of the disks not scaling with random access to files.

[dhicks]

My guess would be sata And the rpm of the disks not scaling with random access to files.

 

As in you think there's some kind of issue with the SATA drives? Just to be clear, this array is handling requests from all users on our network, probably around 100 users at a time - it's the main storage server for user's document, desktops and shared file areas. The physical disks seem to be performing as expected - these are the same model Hitachi 4TB drives that Backblaze use in their large storage arrays, not super-fast or anything but seem to be decent enough and reliable enough over the long term. The RAID controller itself seems capable of decent throughput speeds (see that 700MB/sec figure, above), it's just limited by the physical disks, and I know if we wanted to increase raw disk performance we could use different disks. However, in a nework environment, any disk array will eventually run out of raw read/write capability if enough people are shunting large media files around, and I want to figure out how to best configure the file server to handle such an overload as gracefully as possible, preferably by limiting larger file transfers. It also strikes me that if the server has RAM available it could be using it as a cache, either at the VM level or at the hyporvisor level.

[morganw]

Maybe try passing the RAID controller through directly, instead of passing through the storage as a removeable device?

- shane cunningham | XenServer 6.2 and PCI passthrough for LSI SAS1068E

[psydii]

Cache Cache Cache. Also these updates on both clients and servers:

An enterprise hotfix rollup is available for Windows 7 SP1 and Windows Server 2008 R2 SP1 (including the two hotfixes listed at the top of the article)

List of currently available hotfixes for the File Services technologies in Windows Server 2008 and in Windows Server 2008 R2

Also Firmware and Drivers, again both on clients and servers.

 

When you say you've added the 'recommended' MS registry setting do you mean these:

 

NtfsDisable8dot3NameCreation 1

TreatHostAsStableStorage 1

AdditionalCriticalWorkerThreads 64

MaximumTunnelEntries 32

MaxThreadsPerQueue 64

RequireSecuritySignature 0

MaxMpxCt (not applicable with SMB 2 clients) 32768

 

(we've deployed them as part of general steps to resolve performance issues)

 

For us the biggest improvements to throughput (not the maximum rate, just the consistency) was delivered by taking real time av scanning out of the loop and also to increase the ram allocated to the box. Basically in our case every last spare drop of RAM on my VM host is allocated to the Fileserver.

 

You can check how your RAM is actually being used via the sysinternals tool RAMMAP. We find that over time the "metadata" allocation grows and grows reducing the amount of ram available for the caching of files. A good(ish) rule of thumb is your "mapped files" should be roughly the same size as the roaming profiles of all currently logged on users across your network.

 

 

If metadata is larger than 'mapped files' a reboot is needed. There will likely be an initial performance dip as the system builds up the cache again, however over all performance should be improved.

 

It should be noted that on busy systems (such as ones where the multiple IT classes are working in Photoshop or the Serif Suite), where you have enabled TreatHostAsStableStorage, it can take minutes for change data to actually reach the disks during which time you are at risk if the system were to unexpectedly stop. If you don't have enough cache then the corollary is 'slow performance of the fileserver' as it fills the cache and has to stalll incoming connections while it tries to write them out, and has to go to disk to service further reads.

 

Hope this helps

[dhicks]

Maybe try passing the RAID controller through directly, instead of passing through the storage as a removeable device?

 

The RAID controller is hanlding another 4-disk array acting as the XenServer's main storage, it's not available to completly dedicate to the VM. Acting as removeable storage doesn't seem to be affecting hardware performance - data transfer rates are not limited to USB 2 bus speeds or anything, the hardware is managing 700MS/sec okay. I don't think there's anything wrong with any of the underlying hardware, I'm more trying to figure out how to configure Windows Server better to provide better service to client machines. I've got a classroom of Windows 7 laptops taking a long time to log in this morning - when the file server hits around 100 active TCP connections we seem to start getting slow-downs around the network. Do I simply need faster disks, faster read/write capacity, for 100+ users all using the one file server?

[morganw]

What type of profile are people logging on with, and if you enable verbose startup/logon messages which stage of the logon is taking the most time?

How to enable verbose startup, shutdown, logon, and logoff status Messages in the Windows Server 2003 family

[PotNoodleTech]

I've got a classroom of Windows 7 laptops taking a long time to log in this morning - when the file server hits around 100 active TCP connections we seem to start getting slow-downs around the network. Do I simply need faster disks, faster read/write capacity, for 100+ users all using the one file server?

 

Actually - I'm quite imressed with the speeds you are getting - I've seen 4 disk SATA arrays that have stuggled to get over 60MB/s sequential write speed. Kudos to you buying a decent Raid card I am sure!

 

The quick answer is yes, exactly! You need:

Faster Disks - SAS 10k or 15k for "enterprise load" (really, SATA disks are designed to handle 1 user, not 100)

More Disks - The more disks you have the more you can leverage the benefits of whichever RAID type you are using.

 

Here's mine:

 

 

This is a totally different style setup though - Dell Perc H700 Raid Card with 512MB Cache (as you can see from the "cache" benchies, it's a beast!) and 16x146GB 2.5" 10K SAS Drives set up in Raid 50. Only provides 1.5TB storage.

 

Interestingly this has highlighted an issue I have - my sequential Read speed seems to have dropped - it used to be ~400mb/s when I set up the server a couple of years ago. I'll have to look into that!!!

 

The sequential is the headline benchmark that everyone looks at. If you look at the, 512k, 4k and 4kQD32 results these simurate reading random data, random data is what a fileserver is going to be used for so these benchmarks are the most important to look at. Enterprise SAS drives really stretch their legs here due to faster spindle speed, lower access times.

Edited December 5, 2013 by PotNoodleTech

[dhicks]

Also these updates on both clients and servers:

An enterprise hotfix rollup is available for Windows 7 SP1 and Windows Server 2008 R2 SP1 (including the two hotfixes listed at the top of the article)

List of currently available hotfixes for the File Services technologies in Windows Server 2008 and in Windows Server 2008 R2

 

Right, I'll take a look at those.

 

Also Firmware and Drivers, again both on clients and servers.

 

XenServer 6.2 has some updates to apply, I'll take a look at those. I've just installed XenServer Tools on the VM, which updates the NIC driver (I'd assume it upadtes the harddrive driver, too).

 

When you say you've added the 'recommended' MS registry setting do you mean these:

 

Yes, those are the ones.

 

For us the biggest improvements to throughput (not the maximum rate, just the consistency) was delivered by taking real time av scanning out of the loop and also to increase the ram allocated to the box. Basically in our case every last spare drop of RAM on my VM host is allocated to the Fileserver.

 

That's rather what I figured, too - there's no real-time AV checking, and there VM is assigned 8GB of RAM. It doesn't seem to be being used, though - both Windows Resource Monitor inside the VM and the XenServer resource usage readout within XenCenter reckon the VM is using around 2GB of RAM.

 

You can check how your RAM is actually being used via the sysinternals tool RAMMAP. We find that over time the "metadata" allocation grows and grows reducing the amount of ram available for the caching of files. A good(ish) rule of thumb is your "mapped files" should be roughly the same size as the roaming profiles of all currently logged on users across your network.

 

Ah, very useful, thanks. We seem to have a "mapped file" value of around 6GB, of which most (95%) is listed under "standby". Is there some way I cn chivvy Windows into pre-loading more stuff in the mapped file areas?

 

It should be noted that on busy systems (such as ones where the multiple IT classes are working in Photoshop or the Serif Suite), where you have enabled TreatHostAsStableStorage, it can take minutes for change data to actually reach the disks during which time you are at risk if the system were to unexpectedly stop.

 

Understood - we have a large, decent UPS, that should cover it.

 

If you don't have enough cache then the corollary is 'slow performance of the fileserver' as it fills the cache and has to stalll incoming connections while it tries to write them out, and has to go to disk to service further reads.

 

From the benchmarks I did yesterday, I'm thinking our RAID controller's cache is performing okay, we could possibly just do with a bit more raw speed from the disks themselves.

[dhicks]

What type of profile are people logging on with

 

Good question - all user Documents and Desktops are now redirected folders, which I assumed meant they are not read until neccesary. The rest of the profile should be created locally on each machine, but I guess I need to double-check Active Directory.

[morganw]

You will get strange hangs if using redirected folders, building a new profile during logon (e.g. first logon on a machine), and you are also blocking desktop.ini files on the server. Otherwise if you aren't doing anything like offline files, the fileserver's involvement during logon should be minimal. Are you using Group Policy Preferences for anything?

[psydii]

If you haven't deployed the the Hotfixes, then you should find they make a general overall improvement to the consistency of the performance, especially when the file server is busy.

 

On memory matters.... Windows loads into RAM files from disk as they are accessed, with a bit of opportunistic read ahead pre-loading for good measure. If a file is changed (in memory) the page is marked as dirty (in use) and cannot be re-used for anything other than its current purpose until it is written back out to disk. Once the page is no longer dirty the memory manager marks it as available, however the memory manager knows that it still contains the file contents until such time as it is re-used for something else. If a call is made for the file again, the memory manger serves it from ram (a soft fault) rather than paging it in from disk (a hard fault) again. Thus in a system with sufficient large memory 'available' and 'cached' are approximately equal because they are almost synonymous.

 

Watch this for a deep dive: Mysteries of Memory Management Revealed,with Mark Russinovich (Part 1 of 2) | Tech·Ed North America 2011 | Channel 9

 

 

 

p.

[psydii]

You will get strange hangs if using redirected folders, building a new profile during logon (e.g. first logon on a machine), and you are also blocking desktop.ini files on the server. Otherwise if you aren't doing anything like offline files, the fileserver's involvement during logon should be minimal. Are you using Group Policy Preferences for anything?

 

By default Windows 7 caches locally redirected folders. This would cause massive amounts of data transfer at logon, and impact performance.

[dhicks]

By default Windows 7 caches locally redirected folders.

 

Hang on, so you mean that if a user's Documents folder has been redirected, Windows 7 sits there and downloads it to the local harddrive on logon on a client machine anyway?

[morganw]

Yes, unless you've stopped it.

Configure offline files settings for redirected folders in Windows 7

[psydii]

Hang on, so you mean that if a user's Documents folder has been redirected, Windows 7 sits there and downloads it to the local harddrive on logon on a client machine anyway?

 

Yes, but it does it in the background so as not to block the logon. A great feature if you have a 1:1 environment, less great if you are hot-desking users 6 times a day. We turn it off for student devices.

[dhicks]

Ah, very useful, thanks. We seem to have a "mapped file" value of around 6GB, of which most (95%) is listed under "standby".

 

Right, I figured out the actual issue in the end - the "mapped files" list seemed to be listing every file in one of our shared areas, as if they were being constantly accessed. That was odd, as you'd think there'd just be the occasional file access, not constant. It turned out that a typo in the backup script (un-escaped spaces in share names) was causing rsync to copy the whole shared file area each time as a new backup, not simply sync it with an existing folder, hence all the read activity on the main file server. That problem is hopefully solved now, and we should be getting considerably better read performance from our file server. That's quite interesting, as the network was just-about usable with the backup server hammering away reading the whole time, so hopefully it turns out that the raw read speed of our harddrives and the amount of cache provided by both our RAID card and Windows should be enough to provide quite decent performance for 130-odd users at a time.