+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 25
Windows Server 2008 R2 Thread, Windows Server 2008 R2 File Server Performance in Technical; Hello All, I seem to be having an issue getting the best performance out of our Windows Server 2008 R2 ...
  1. #1

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,621
    Thank Post
    1,239
    Thanked 776 Times in 673 Posts
    Rep Power
    235

    Windows Server 2008 R2 File Server Performance

    Hello All,

    I seem to be having an issue getting the best performance out of our Windows Server 2008 R2 file server. This is running as a VM under XenServer 6.2, and is currently assigned 4 virtual cores, 8GB of RAM and its own dedicated disk array passed through as a removeable device from XenServer. This disk array consists of 4 4TB drives on a hardware RAID controller running as a RAID 5 array. It has a virtual network interface assigned to it consisting of 3 physical gigabit NICs bonded together. All things considered, it should have plenty of resources available, and from what I can judge from Windows resource monitor the hardware isn't overly-taxed - CPU isn't going much above 10%, RAM usage around 2GB, network utilisation around 500Mb/s. If I've read the documentation right, a disk queue length of anything less than number of spindles + 2 for a RAID array is okay, and our disk queue length seems to hover around the "1" mark - plenty of capacity to spare, there. I've recently made sure the XenServer tools are installed on the VM, which changed the virtual NIC's driver and made it gigabit-capable.

    This file server is now handling all shared file areas as well as all users' My Documents and Desktop folders and, mostly, seems to be running okay. However, we still seem to be having performance issues in some particular cases - by the looks of it, when there are simply lots of people using the system at a time. During busy times (i.e. when a class all logs on at the same time), some users will seem to be "locked out" during the login process, as if the file server is simply too busy to process their request right at the moment and is queing requests.

    The only readout that I an see on the performance monitor that seems to be constantly at the top of its limit is the "TCP Connections" readout - it seems to hover near its "100" top limit. Are we running out of available TCP connections to the file server? Is this even a limitable figure? I get that the graph auto-scales to fit the figure that it's displaying, but it rather suspiciously never seems to go above 100, even though it seems to hit 100 at times. Is something limiting the number of TCP connections? If so, is that a limit that can be set in Windows, or is that a limit set from the XenServer side of things? I've investigated settings for Windows file servers and added Microsoft's suggested registry settings, however no mention seems to be made of a TCP connection limit. Is that limit actually a result of something else being limited - number of threads or similar?

    Are there some diagnostic / benchmark tools I can run on the file server to judge how well it is performing overall? Will benchmarking to get an IOPS figure of some kind be any help, or will another benchmark be better in this case?

  2. #2
    AButters's Avatar
    Join Date
    Feb 2012
    Location
    Wales
    Posts
    462
    Thank Post
    139
    Thanked 107 Times in 82 Posts
    Rep Power
    41
    Does the hardware Raid controller have battery backup and cache (usually look like RAM sticks). Without Battery backup to enable write caching, and the RAM cache, RAID 5 on SATA drives is going to be slow, especially when writing data.

    A great little program for identifying disk speed problems is "CrystalDiskMark" Pop this on your VM and do two tests for me.

    1. "All" 100MB Test with 3 Test runs. This will identify if you have raid controller cache or not.

    2. "All" 1000MB Test with 3 Test runs. This wil give a general performance indicator.

    Post your screenies for us to have a gander at.

  3. Thanks to AButters from:

    dhicks (4th December 2013)

  4. #3

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,621
    Thank Post
    1,239
    Thanked 776 Times in 673 Posts
    Rep Power
    235
    Quote Originally Posted by AButters View Post
    Does the hardware Raid controller have battery backup and cache (usually look like RAM sticks).
    Yes, the RAID card is an LSI 9271-8i with the battery backup module attached and 1GB of cache RAM. I've ensured the array settings in the RAID card's BIOS have caching turned on and that the cache mode is set to "always write-back" to ensure the cache controller doesn't decide to switch back to write-through caching on a reboot or something. And with an average disk queue length of 0.5 on a 4-spindle array, physical disk performance looks pretty good, so I don't think it's a cache / controller problem.

    Interestingly, the server's TCP Connections graph just broke the 100 limit - now up to 130 TCP conecctions, so it turns out that 100 wasn't a hard limit after all. Disk queue length seemed to go up to around 5 as a whole class logged on this morning, with the ICT teacher saying that some operations seemed "a bit slow". It seems that some users get "locked out" fora few minutes when logging on, as if the server is simply not able to process their requests just yet. All pupils' My Documents and Desktop folders are redirected to the file server, so on logon I guess it's having to do a fair few read / directory list operations, although I shouldn't have thought there'd be that much whole-file reading or writing neccesary.

    A great little program for identifying disk speed problems is "CrystalDiskMark" Pop this on your VM and do two tests for me.
    1. "All" 100MB Test with 3 Test runs. This will identify if you have raid controller cache or not.
    2. "All" 1000MB Test with 3 Test runs. This wil give a general performance indicator.
    Good plan, I'll give that a try - I might have to wait until this evening to run test on a live server, though.

  5. #4
    AButters's Avatar
    Join Date
    Feb 2012
    Location
    Wales
    Posts
    462
    Thank Post
    139
    Thanked 107 Times in 82 Posts
    Rep Power
    41
    Quote Originally Posted by dhicks View Post
    Yes, the RAID card is an LSI 9271-8i with the battery backup module attached and 1GB of cache RAM. I've ensured the array settings in the RAID card's BIOS have caching turned on and that the cache mode is set to "always write-back" to ensure the cache controller doesn't decide to switch back to write-through caching on a reboot or something. And with an average disk queue length of 0.5 on a 4-spindle array, physical disk performance looks pretty good, so I don't think it's a cache / controller problem.
    Ok that all sounds good.

    Quote Originally Posted by dhicks View Post
    Interestingly, the server's TCP Connections graph just broke the 100 limit - now up to 130 TCP conecctions, so it turns out that 100 wasn't a hard limit after all. Disk queue length seemed to go up to around 5 as a whole class logged on this morning, with the ICT teacher saying that some operations seemed "a bit slow". It seems that some users get "locked out" fora few minutes when logging on, as if the server is simply not able to process their requests just yet. All pupils' My Documents and Desktop folders are redirected to the file server, so on logon I guess it's having to do a fair few read / directory list operations, although I shouldn't have thought there'd be that much whole-file reading or writing neccesary.
    Yeh we're running at 190 connections here so there definatly should be no limit on that (2003 server on Vmware tho so a bit different to yours). We use redirected docs desktop and start menu and roaming profiles so no shortage of operations needed.

    Quote Originally Posted by dhicks View Post
    Good plan, I'll give that a try - I might have to wait until this evening to run test on a live server, though.
    Yes if you're getting disconnects under some sort of load then don't run the benchmark "in hours". It's unusual to get "locked out" when busy as you suggest, I've only ever seen things slow down here, not stop.

  6. Thanks to AButters from:

    dhicks (4th December 2013)

  7. #5

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,621
    Thank Post
    1,239
    Thanked 776 Times in 673 Posts
    Rep Power
    235
    Quote Originally Posted by AButters View Post
    Yeh we're running at 190 connections here so there definatly should be no limit on that (2003 server on Vmware tho so a bit different to yours). We use redirected docs desktop and start menu and roaming profiles so no shortage of operations needed.
    Load-wise you sound not too dissimilar to us - what kind of disk queue length, disc performance and network peformance are you seeing? Our disk queue length seems to average around 1, disc performance is around 70MB/sec and network around 500Mb/sec.

    CrystalDiskInfo doesn't seem to find any disks on our VM - probably something to do with the drivers. I'll investigate further, and probably try IOMeter.

    It's unusual to get "locked out" when busy as you suggest, I've only ever seen things slow down here, not stop.
    I'm starting to think it's maybe a client configuration issue. The laptops in the ICT room seem to take a while to do some file operations, like when you go to "File, Open..." in a program there'll be a pause of a minute or so until the machine responds, rather as if the machine thinks it's searching an old path or something. No actual complaints from the reast of the school, so maybe it's time to reimage the ICT room and see if that solves the problem. I'd still like to make sure our file server is giving top performance, though.

  8. #6
    AButters's Avatar
    Join Date
    Feb 2012
    Location
    Wales
    Posts
    462
    Thank Post
    139
    Thanked 107 Times in 82 Posts
    Rep Power
    41
    Is it in MS office you're having the problem? Have problems with Office 2003 and 2010 sometimes hanging when using the "File..Open" box and using the drop down box at the top to navigate - if you click on the left hand options "Documents" or "Computer" or whatever it works a treat.

    That is an issue with MS Office trying to browse the network neighbourhood which you can't seem to disable on Windows 7. (didnt happen on XP).

  9. Thanks to AButters from:

    dhicks (4th December 2013)

  10. #7

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,621
    Thank Post
    1,239
    Thanked 776 Times in 673 Posts
    Rep Power
    235
    Quote Originally Posted by AButters View Post
    Have problems with Office 2003 and 2010 sometimes hanging when using the "File..Open" box and using the drop down box at the top to navigate - if you click on the left hand options "Documents" or "Computer" or whatever it works a treat.
    I got the impression there's simply a delay between the user hitting "File, Open..." and the "open" dialog actually appearing. I'm going to try reimaging the room's worth of machines - I've been moving file areas around quite a bit recently, so the machines could well just be confused.

  11. #8

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,621
    Thank Post
    1,239
    Thanked 776 Times in 673 Posts
    Rep Power
    235
    Quote Originally Posted by dhicks View Post
    CrystalDiskInfo doesn't seem to find any disks on our VM
    Right, sorry, am being a wolly - completly different program. After running CrystalDiskMark, I have the following results:

    benchmarks.png

    Do those look reasonable?

  12. #9

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,621
    Thank Post
    1,239
    Thanked 776 Times in 673 Posts
    Rep Power
    235
    Quote Originally Posted by dhicks View Post
    Do those look reasonable?
    <Thinks for a bit, reads Google for a while> I don't think the above benchmark is really telling me an awful lot. A sequential read/write rate of 700MB/sec for smaller files is good (up there with SSD speeds), but then we'd expect that as we're actually reading/writing the cache RAM of the RAID controller. Speeds for larger chunks of data drop to the physical speed of the underlying disks (Hitachi 4TB 7,200RPM SATA drives). I guess what I'm trying to figure out is a way to stop large files (say I copy a large video file off an SD card onto a file share or my desktop) affecting other users - someone copying a 4GB file can just wait, it shouldn't start locking out people from logging in to their desktops.

    Is there a way to get Windows to rate-limit file transfers - any file transfers over a certain size get bandwidth throttled?

    Is there a way to get Windows to utilise the spare RAM as a disk cache, like Linux does? If I understand correctly, this is a feature of Server 2012's Cluster Shared Volumes - time to upgrade the file server to Server 2012 (it's currently running Server 2008 R2)?

    Can I get XenServer to do the caching instead? The disk array is passed to the Windows VM directly as a removeable device, there's no virtual disk image on there. If I'd set it up differently - assigning the disk array as a XenServer Storage Repository, creating a disk image and assigning that to the VM - would XenServer then use any spare RAM fopr caching?
    Last edited by dhicks; 4th December 2013 at 11:10 PM.

  13. #10

    Join Date
    Jun 2012
    Posts
    47
    Thank Post
    3
    Thanked 5 Times in 5 Posts
    Rep Power
    5
    My guess would be sata And the rpm of the disks not scaling with random access to files.

  14. Thanks to gaz350b from:

    dhicks (5th December 2013)

  15. #11

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,621
    Thank Post
    1,239
    Thanked 776 Times in 673 Posts
    Rep Power
    235
    Quote Originally Posted by gaz350b View Post
    My guess would be sata And the rpm of the disks not scaling with random access to files.
    As in you think there's some kind of issue with the SATA drives? Just to be clear, this array is handling requests from all users on our network, probably around 100 users at a time - it's the main storage server for user's document, desktops and shared file areas. The physical disks seem to be performing as expected - these are the same model Hitachi 4TB drives that Backblaze use in their large storage arrays, not super-fast or anything but seem to be decent enough and reliable enough over the long term. The RAID controller itself seems capable of decent throughput speeds (see that 700MB/sec figure, above), it's just limited by the physical disks, and I know if we wanted to increase raw disk performance we could use different disks. However, in a nework environment, any disk array will eventually run out of raw read/write capability if enough people are shunting large media files around, and I want to figure out how to best configure the file server to handle such an overload as gracefully as possible, preferably by limiting larger file transfers. It also strikes me that if the server has RAM available it could be using it as a cache, either at the VM level or at the hyporvisor level.

  16. #12
    morganw's Avatar
    Join Date
    Apr 2009
    Location
    Cambridge
    Posts
    816
    Thank Post
    46
    Thanked 132 Times in 126 Posts
    Rep Power
    39
    Maybe try passing the RAID controller through directly, instead of passing through the storage as a removeable device?
    - shane cunningham | XenServer 6.2 and PCI passthrough for LSI SAS1068E

  17. Thanks to morganw from:

    dhicks (5th December 2013)

  18. #13

    Join Date
    Jul 2006
    Location
    London
    Posts
    1,254
    Thank Post
    111
    Thanked 242 Times in 193 Posts
    Blog Entries
    1
    Rep Power
    74
    Cache Cache Cache. Also these updates on both clients and servers:
    An enterprise hotfix rollup is available for Windows 7 SP1 and Windows Server 2008 R2 SP1 (including the two hotfixes listed at the top of the article)
    List of currently available hotfixes for the File Services technologies in Windows Server 2008 and in Windows Server 2008 R2
    Also Firmware and Drivers, again both on clients and servers.

    When you say you've added the 'recommended' MS registry setting do you mean these:

    NtfsDisable8dot3NameCreation 1
    TreatHostAsStableStorage 1
    AdditionalCriticalWorkerThreads 64
    MaximumTunnelEntries 32
    MaxThreadsPerQueue 64
    RequireSecuritySignature 0
    MaxMpxCt (not applicable with SMB 2 clients) 32768

    (we've deployed them as part of general steps to resolve performance issues)

    For us the biggest improvements to throughput (not the maximum rate, just the consistency) was delivered by taking real time av scanning out of the loop and also to increase the ram allocated to the box. Basically in our case every last spare drop of RAM on my VM host is allocated to the Fileserver.

    You can check how your RAM is actually being used via the sysinternals tool RAMMAP. We find that over time the "metadata" allocation grows and grows reducing the amount of ram available for the caching of files. A good(ish) rule of thumb is your "mapped files" should be roughly the same size as the roaming profiles of all currently logged on users across your network.

    rammap too much metadata.png

    If metadata is larger than 'mapped files' a reboot is needed. There will likely be an initial performance dip as the system builds up the cache again, however over all performance should be improved.

    It should be noted that on busy systems (such as ones where the multiple IT classes are working in Photoshop or the Serif Suite), where you have enabled TreatHostAsStableStorage, it can take minutes for change data to actually reach the disks during which time you are at risk if the system were to unexpectedly stop. If you don't have enough cache then the corollary is 'slow performance of the fileserver' as it fills the cache and has to stalll incoming connections while it tries to write them out, and has to go to disk to service further reads.

    Hope this helps

  19. 2 Thanks to psydii:

    AButters (5th December 2013), dhicks (5th December 2013)

  20. #14

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,621
    Thank Post
    1,239
    Thanked 776 Times in 673 Posts
    Rep Power
    235
    Quote Originally Posted by morganw View Post
    Maybe try passing the RAID controller through directly, instead of passing through the storage as a removeable device?
    The RAID controller is hanlding another 4-disk array acting as the XenServer's main storage, it's not available to completly dedicate to the VM. Acting as removeable storage doesn't seem to be affecting hardware performance - data transfer rates are not limited to USB 2 bus speeds or anything, the hardware is managing 700MS/sec okay. I don't think there's anything wrong with any of the underlying hardware, I'm more trying to figure out how to configure Windows Server better to provide better service to client machines. I've got a classroom of Windows 7 laptops taking a long time to log in this morning - when the file server hits around 100 active TCP connections we seem to start getting slow-downs around the network. Do I simply need faster disks, faster read/write capacity, for 100+ users all using the one file server?

  21. #15
    morganw's Avatar
    Join Date
    Apr 2009
    Location
    Cambridge
    Posts
    816
    Thank Post
    46
    Thanked 132 Times in 126 Posts
    Rep Power
    39
    What type of profile are people logging on with, and if you enable verbose startup/logon messages which stage of the logon is taking the most time?
    How to enable verbose startup, shutdown, logon, and logoff status Messages in the Windows Server 2003 family

  22. Thanks to morganw from:

    dhicks (5th December 2013)

SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. New Server 2008 R2 NAS Server / File Server Advice
    By burgemaster in forum Windows Server 2008 R2
    Replies: 0
    Last Post: 2nd September 2013, 03:00 PM
  2. Replies: 47
    Last Post: 30th July 2012, 10:03 AM
  3. Minimum specs for server 2008 R2? File share/Print/DHCP server.
    By dany2010 in forum Windows Server 2008 R2
    Replies: 22
    Last Post: 6th January 2012, 10:12 AM
  4. [Windows Software] Server 2008 r2 ISA Server
    By denon101 in forum Licensing Questions
    Replies: 5
    Last Post: 25th January 2010, 12:03 PM
  5. Linux NAS or Server 2003 R2 File server
    By jonathan.lees in forum *nix
    Replies: 8
    Last Post: 27th March 2008, 01:09 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •