lmgtfy Posted January 13, 2015 Posted January 13, 2015 Hi all, Had the morning from hell, came in happy and instantly met with a barrage of phone calls and staff banging at the door. Basically last night during backups (we use Veeam) a number of our VM's switched off with various different errors. Veeam seems to have initiated (but maybe not caused) the problems. I have rung Veeam sent them all the logs and they are looking into it. One thing I have noticed on our 4 hosts is some kind of a memory leak. They are all using well in excess of the used RAM as committed. See the screenshot for what I mean. We should have plenty of room to move machines around on the cluster but when I tried to start a VM with 20GB of RAM it said not enough available resources to start VM even though in available RAM I have 90GB odd free. Does anyone have any ideas before I go home? I am sure restarting the hosts will get rid of the problem but I can't really do that right now and I'm sure this angry bug will come back to bite me again. Cheers
lmgtfy Posted January 15, 2015 Author Posted January 15, 2015 Hi @TheRobins thanks for the reply no we are using Hyper-V. In the end I have restarted 3 of the hosts and they are back down to having committed memory levels matching exactly the in use memory. I left 1 host so I could play around and see if I could discover the memory leak but no such joy I am up for trying anything. On this one remaining host I removed all VM's stopped all services relating to hyper-v, iscsi, basically anything I could stop or kill I did and now the in use memory is 4GB and the committed is 70GB. Where on earth has 66GB of RAM floated away to? I have found nothing online which can help me I have tried running various tools such as rammap and vmmap to try and find the cause but nothing. I have thought about contacting Microsoft support but now they have got rid of technet I have no support available even though we have an EES contract. If anyone has any ideas I'd be greatful Cheers
Steve21 Posted January 15, 2015 Posted January 15, 2015 Just to query when you say memory leak is there actually one or is this just referring to the commited RAM? The commited RAM isn't what you're using, and will always be higher than the normal physical RAM In your example the 38GB real and 78GB page file is the 114GB you're using. The commit limit is your page file limit (currently so guessing 100GB) + real RAM size, = 136GB I'd more query why your page file is that high currently. Steve
TheRobins Posted January 15, 2015 Posted January 15, 2015 Im going to ditto Steve, That is page file. Thats quite large to be honest. Only time I have seen this before is with an AV configuration using Sophos on the Host which well, not good. Being as you have 128GB of RAM pagefile usage will be minimum. I am still currently running 2008R2 here and even on a host with 32GB I have seen 1 or 2 GB pagefile when I testing failover by switching the other host off. Im not sure how cost effective contact Microsoft will be. In your situation I would probably start with the basics. Stop all VM's, Leave just the host running on its own for an hour, Dont switch off any services. Just watch any memory usuage, it will naturally increase a small amount over time but hopefully not too much, and then one by one switch them on individually. But wait you cannot do that during school time unless you have remote access from home and can do it over night or a weekend. I cannot be much further help than that, If its becoming such an issue there are daily issues and servers/backups failing I would personally considering getting in consultancy support. By finding someone who's job it is to manage Hyper-V every day as a second pair of eyes I find much easier, yes its costly but at the end of the day It could save you months of issues. There doesn't seem to be much to go on. Have you had any response from VMWARE?
lmgtfy Posted January 15, 2015 Author Posted January 15, 2015 Hi Steve, Thanks for the reply. I'm presuming there is some sort of memory leak but I am only referring to the committed (commit charge) and paged pool size. I thought that the committed figure total in my 136GB is higher than the physical RAM as I understand this figure to be the RAM size plus page pool size which in my case is set to a static page pool of 12GB on the system drive. So in my example 36GB is shown as in use that makes sense as I added up the memory allocated to each VM switched on within the host(there are no dynamic memory configs on any VM servers here) and added the other apps running on the host in the processes and it came to around the figure of 36GB which is the in use memory. The free figure of 96GB in my example is all well and good but the committed (commit charge) is 114GB so I could not start another VM with 20GB of ram as it said I did not have enough memory. Yes your right the paged pool size plus the in use memory is the total committed charge but what is this paged pool figure about do you know? Cheers
TheRobins Posted January 15, 2015 Posted January 15, 2015 Memory Pools (Windows) [h=1]Memory Pools[/h] The memory manager creates the following memory pools that the system uses to allocate memory: nonpaged pool and paged pool. Both memory pools are located in the region of the address space that is reserved for the system and mapped into the virtual address space of each process. The nonpaged pool consists of virtual memory addresses that are guaranteed to reside in physical memory as long as the corresponding kernel objects are allocated. The paged pool consists of virtual memory that can be paged in and out of the system. To improve performance, systems with a single processor have three paged pools, and multiprocessor systems have five paged pools. The handles for kernel objects are stored in the paged pool, so the number of handles you can create is based on available memory. The system records the limits and current values for its nonpaged pool, paged pool, and page file usage. For more information, see Memory Performance Information.
lmgtfy Posted January 15, 2015 Author Posted January 15, 2015 Thanks for your help it got me thinking it must be a pool memory leak then. I found this page Blogs - Mark's Blog - Site Home - TechNet Blogs so I am going to download poolmon and do some digging. Cheers
lmgtfy Posted February 25, 2015 Author Posted February 25, 2015 Finally have a resolution for this if anyone is interested. Veeam were unfortunately little help and after over a month with a support call open blamed it on the Dell. I then logged a call with Dell who after a couple of days came up with this hotfix Memory leak occurs when you create or delete CSV snapshots by using a VSS hardware provider in Windows which is exactly our problem. I can't believe Veeam has never heard of this hotfix and also can't believe Microsoft isn't pushing it out via windows update for 2012 only via a hotfix. Just hoping this post will help someone as there is nothing on Google regarding this issue and it took me a long time to resolve. Cheers 3
woodham Posted February 26, 2015 Posted February 26, 2015 from the link it is being pushed out by Microsoft as a patch for 2012r2 in the normal way as per the December update or if you don't do regular update then you can apply the hotfix see December 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2
lmgtfy Posted February 26, 2015 Author Posted February 26, 2015 Thanks @woodham I was aware of this but we use Windows Server 2012 not 2012 r2 so we would have never come across this update/fix unless Dell pointed it out to us Cheers
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now