LanmanServer (server) Service Hanging

[Stuajnht]

Hi all,

 

We've been having a problem with one of our servers over the past week. The LanmanServer service keeps hanging, and once it does it can't be restarted through the services MMC or using taskkill without force restarting the whole server (normal shutdown / restart just wait on the 'Restarting' screen). When it hangs, file shares become unresponsive and users are not able to log on or access files. There is nothing obvious logged in the event logs leading up to the problem, and it happens at random times during the day (it's happened twice within an hour and a half today, and not just when students are logging on for lessons). The server is fully responsive if you log into the console via Hyper-V manager (through RDP does not work) and there are no spikes or high usage logged in task manager or resource montor. I have tried to log it through Process Monitor, but it crashes out and doesn't save the backing files.

 

Over the weekend this problem was not happening (as in I could browse file shares throughout the weekend, but there weren't any other users on the network) and all of our servers were restarted just in case there was something funny going on between them.

 

The server in question is a Windows Server 2012 Hyper-V client, running AD, DNS and user home file shares, and the problem started last Tuesday. I have only been able to find one other question like this on the Internet, located here Windows Server Enterprise server service flakes out - Stack Overflow

 

Members of our SMT are now asking what is causing it and for it to be resolved, but so far all that definitely fixes the problem for a temporary measure is force restarting the server which obviously isn't ideal.

 

Any help or advice will be appreciated. Please ask if you need more information or clarifications.

Edited October 12, 2015 by Stuajnht

[rjs_essex]

Odd that I should stumble on this thread after you have only just recently posted it. I am having a very similar problem that I am currently investigating. I have had one Hyper-V VM display the same behaviour in recent weeks, however, it is running Windows Server 2008 R2. I experience exactly the same problem as you, if I try to restart the server service it hangs in a stopping state. If I try and restart the server it crashes at 'Shutting Down'. I then have to force it off.

 

Today, I now have two other servers that are displaying the same behaviour. One is a physical machine, the other is a VM hosted on a different Hyper-V host. As with the first, these do the same thing. There seems to be no correlation with any events and the event logs on all three servers are completely clear. There are no errors an no indications as to what is causing the problem. One system is an applications share server, the other two are domain controllers, print and file servers.

 

I have force restarted one of the servers today and on restart I am able to restart the server service without any problem. It's not until a long period of uptime that the system them runs into problems.

 

Searches have yielded little results due to the generic nature of 'server service', however, I've had a little more success with searches for the lanmanserver service, yet still have no solutions...

 

I have run protected file system scans, trawled event logs and run various tests on the systems but there are no apparent reasons for the behaviour.

 

On service restart I get the generic .net framework related 'Error 1053: The service did not respond to the start or control request in a timely fasion.' error which is little help.

 

Could this be nic, network or switch related?

 

Have you hit on anything as yet?

[Stuajnht]

I believe I have found out what was causing this for our server (not sure if it'll help you @rjs_essex but it's worth a shot) but for the past few days it's been stable as it should be.

 

After the server crashed again, I believed that it had to be due to some files being accessed. I came to this conclusion as 1) it didn't crash over the weekend when there wasn't anybody accessing any files and 2) it was a service that looked after files and shares that was crashing.

 

To start off with I looked for ways to check that all share permissions were set correctly. I came across Recursively obtain a list of all permissions on Windows file shares - Server Fault which gave a few tools to use, along with Cjwdev | NTFS Permissions Reporter. However, running these on my local computer took a while, and the amount of data that was produced was too much to be able to used at all, so even trying to run this on a file server was already a no-go.

 

My next idea came from thinking that it had to be a particular file or set of files that were causing it to crash. Therefore I looked into logging file access through the built in Windows file auditing tools and the results being logged through event viewer (here's one such guide: Audit File Access and Change in Windows | Splunk Blogs, more resources can be found on https://www.google.co.uk/search?q=windows+server+log+files+accessed) but I couldn't seem to get it to log properly in event viewer.

 

I decided then to use Everything as I have used it in the past to perform quick searches of files, and it includes the dates files were last modified. You can download the portable version and run that. Let it build its index, which if it's a file server will take a bit of time. I'd advise you that "Everything" should be closed and opened again, as this saves it having to rebuilt the index if you need to force power off the server. If you've been keeping a log of the approximate times the server has been going down, then you can run a search in "Everything" for a particular date (use dm:, see: Searching), then sort the columns by date modified, which will take a long time... so grab a coffee or something. When the list is sorted, scroll down to find the time that the server stopped responding (there should be a jump of a few minutes between the last file accessed and the next file while the server rebooted, whereas all files preceding the crash will be accessed many times per minute). Once you have found this, make a note of the last 20 files or so accessed in an Excel document before the server stopped working, and then perform the search again but for the next date and time it crashed at. Repeat as many times as needed.

 

Hopefully by this point you should have a table with a few files that appear over the course of the crashes, and some that are accessed pretty much every time the server goes down. As the server going down for me holds the users home directories, I just put their usernames in, such as the example below:

 

[TABLE=class: grid, width: 500]

[TR]

[TD]User[/TD]

[TD]Crash 1[/TD]

[TD]Crash 2[/TD]

[TD]Crash 3[/TD]

[TD]Crash 4[/TD]

[TD]Crash n[/TD]

[/TR]

[TR]

[TD]j.smith[/TD]

[TD]y[/TD]

[TD]y[/TD]

[TD][/TD]

[TD][/TD]

[TD][/TD]

[/TR]

[TR]

[TD]b.hope[/TD]

[TD][/TD]

[TD]y[/TD]

[TD][/TD]

[TD]y[/TD]

[TD][/TD]

[/TR]

[TR]

[TD]a.name[/TD]

[TD][/TD]

[TD]y[/TD]

[TD]y[/TD]

[TD]y[/TD]

[TD]y[/TD]

[/TR]

[TR]

[TD]u.getit[/TD]

[TD]y[/TD]

[TD][/TD]

[TD]y[/TD]

[TD][/TD]

[TD][/TD]

[/TR]

[/TABLE]

 

This should give you a fair idea on what files or users may be causing the problem. In this example, "a.name" is looking like a possible cause.

 

You now need to check to see if this detective work is correct or not. Run "Everything" again, without any search parameters but sorted by the date modified column. This will list the most recent file access at the top of the screen, with it automatically moving the older accessed files further down the page. Keep this running until there is a crash with the server again. If you are RDP'd into the server, take a screenshot to check later after the server is rebooted, otherwise make a note of the last file accessed, then reboot the server. This should correspond to one of the files that have been flagged up in the table you made earlier. If it is, browse to the offending file (or folder if there's many files) and move it off the server to somewhere it can't be accessed, and ideally let the user(s) who's file it is know that you have moved it.

 

Keep monitoring the server through "Everything" to make sure that it's still working as it should be, and that there are no more files causing it to stop working properly. If there are, move them off as needed. Hopefully the server should then be fine and running as normal.

 

For the record, our server crashes were caused by Firefox Portable that a sixth former student had downloaded, but I do not think this would be the problem, just that there was something fishy going on with the files. Cross checking their logon times with their timetable, every time they had a free period they would open Firefox, and the server would go down.

[rjs_essex]

That's some good investigation work there! Thanks for that information, I will certainly look into some of these suggestions. Like you say, the issue I have only occurs on file servers that host multiple file shares and directories. All of which have varying characteristics.

 

Using the techniques above I will need to wait until one of the servers in question crashes again. The annoying thing here is that I have no idea when that will be. It is not all that frequent, although there does seem to be an emerging pattern as it always seems to happen on a Thursday at around midday...

 

I can force the server to crash simply by restarting the Server/Lanman service, but this doesn't really give me any indications other than that the problem still exists and will continue to happen as and when the faulting application/file/process occurs at some point in the future.

 

Beyond this, I have checked all of the file shares, permissions, protected system files and nothing else is showing itself (yet) as a contributing factor.

 

I'll hopefully be able to update this thread when I have more information!

[rjs_essex]

I've been running 'Everything' for a couple of days but no server hangs as yet so have been researching in other directions. After searching for every conceivable thread or article on the Internet related to this issue, clutching at straws begins and potentially fixes the problem!

 

Came across a very old thread on the Internet whereby someone else had a similar issue with the Server service. After reading through the thread (which I think I had stumbled on before and dismissed as nonsense :-/ ) decided to give it a try.

 

Their suggestion was to change the 'Log on as' account for the Server service from Local System account (the default), to a domain administrator account, accept the change and restart the service (which for me on one out of three systems crashed the o/s). Then change it back again to Local System Account.

 

I have noticed that this invokes a key change. The Dependencies list for the 'Server Lanmanserver' service changes. It adds in Computer Browser, which as far as I am aware, is disabled and not running IF network discovery is switched off. Maybe someone else could confirm this? And or how their environment is configured by default?

 

So, I changed the Computer Browser service startup type to Automatic and now when I restart the Server service, all dependencies restart and the Server service starts correctly as expected!

 

Conclusion? A misconfiguration in the registry *somewhere* due to an update/upgrade/previous change that has caused the Log On as credentials strings to become corrupt in some way. Or a similar upgrade/update/previous change that has affected the dependencies of the Server service which in turn has affected its ability to restart/start after a certain number of shares/file share sessions are created.