Server 2012 File Server - suddenly stops serving requests (but otherwise looks fine)

[tcgood]

powershell you can | (pipe) Select-String -pattern "what you are looking for". Like above netstat | select-string -pattern ":389".

[DJL]

Microsoft have suggested disabling SMB Leasing - it appears to have solved the problem for us.

 

You need to add the following reg key: reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters /v DisableLeasing /t REG_DWORD /d 1 /f

 

More info on leasing can be found here: What's New in SMB

 

The following will be logged in the event log when you add the reg key:

 

File leasing has been disabled for the SMB2 and SMB3 protocols. This reduces functionality and can decrease performance.

 

Registry Key: HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters

Registry Value: DisableLeasing

Default Value: 0 (or not present)

Current Value: non-zero

 

Guidance:

 

You should expect this event when disabling SMB 3 Leasing. Microsoft does not recommend disabling SMB Leasing. Once disabled, traffic from client to server may increase since metadata and data may no longer be retrieved from a local cache.

[tcgood]

the consultant we hired recommended this when he found out what was going on at the high level. You do lose all the benefits of smb 2 and 3. Let me know how long you go with out issues.

[bergmbe]

I had previously disabled smb 2/3 on our server anyway, thinking it might be related given that it seemed to fix similar problems in 2008R2. All smb connections are at 1.5. If I'm understanding correctly, leasing wasn't introduced until SMB 2 or 2.1. If I'm still having the issues, it seems (logically) like disabling leasing wouldn't necessarily help.

 

That said, I'll probably try re-enabling smb 2/3 later today or tonight and then disabling leasing. Because I'm flat out of ideas at this point.

 

ETA: Had our clients begin losing access, after 1/3rd of the company had lost access, I opted to re-enable SMB2 & 3 in powershell, then add the registry key DJL recommended prior to rebooting the server to restore access. I'll let you know if/when we have the same problem again.

 

After looking at this a second time: http://support.microsoft.com/kb/2696547 I noticed that SMB Leasing isn't listed as being disabling when you turn smb2 & 3 off, so it's possible that my previously disabling smb 2 & 3 wouldn't have also disabled leasing. I've since verified with get-smbserverconfiguration in powershell that leasing is, in fact, disabled now.

Edited March 12, 2014 by bergmbe

[DJL]

Problems resurfaced - so the registry key isn't a fix after all

[bergmbe]

Same here. Reoccurred this morning, going to see how today goes. If it changes the frequency of problems at all. In the meantime, I'll proceed with moving all of our files over to Ubuntu LTS and getting a samba fileshare up. It's not an ideal solution, but at least I won't have to reboot it every few hours.

[xenonive]

Just out of interest what antivirus are people using ? We have the issue and uninstalled Mcafee from the server and now we are down to one reboot a day. Wonder if it's a conflicting windows update/antivirus issue

[bergmbe]

You're going to laugh, but at the moment I'm rebuilding the infrastructure from the ground up so that's my excuse. We have a nearly pristine Windows Server 2012 R2 install with no antivirus currently running. Individual workstations are mostly running microsoft security essentials for now.

[Gibson335]

According to a contact who runs a support provider business, MS are desperately trying to say it has something to do with Anti-Virus...at first they insisted Sophos, but then when they were told it happens with their AV software as well, they were oddly silent...they suggested not having AV on the server for a month!!

[xenonive]

Guess rules that out. I'm going to try having windows 8.1 machines off the network for couple of days as seems like issue was after they were updated at the half term but don't think they would be the issue

[DJL]

We have no AV on our file servers at the moment. We built a new file server, clean install..just windows and we have the same problem with it. It definitely has nothing to do with 3rd party software on the server, or even other Microsoft products.

 

We use System Center Endpoint Protection 2012 on our workstations

[psydii]

Here's something that looks promising:

 

http://support.microsoft.com/kb/2928360

 

"When an SMB2 or an SMB3 session has explicit credentials, it will hold a reference to the credential object. However, currently the cleanup for the session entry does not remove the reference. Therefore, the credential object becomes leaked."

 

It is a client fix but the advice is to 'install this everywhere'.

 

Post back your experiences.

Edited March 13, 2014 by psydii

[Sagima]

I don't know if it is being helpful but I have a 2012 fileserver and I don't seem to have the problems you all seem to be having. If there are any settings you would like me to look at/compare let me know. I'll be updating it to 2012R2 tomorrow to match its DFS replication partner (and because I like R2 better than the R1 version) so be quick if you do.

[DJL]

Good luck with the upgrade - i'd recommend you create a BMR image just in case you want to go back to 2012! We didn't see the problem at all on 2012 but 2012 R2 has been falling over every 20 mins since the upgrade if left under full load.

[2wheeltinker]

Solved the problem on my end. I don't know why it should matter, but I was using an E1000 network adapter on the VM's giving me this issue, I switched them to VMXNET 3 and have not seen the issue re-occur. This was almost a daily problem and it has not happened for the 8 days that I have been running the VMXNET 3 adapters.

 

I know some of you are not using VMWare ESXI hosts, but for those that are, give this a shot!

 

ESXI 5.1

Server 2012R2

[psydii]

Hi Guys,

 

The hotfix I linked to was pointed out to me by a well known MS employee/troubleshooter. It would be great to get some feedback as to whether installing it all clients and servers makes a difference.

 

 

I note that there do not appear to have been any update rollups to 2012 since 2012R2 (i.e. 2012R2 *is* the next rollup to 2012)... so I postulate that any occurrence of the issue on 2012R2 is either due to a regression, or perhaps the presence of win8 clients on the network?

 

Thanks

P.

[Harry_Greatorex]

I have this problem, only i am running 2012 R2. I only have the issue on my XP clients! My windows 7 clients work no problem. I can access the file shares by using "\\10.0.0.1\share rather than \\filesrv1\share". I haven't installed updates recently.

[Seb1780]

Well, we seem to have a resolution at our end - three weeks and no further episodes of the issues with accessing the file server.

 

This is a transcript from MS Support, I hope it helps some of you guys too (this is as received!):-

 

RESOLUTION:

1. We Removed all third party from Server, disabled the same on Host,Bypassed NIC teaming, disabled Offloading features, Updated NIC drivers and Disabled SMB 3.

2. We Also have put the Server service in a Separate Container

Sc config lanmanserver type= own

3. Check if this Latest patch is installed Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 update rollup: February 2014

4. After observing the server for couple of days we reverted the changesdone from Step 1

5. After analyzing the Dump

The investigation reveals that we arestuck due to oplock. We have seen lot of oPlock requests queued up. The server Sends the break Request for the same but does notreceive any response. Lease Break never happens due to which all the requestafter this point get stuck in redirector and after a point it gives up. Following Registry Key will help in this case

If the issue is noticed again Try the below registry key and check if issue re-occurs

Run “Reg addHKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters /vDisableLeasing /t REG_DWORD /d 1 /f”

Reboot the SMB Server

We have multiple instances of this issuebeing reported.

There have been some issues identifiedwith oplock/ leasing. Opportunistic locks ,or oplock is a client cachingmechanism that allows SMB/SMB2 clients to dynamically decide the client-side bufferingstrategy so the network traffic can be minimized. For example, aclient can buffer data for writing locally to reduce networkpackets if the client is notified there are no other processes accessing data. The same buffering can be done on reading , wheredata from a remote file is read ahead to reduce the round trips between theclient and the server if the client knows that there are no otherprocesses writing data to the file.

In SMB2.1, which is the new SMB2 dialectimplemented in Windows 7, Lease is introduced as a new type of clientcaching mechanism. It offers more flexibility for controllingclient caching and thus results in significant performance improvement in highlatency network[2]. Actually, the newly added lease types correspond tothe new oplock types introduced in Windows 7 file system [1]. SMB2.1 just gives it a different name to distinguish it from the existingoplock functionality.

After an oplock is granted to a client,if the server receives another operation that conflicts with the currentopLock held by the client, an oplock break notification will be sent to theclient. The client should send an acknowledgement for the oplockbreak to the server before the server can proceed. For example, ifa client holds a Level II oplock and another client opens the same file on theserver for writing, the level II oplock must be broken and the client cachinghas to be disabled. As there is no default value for the client oplock. Withtime the number of threads for Opplock increase in server Stack .Thiseventually causes the server to stop responding to SMB connections.

Client caching features: Oplock vs.Lease

http://blogs.msdn.com/b/openspecification/archive/2009/05/22/client-caching-features-oplock-vs-lease.aspx

SMB Opportunistic Locking Behavior

http://blogs.technet.com/b/netmon/archive/2009/09/22/smb-opportunistic-locking-behavior.aspx

[DJL]

Seb - thanks for that.

 

Can you just confirm that step 1 was reverted? In which case the only change is putting the Server service in a Separate Container using Sc config lanmanserver type=own?

 

Did you add the leasing registry key?

[Seb1780]

Can you just confirm that step 1 was reverted?

 

Yes, as per item 4, the changes in step 1 were all reverted.

 

We have added the Reg Key

[DJL]

Thanks,

 

I've tried implementing the reg key and step 2 and 3 but the problem still persists for us.

[laserblazer]

Happened here for the first time in a few weeks. I tried restarting the server service rather than rebooting. It wouldn't let me. I got a very slow progress bar when it tried stopping the server service and then a message to say it couldn't be stopped. I had to reboot because the everyone was in panic mode. I couldn't find anything relevant in the logs.

[psydii]

I note that the MS case notes imply that client behaviour causes resource exhaustion on the server... This does somewhat suggest that the client side hotfix linked above may we'll deliver the solution.

 

It would be great to get feedback.

[TheCrust]

Found that in my case, having the PDC and Infrastructure FSMO roles on the server definitely made it worse.

 

- For the past three weeks I have been running this "problematic" 2012 R2 DC (which doubles as the staff file server) as just a DC - no roles, no global catalogue and, importantly, no errors or misbehaviour

- This morning I rehomed the PDC and Infrastructure roles back on to it. Immediately I started to suffer the same problems I had been with failure to serve files.

- Re-rehomed the FSMO roles onto a new 2012 R2 DC I had brought online to replace one of our older 2008 R2 DCs, problem vanished as immediately as it had returned.

 

I've got a call open with my own support company and have been providing MPS reports and network traces on a regular basis. They noticed that this morning after putting the PDC and Infrastructure roles back onto the problem server that the number of errors and reconnects went up dramatically. Remove the roles, and it drops down to normal "background" levels.

 

Wouldn't say we are at the same point as Seb, but we're moving forwards with it. At the moment it seems I only have to remember not to put any FSMO roles on the box!

Edited March 20, 2014 by TheCrust
Additional detail

[synaesthesia]

Watching this thread closely - have 2 x 2012 R2 file servers and 2 2012 R2 DCs. We've been running all since February without a hitch so far, hopefully staying that way.

 

Setup:

 

DFS-R in use on file shares, DeDuplication enabled. All are guests on ESXi 5.5 hosts on the VMXNET3 adapter. Haven't applied any fixes other than what come throgh from windows update. All 4 servers were in-place upgrades from 2008R2. One thing that may or may not be of consequence is the 2008R2 had the TCPChimney issue resolved.