Office 365: dirsync error 611 and Invalid Namespace warning

[sonofsanta]

Just starting a test deployment of Office 365 to migrate away from Exchange 2007. Installed dirsync yesterday on my SCCM server using the full SQL install, all fine, sync working, happy days.

 

Checked event logs this morning, password sync is failing with error 611, "The operation was aborted because the client side timeout limit was exceeded." The only relevant info I can find on this specific error is this thread on office365.com where it's mentioned that it's a bug where "the pwdLastSet value of an user account is set to 0, which is invalid for DirSync to process" - this is believable as we don't force password changes. That thread's from August and I only downloaded & installed yesterday, so apparently that bug was just never fixed (marvellous).

 

In an attempt to find a workaround, though, I tried to rerun the dirsync config tool. Gave it all the credentials/details, told it to get on with it and it failed with the error Invalid Namespace. Nothing in event viewer, no relevant logs I can spot in the install directory. Anytime I try and open the config tool now, I get that warning with a lot of exception text:

See the end of this message for details on invoking 
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.Management.ManagementException: Invalid namespace 
  at System.Management.ManagementScope.Initialize()
  at System.Management.ManagementObjectSearcher.Initialize()
  at System.Management.ManagementObjectSearcher.Get()
  at Microsoft.Online.DirSync.Common.MiisAction.GetTargetMA()
  at Microsoft.Online.DirSync.Common.MiisAction.IsSyncInProgress()
  at Microsoft.Online.DirSync.Common.PrerequisiteChecks.ThrowIfSyncInProgress()
  at Microsoft.Online.DirSync.UI.IntroductionWizardPage.PrerequisiteValidation()
  at Microsoft.Online.DirSync.UI.IntroductionWizardPage.OnLoad(EventArgs e)
  at System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
  at System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
  at System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
  at System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
  at System.Windows.Forms.Control.CreateControl()
  at System.Windows.Forms.Control.WmShowWindow(Message& m)
  at System.Windows.Forms.Control.WndProc(Message& m)
  at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
  at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
   Assembly Version: 2.0.0.0
   Win32 Version: 2.0.50727.5472 (Win7SP1GDR.050727-5400)
   CodeBase: file:///C:/Windows/Microsoft.NET/Framework64/v2.0.50727/mscorlib.dll
----------------------------------------
ConfigWizard
   Assembly Version: 1.0.0.0
   Win32 Version: 1.0.6567.18
   CodeBase: file:///C:/Program%20Files/Windows%20Azure%20Active%20Directory%20Sync/ConfigWizard.exe
----------------------------------------
System
   Assembly Version: 2.0.0.0
   Win32 Version: 2.0.50727.5467 (Win7SP1GDR.050727-5400)
   CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System/2.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
Microsoft.Online.DirSync.Common
   Assembly Version: 1.0.0.0
   Win32 Version: 1.0.6567.18
   CodeBase: file:///C:/Program%20Files/Windows%20Azure%20Active%20Directory%20Sync/Microsoft.Online.DirSync.Common.DLL
----------------------------------------
System.Windows.Forms
   Assembly Version: 2.0.0.0
   Win32 Version: 2.0.50727.5468 (Win7SP1GDR.050727-5400)
   CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Windows.Forms/2.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
   Assembly Version: 2.0.0.0
   Win32 Version: 2.0.50727.5467 (Win7SP1GDR.050727-5400)
   CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Drawing/2.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
Microsoft.Online.DirSync.Resources
   Assembly Version: 1.0.0.0
   Win32 Version: 1.0.6567.18
   CodeBase: file:///C:/Program%20Files/Windows%20Azure%20Active%20Directory%20Sync/Microsoft.Online.DirSync.Resources.DLL
----------------------------------------
System.Core
   Assembly Version: 3.5.0.0
   Win32 Version: 3.5.30729.5420 built by: Win7SP1
   CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Core/3.5.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.DirectoryServices
   Assembly Version: 2.0.0.0
   Win32 Version: 2.0.50727.5467 (Win7SP1GDR.050727-5400)
   CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.DirectoryServices/2.0.0.0__b03f5f7f11d50a3a/System.DirectoryServices.dll
----------------------------------------
System.Configuration
   Assembly Version: 2.0.0.0
   Win32 Version: 2.0.50727.5476 (Win7SP1GDR.050727-5400)
   CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Configuration/2.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
   Assembly Version: 2.0.0.0
   Win32 Version: 2.0.50727.5476 (Win7SP1GDR.050727-5400)
   CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Xml/2.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
System.ServiceProcess
   Assembly Version: 2.0.0.0
   Win32 Version: 2.0.50727.5420 (Win7SP1.050727-5400)
   CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.ServiceProcess/2.0.0.0__b03f5f7f11d50a3a/System.ServiceProcess.dll
----------------------------------------
System.Management
   Assembly Version: 2.0.0.0
   Win32 Version: 2.0.50727.5420 (Win7SP1.050727-5400)
   CodeBase: file:///C:/Windows/assembly/GAC_MSIL/System.Management/2.0.0.0__b03f5f7f11d50a3a/System.Management.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:


   


When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

 

I can continue or quit there. If I continue, I can get to the same point and get the same error, regardless of whether password sync is ticked or not (I'd really like it ticked, though).

 

More frustratingly, if I try and uninstall from Programs and Features, it just pops up and tells me, what else, Invalid Namespace. I can't even re-install!

 

All of this applies even after a server restart.

 

Any clues? Don't really want to have to restore the whole server from backup especially as it seems like the kind of fault that will recur.

Edited January 10, 2014 by sonofsanta

[sonofsanta]

Looks like the invalid namespace is related to this: DirSync installation and/or operation fails Invalid Namespace (System.Management.ManagementException: Invalid namespace / just-in-time (JIT) debugging) | Jesper Stahle's Notes From the Field as it's the SCCM server (though I am on SCCM2012 SP1, but the server did start out on 2012RTM).

 

However, I still can't bloomin' uninstall dirsync to reinstall without the error happening. Argh! Is there any way of rebuilding the WMI namespace for dirsync?

 

EDIT: nevermind, found the MOF file, need to open a command prompt, then:

cd "C:\Program Files\Windows Azure Active Directory Sync\SYNCBUS\Synchronization Service\Bin"
mofcomp mmswmi.mof

 

Now letting me uninstall.

 

Still: 611 error, are there any workarounds for that? Forcing a password change domain wide (to clear the zeroes) wouldn't work as staff on maternity leave wouldn't be able to change their password.

Edited January 10, 2014 by sonofsanta

[EduTech]

Hi,

 

I have seen this quite a few times when people have installed DirSync on a Server that holds any System Centre Role. The quickest way to uninstall DirSync if you get that error is to just Stop the 'Forefront Identity Manager' Service and then go and attempt the Uninstall.

 

In terms of Password Sync have you done the following:

 

• Run Enable-MSOnlinePasswordSync

• Run Full-Sync

• Set-FullPasswordSync

• Restart FIM service

 

Once the FULL SYNC has occurred are you seeing repeated failures with Event 611?

 

- Can you confirm that you are running the latest version of Directory Sync Tool? [Latest Version: 6567.0018]

- Can you attach further information contacted within Event ID 611 as the actual error varies but the ID is the same.

 

Many Thanks,

James.

Edited January 11, 2014 by EduTech

[sukh]

Can you check the local sec policy on server and see if network security: LDAP client signing requirements " is enabled?

If yes, can you disable and monitor/check.

[EduTech]

also, if you can get a Network Trace on the DirSync Server when the Event 611 occurs that would help we see what is going on from that level.

 

I am interested to see what kind of request/response appears when a query is run against your domain controllers... :-) interesting you can see this stuff on a NETMON trace and can point you in the right direction.

 

Download NetMon: http://www.microsoft.com/en-gb/download/details.aspx?id=4865

 

You will probably find that EVENT ID 611 will occur every 30 minutes... so you should be able to judge when the event occurs next.

 

...

 

Thanks,

James.

Edited January 11, 2014 by EduTech

[sonofsanta]

I managed to uninstall & reinstall after recompling the MOF, I've just run through the config again and gotten through successfully so that issue appears to be dealt with. I'd probably have hit it no matter what server of mine I installed this on, the only servers I have that haven't got something System Center on are web facing, and the documentation recommended security and caution about where it was installed.

 

The 611s have been logged throughout the weekend, every half hour as you say. After the reinstall prompted a full sync the passwords seemed to sync up fine, and looking at the logs now (as I've triggered another full sync from going through config) I'm being told that Enable-MSOnlinePasswordSync executed successfully. The system seems capable of syncing passwords, it's just... choosing not to most of the time.

 

611 error text is:

Password synchronization failed for domain: domain.local. Details: 
System.DirectoryServices.Protocols.LdapException: The operation was aborted because the client side timeout limit was exceeded. 
  at System.DirectoryServices.Protocols.LdapConnection.ConstructResponse(Int32 messageId, LdapOperation operation, ResultAll resultType, TimeSpan requestTimeOut, Boolean exceptionOnTimeOut)
  at System.DirectoryServices.Protocols.LdapConnection.SendRequest(DirectoryRequest request, TimeSpan requestTimeout)
  at Microsoft.Online.PasswordSynchronization.DirectoryAttributeSearcher.d__8.MoveNext()
  at Microsoft.Online.PasswordSynchronization.DirectoryAttributeSearcher.d__0.MoveNext()
  at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.BuildPasswordBatch(IList`1 passwordChanges, IEnumerable`1 changeObjects)
  at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.BuildPasswordBatch(IList`1 changeSetObjects)
  at Microsoft.Online.PasswordSynchronization.DeltaSynchronizationTask.SynchronizeCredentialsToCloud()
  at Microsoft.Online.PasswordSynchronization.PasswordSynchronizationTask.SynchronizeSecrets()
  at Microsoft.Online.PasswordSynchronization.SynchronizationExecutionContext.SynchronizeDomain()
  at Microsoft.Online.PasswordSynchronization.SynchronizationManager.SynchronizeDomain(SynchronizationExecutionContext syncExecutionContext)
.

 

Version is 1.0.6567.0018 according to the installer, running on SQL Server 2008 (so a /fullsql install) on Windows 2008R2. Domain controllers are 2008R2 as well. @sukh: policy is set to negotiate signing, which sounds like it can cope with no signature; do you think None is still worth trying?

 

I'll run NetMon later this morning - staff briefings and health & safety meetings first, I'm afraid

 

Thanks for the responses.

[sonofsanta]

You know what, since rerunning the config (and changing no settings there or anywhere else), I've had no 611's logged. I'm not sure if it's fixed or just trolling me but I can see event 657 (successful password change) logged a few times, and we have had some passwords changed in the office this morning.

 

Maybe it's just another symptom of sitting alongside System Center? I'll keep an eye and see how it goes.

[EduTech]

Hi,

 

Can you please monitor this, If the Event IDs come back in the next few hours (couple of days possibly) do the following:

 

+ Open a Support Case with Microsoft

 

+ Enable Password Sync Verbose Logging

 

[Launch DirSync Configuration Console - C:\Program Files\Windows Azure Active Directory Sync\DirSyncConfigShell.psc1

Enable-PasswordSyncLog -TraceLevel Verbose -FilePath "C:\Dirsync_Log\PasswordSync.log]

 

+ Take a Network Trace - at rough time of an event occurrence

 

Upload to the Support Request for Analysis.

 

Many Thanks,

J

[sonofsanta]

Will do - thanks for the help. Looking back through logs the first 611 of this ilk occurred within an hour of the initial install last Thursday and then recurred every half an hour, so going 4 hours without issue is an improvement. I suspect the tell will be tomorrow morning though, making sure that System Center hasn't borked it overnight.

[EduTech]

Will do - thanks for the help. Looking back through logs the first 611 of this ilk occurred within an hour of the initial install last Thursday and then recurred every half an hour, so going 4 hours without issue is an improvement. I suspect the tell will be tomorrow morning though, making sure that System Center hasn't borked it overnight.

 

Don't be fooled by the length of time it may work! ;-) if this is what I think it is...

 

James.

[sonofsanta]

Aaaaand saying that it's just logged a 611 client side timeout limit. Will try and get details - getting a case ready to open, turned logging on and just waiting to get a network trace at 13:35. EDIT: service request opened, PMed the reference, files are attached to the service request. Edited January 13, 2014 by sonofsanta

[EduTech]

Aaaaand saying that it's just logged a 611 client side timeout limit. Will try and get details - getting a case ready to open, turned logging on and just waiting to get a network trace at 13:35. EDIT: service request opened, PMed the reference, files are attached to the service request.

 

Thanks.

 

James.

[sonofsanta]

FWIW, just spoke to a support guy on the phone, knocked LDAP client signing requirements down to None and a 611 has popped up again, half an hour after the last one. Ran two full syncs while I was on the phone and they were fine, and interestingly seem to have had no effect on the schedule of the 611 events. Will update the service call with this as well.

[EduTech]

I have just spoken with the Engineer. Please Monitor and report back when the event occurs again (as I am convinced it will) :-)

 

James.

[sonofsanta]

Still occurring every half an hour - just done so now, in fact - even if I knock the DC down to None on the LDAP requirements as well. Need to finish emailing the support engineer on that one, actually.

 

D'you have your suspicions, then?

[EduTech]

Still occurring every half an hour - just done so now, in fact - even if I knock the DC down to None on the LDAP requirements as well. Need to finish emailing the support engineer on that one, actually.

 

D'you have your suspicions, then?

 

Yes, but i am not in a place to say at the moment as I am still debugging the issue at a very low level.

 

Q. Do you have any windows 2003 domain controllers?

 

Thanks,

J

[sonofsanta]

Yes, but i am not in a place to say at the moment as I am still debugging the issue at a very low level.

 

Q. Do you have any windows 2003 domain controllers?

 

Thanks,

J

 

Did have - do not now. Domain functional level is 2008R2, two DCs.

[EduTech]

Thanks.

 

Will be in touch... as no doubt this will come to me soon.

 

Jame.s

[sonofsanta]

Alrighty then not an urgent case at the moment as the system isn't live for users yet here.

 

Thanks a lot!

[EduTech]

Alrighty then not an urgent case at the moment as the system isn't live for users yet here.

 

Thanks a lot!

 

No Problem, I'm confident *just got to do some more tests* that I have found the root cause now, will speak with you via SR [i have taken ownership as your case has now been escalated] + check your environment :-)

 

Thanks,

 

James.

Edited January 13, 2014 by EduTech

[sonofsanta]

No Problem, I'm confident *just got to do some more tests* that I have found the root cause now, will speak with you via SR [i have taken ownership as your case has now been escalated] + check your environment :-)

 

Thanks,

 

James.

 

Very cool - anything you need from me, give me a shout, I'm in now till 4pm and can hook you up with remote sessions as necessary (on a laptop if you want to be left to look around for a while).

[EduTech]

I just thought I would update this thread, in case anyone else got the particular Error

 

Password synchronization failed for domain: domain.local. Details:

System.DirectoryServices.Protocols.LdapException: The operation was aborted because the client side timeout limit was exceeded.

at System.DirectoryServices.Protocols.LdapConnection.ConstructResponse(Int32 messageId, LdapOperation operation, ResultAll resultType, TimeSpan requestTimeOut, Boolean exceptionOnTimeOut)

 

The DirSync components need to make an LDAP query to the DC’s as part of the password sync process, there is an issue in the LDAP library in windows that causes binds to DC’s to fail when chasing LDAP referrals. DirSync does not need to chase referrals to work and therefore does not normally raise an issue for the product.

 

However, if the GUEST account is enabled the bind will work, but a subsequent query will not be processed correctly and an LDAP timeout error generated. This causes the password sync component of DirSync to fail.

 

This issue in the LDAP library is in the windows operating system and is only currently exposed by DirSync. The windows product group are currently looking at fixing this in a future version of windows (i.e not one that is currently released)

 

Further more a change maybe made to DirSync Product in a later version!

 

The current solution is to disable the GUEST account.

 

James.

Edited February 13, 2014 by EduTech