Techdw Posted October 9, 2015 Posted October 9, 2015 Hello All, Got an issue with Exchange. First of all, Staff mailboxes become dismounted and show disconnected on the client site. Content index corrupted. We repaired the mailbox database which worked for a while, repaired content index worked for 2-3 days. Students mail box become dismounted, done the same as above for repair, worked for a while, after both dismounted again. OWA works fine. Anyone have any idea's on what this could be.
IanT Posted October 10, 2015 Posted October 10, 2015 Whats in the event logs? Is exchange fully patched? Do you have enough space on the server?
Techdw Posted October 10, 2015 Author Posted October 10, 2015 Havent got the latest patch on. Event view, Log Name: Application Source: ESE Date: 10/10/2015 16:46:28 Event ID: 516 Task Category: Logging/Recovery Level: Error Keywords: Classic User: N/A Computer: ExchangeSRV.Domain.local Description: Information Store - Staff Mailboxes (5596) Staff Mailboxes: Database E:\Staff Mailboxes\Staff Mailboxes.edb: Page 5882920 (0x0059c428) failed verification due to a timestamp mismatch. The 'before' timestamp persisted to the log record was 0x73c988e5 but the actual timestamp on the page was 0x73c97b39. The 'after' update timestamp 0x73c9966f that would have updated the on page timestamp. Recovery/restore will fail with error -566. If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware "losing" one or more flushes on this page sometime in the past. Please contact your hardware vendor for further assistance diagnosing the problem. For more information, click http://www.microsoft.com/contentredirect.asp. Event Xml: 516 2 3 0x80000000000000 15999239 Application ExchangeSRV.domain.local Information Store - Staff Mailboxes 5596 Staff Mailboxes: E:\Staff Mailboxes\Staff Mailboxes.edb 5882920 (0x0059c428) 0x73c988e5 0x73c97b39 -566 0x73c9966f -------------------------------------------- Log Name: Application Source: ExchangeStoreDB Date: 10/10/2015 16:46:28 Event ID: 235 Task Category: Database recovery Level: Error Keywords: Classic User: N/A Computer: ExchangeSRV.Domain.local Description: At '10/10/2015 16:46:28', the copy of database 'Staff Mailboxes' on this server encountered a serious I/O error that may have affected all copies of the database. Consult the Event log on the server for "ExchangeStoreDb" or "msexchangerepl" events for information about the failure. All data should be immediately moved out of this database into a new database. Event Xml: 235 2 1 0x80000000000000 15999241 Application ExchangeSRV.Domain.local Staff Mailboxes f9d75c14-7a23-4b3e-9f10-2911508ff498 Ese LostFlushDbTimeTooOld Staff Mailboxes There is only one copy of this mailbox database (Staff Mailboxes). Automatic recovery is not available. 10/10/2015 16:46:28 None E:\Staff Mailboxes\Staff Mailboxes.edb 192771555328 32768 --------------------------------------------------------- Log Name: Application Source: MSExchangeIS Date: 10/10/2015 16:46:30 Event ID: 2006 Task Category: Physical Access Level: Error Keywords: Classic User: N/A Computer: ExchangeSRV.Domain.local Description: Microsoft Exchange Information Store worker process (5596) has encountered an unexpected database error (dbtime on page smaller than dbtimeBefore in record) for database 'Staff Mailboxes' with a call stack of at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetDatabase.TryOpen(Boolean lossyMount) at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountActive(Context context, Boolean allowLoss) at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountDatabase(Context context, MountFlags flags, Boolean& errorOnTheThreadExecutingTheMount) at Microsoft.Exchange.Server.Storage.StoreCommonServices.Storage.MountDatabase(Context context, StoreDatabase database, MountFlags flags) at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcMountDatabase.EcExecuteRpc(MapiContext context) at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics) at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.<>c__DisplayClass1.b__0() at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](TryDelegate tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state) at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute() at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcServer.EcMountDatabase50(ClientSecurityContext callerSecurityContext, Guid mdbGuid, UInt32 flags, Byte[] auxiliaryIn, Byte[]& auxiliaryOut) at EcMountDatabaseRpc.EcDispatchCall(EcMountDatabaseRpc* , IAdminRpcServer server, ClientSecurityContext callerSecurityContext, Byte[] auxiliaryIn, Byte[]& auxiliaryOut) at Microsoft.Exchange.Rpc.AdminRpc.AdminRpcServer_Wrapper.InternalExecute(AdminRpcServer_Wrapper* ) at Microsoft.Exchange.Rpc.ManagedExceptionCrashWrapper.Execute(ManagedExceptionCrashWrapper* ) at EcMountDatabase50_Managed(Void* hBinding, _GUID* pguidStorageGroup, _GUID* pguidMdb, UInt32 ulFlags, UInt32 cbAuxIn, Byte* rgbAuxIn, UInt32* pcbAuxOut, Byte** prgbAuxOut) . Event Xml: 2006 2 2 0x80000000000000 15999245 Application ExchangeSRV.Domain.local dbtime on page smaller than dbtimeBefore in record at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetDatabase.TryOpen(Boolean lossyMount) at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountActive(Context context, Boolean allowLoss) at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountDatabase(Context context, MountFlags flags, Boolean& errorOnTheThreadExecutingTheMount) at Microsoft.Exchange.Server.Storage.StoreCommonServices.Storage.MountDatabase(Context context, StoreDatabase database, MountFlags flags) at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcMountDatabase.EcExecuteRpc(MapiContext context) at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics) at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.<>c__DisplayClass1.b__0() at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](TryDelegate tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state) at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute() at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcServer.EcMountDatabase50(ClientSecurityContext callerSecurityContext, Guid mdbGuid, UInt32 flags, Byte[] auxiliaryIn, Byte[]& auxiliaryOut) at EcMountDatabaseRpc.EcDispatchCall(EcMountDatabaseRpc* , IAdminRpcServer server, ClientSecurityContext callerSecurityContext, Byte[] auxiliaryIn, Byte[]& auxiliaryOut) at Microsoft.Exchange.Rpc.AdminRpc.AdminRpcServer_Wrapper.InternalExecute(AdminRpcServer_Wrapper* ) at Microsoft.Exchange.Rpc.ManagedExceptionCrashWrapper.Execute(ManagedExceptionCrashWrapper* ) at EcMountDatabase50_Managed(Void* hBinding, _GUID* pguidStorageGroup, _GUID* pguidMdb, UInt32 ulFlags, UInt32 cbAuxIn, Byte* rgbAuxIn, UInt32* pcbAuxOut, Byte** prgbAuxOut) 5596 Staff Mailboxes Microsoft.Isam.Esent.Interop.EsentDbTimeTooOldException: dbtime on page smaller than dbtimeBefore in record at Microsoft.Isam.Esent.Interop.Instance.Init(JET_RSTINFO recoveryOptions, InitGrbit grbit) at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetDatabase.InitializeJetInstance(InitType initType) at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetDatabase.TryOpen(Boolean lossyMount) 5B444941475F4354585D000064000000FF0C00000000000000024800000040B0101072A4E20A80E11010F8A5E20A40B4101048A8E20A4088101048A8E20AC8874010CAFDFFFF80A18030145CD7F9237A3E4B9F102911508FF4980000000080891010DC150000
Homer Posted October 10, 2015 Posted October 10, 2015 Is it on some kind of shared storage, and if so how is it published? Has anything been changed before this started that could do it? If shared storage maybe some kind of multipathing problem is dropping the link which is upsetting Exchange but might not be noticeable on something else
Techdw Posted October 10, 2015 Author Posted October 10, 2015 Haven't change anything. The VM was pulled over onto a new SAN and 2 new host, but that was 6-8 weeks ago. Its been working fine since.
psydii Posted October 11, 2015 Posted October 11, 2015 My default response to big exchange problems is to call Microsoft PSS with the school credit card in hand. It has always got us running in the shortest time possible, and email is one of those services that give a baseline impression of the IT service team competence, that is extremely important. It will be the best £220 you have ever spent. You don't have replication/redundancy enabled do you? Modern exchange servers have been know to fail over databases at the drop of a hat.
forkies Posted October 11, 2015 Posted October 11, 2015 I would be looking at the storage with those errors and that setup. The part "dbtime on page smaller than dbtimebefore in record" suggests to me a problem with either writing to the db (could be a network or San problem) or the log buffer. Either way most likely a storage problem if your server appears to be functioning normally.
Homer Posted October 12, 2015 Posted October 12, 2015 Does it seem to happen at the same kind of time when it does happen? Could be something smashing the I/O on the array and upsetting Exchange (unlikely) or perhaps a backup utility is causing problems and not dealing with snapshots or VSS correctly? I would certainly go with psydii's idea though mate at least if there's a problem with the SAN and another company put it in you can tear strips off of them legitimately Good luck!
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now