Exchange 2013 Issues

[Techdw]

Hello All,

 

Got an issue with Exchange.

 

First of all, Staff mailboxes become dismounted and show disconnected on the client site. Content index corrupted. We repaired the mailbox database which worked for a while, repaired content index worked for 2-3 days.

 

Students mail box become dismounted, done the same as above for repair, worked for a while, after both dismounted again.

 

OWA works fine.

 

Anyone have any idea's on what this could be.

[3s-gtech]

Which CU are you on?

[Homer]

What storage are the databases on?

[IanT]

Whats in the event logs?

 

Is exchange fully patched?

 

Do you have enough space on the server?

[Techdw]

Havent got the latest patch on.

 

Event view,

 

Log Name: Application

Source: ESE

Date: 10/10/2015 16:46:28

Event ID: 516

Task Category: Logging/Recovery

Level: Error

Keywords: Classic

User: N/A

Computer: ExchangeSRV.Domain.local

Description:

Information Store - Staff Mailboxes (5596) Staff Mailboxes: Database E:\Staff Mailboxes\Staff Mailboxes.edb: Page 5882920 (0x0059c428) failed verification due to a timestamp mismatch. The 'before' timestamp persisted to the log record was 0x73c988e5 but the actual timestamp on the page was 0x73c97b39. The 'after' update timestamp 0x73c9966f that would have updated the on page timestamp. Recovery/restore will fail with error -566. If this condition persists then please restore the database from a previous backup. This problem is likely due to faulty hardware "losing" one or more flushes on this page sometime in the past. Please contact your hardware vendor for further assistance diagnosing the problem.

 

For more information, click http://www.microsoft.com/contentredirect.asp.

Event Xml:

516

2

3

0x80000000000000

15999239

Application

ExchangeSRV.domain.local

Information Store - Staff Mailboxes

5596

Staff Mailboxes:

E:\Staff Mailboxes\Staff Mailboxes.edb

5882920 (0x0059c428)

0x73c988e5

0x73c97b39

-566

0x73c9966f

 

--------------------------------------------

 

Log Name: Application

Source: ExchangeStoreDB

Date: 10/10/2015 16:46:28

Event ID: 235

Task Category: Database recovery

Level: Error

Keywords: Classic

User: N/A

Computer: ExchangeSRV.Domain.local

Description:

At '10/10/2015 16:46:28', the copy of database 'Staff Mailboxes' on this server encountered a serious I/O error that may have affected all copies of the database. Consult the Event log on the server for "ExchangeStoreDb" or "msexchangerepl" events for information about the failure. All data should be immediately moved out of this database into a new database.

Event Xml:

235

2

1

0x80000000000000

15999241

Application

ExchangeSRV.Domain.local

Staff Mailboxes

f9d75c14-7a23-4b3e-9f10-2911508ff498

Ese

LostFlushDbTimeTooOld

Staff Mailboxes

There is only one copy of this mailbox database (Staff Mailboxes). Automatic recovery is not available.

10/10/2015 16:46:28

None

E:\Staff Mailboxes\Staff Mailboxes.edb

192771555328

32768

---------------------------------------------------------

 

Log Name: Application

Source: MSExchangeIS

Date: 10/10/2015 16:46:30

Event ID: 2006

Task Category: Physical Access

Level: Error

Keywords: Classic

User: N/A

Computer: ExchangeSRV.Domain.local

Description:

Microsoft Exchange Information Store worker process (5596) has encountered an unexpected database error (dbtime on page smaller than dbtimeBefore in record) for database 'Staff Mailboxes' with a call stack of

at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetDatabase.TryOpen(Boolean lossyMount)

at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountActive(Context context, Boolean allowLoss)

at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountDatabase(Context context, MountFlags flags, Boolean& errorOnTheThreadExecutingTheMount)

at Microsoft.Exchange.Server.Storage.StoreCommonServices.Storage.MountDatabase(Context context, StoreDatabase database, MountFlags flags)

at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcMountDatabase.EcExecuteRpc(MapiContext context)

at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics)

at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.<>c__DisplayClass1.b__0()

at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](TryDelegate tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state)

at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute()

at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcServer.EcMountDatabase50(ClientSecurityContext callerSecurityContext, Guid mdbGuid, UInt32 flags, Byte[] auxiliaryIn, Byte[]& auxiliaryOut)

at EcMountDatabaseRpc.EcDispatchCall(EcMountDatabaseRpc* , IAdminRpcServer server, ClientSecurityContext callerSecurityContext, Byte[] auxiliaryIn, Byte[]& auxiliaryOut)

at Microsoft.Exchange.Rpc.AdminRpc.AdminRpcServer_Wrapper.InternalExecute(AdminRpcServer_Wrapper* )

at Microsoft.Exchange.Rpc.ManagedExceptionCrashWrapper.Execute(ManagedExceptionCrashWrapper* )

at EcMountDatabase50_Managed(Void* hBinding, _GUID* pguidStorageGroup, _GUID* pguidMdb, UInt32 ulFlags, UInt32 cbAuxIn, Byte* rgbAuxIn, UInt32* pcbAuxOut, Byte** prgbAuxOut)

.

Event Xml:

2006

2

2

0x80000000000000

15999245

Application

ExchangeSRV.Domain.local

dbtime on page smaller than dbtimeBefore in record

at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetDatabase.TryOpen(Boolean lossyMount)

at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountActive(Context context, Boolean allowLoss)

at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountDatabase(Context context, MountFlags flags, Boolean& errorOnTheThreadExecutingTheMount)

at Microsoft.Exchange.Server.Storage.StoreCommonServices.Storage.MountDatabase(Context context, StoreDatabase database, MountFlags flags)

at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcMountDatabase.EcExecuteRpc(MapiContext context)

at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics)

at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.<>c__DisplayClass1.b__0()

at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](TryDelegate tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state)

at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute()

at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcServer.EcMountDatabase50(ClientSecurityContext callerSecurityContext, Guid mdbGuid, UInt32 flags, Byte[] auxiliaryIn, Byte[]& auxiliaryOut)

at EcMountDatabaseRpc.EcDispatchCall(EcMountDatabaseRpc* , IAdminRpcServer server, ClientSecurityContext callerSecurityContext, Byte[] auxiliaryIn, Byte[]& auxiliaryOut)

at Microsoft.Exchange.Rpc.AdminRpc.AdminRpcServer_Wrapper.InternalExecute(AdminRpcServer_Wrapper* )

at Microsoft.Exchange.Rpc.ManagedExceptionCrashWrapper.Execute(ManagedExceptionCrashWrapper* )

at EcMountDatabase50_Managed(Void* hBinding, _GUID* pguidStorageGroup, _GUID* pguidMdb, UInt32 ulFlags, UInt32 cbAuxIn, Byte* rgbAuxIn, UInt32* pcbAuxOut, Byte** prgbAuxOut)

5596

Staff Mailboxes

Microsoft.Isam.Esent.Interop.EsentDbTimeTooOldException: dbtime on page smaller than dbtimeBefore in record

at Microsoft.Isam.Esent.Interop.Instance.Init(JET_RSTINFO recoveryOptions, InitGrbit grbit)

at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetDatabase.InitializeJetInstance(InitType initType)

at Microsoft.Exchange.Server.Storage.PhysicalAccessJet.JetDatabase.TryOpen(Boolean lossyMount)

5B444941475F4354585D000064000000FF0C00000000000000024800000040B0101072A4E20A80E11010F8A5E20A40B4101048A8E20A4088101048A8E20AC8874010CAFDFFFF80A18030145CD7F9237A3E4B9F102911508FF4980000000080891010DC150000

[Homer]

Is it on some kind of shared storage, and if so how is it published? Has anything been changed before this started that could do it?

 

If shared storage maybe some kind of multipathing problem is dropping the link which is upsetting Exchange but might not be noticeable on something else

[Techdw]

Haven't change anything.

 

The VM was pulled over onto a new SAN and 2 new host, but that was 6-8 weeks ago. Its been working fine since.

[psydii]

My default response to big exchange problems is to call Microsoft PSS with the school credit card in hand.

 

It has always got us running in the shortest time possible, and email is one of those services that give a baseline impression of the IT service team competence, that is extremely important.

 

It will be the best £220 you have ever spent.

 

 

You don't have replication/redundancy enabled do you? Modern exchange servers have been know to fail over databases at the drop of a hat.

[forkies]

I would be looking at the storage with those errors and that setup. The part "dbtime on page smaller than dbtimebefore in record" suggests to me a problem with either writing to the db (could be a network or San problem) or the log buffer. Either way most likely a storage problem if your server appears to be functioning normally.

[Homer]

Does it seem to happen at the same kind of time when it does happen? Could be something smashing the I/O on the array and upsetting Exchange (unlikely) or perhaps a backup utility is causing problems and not dealing with snapshots or VSS correctly?

 

I would certainly go with psydii's idea though mate at least if there's a problem with the SAN and another company put it in you can tear strips off of them legitimately

 

Good luck!