POST Disaster Recovery - Exchange Database issues.

[PHAICT2]

We've had major issues with our servers recently which (among other things) forced us into doing a full restore of our virtual exchange server to a new cluster.

 

Initially everything looked OK and majortity of staff are working fine, however today we have found that when we open exchange powershell we get....

 

?*?????%??*???'??????'???→??→?+?????????*?????%??*???'??????'???→??→?+??I??????*?????%??*???'???????'???→??→?+??H??????

*?????%??*???'???????'???→??→?+??K??????*?????%??*???'???????'???→??→?+??J??????*?????%??*???'???????'???→??→?+??E?????

?*?????%??*???'???????'???→??→?+??D??????*??????'????'???→??→?+??G??????*??????'??????'???→??→?+??F??????*?????%??*???'

?????'???→??→?+??Q??????*???????↨?%??*!↑+¶§??* ?????????'????'???→??→?+??P??????*??????%??*%+???↨?%??*!↑+¶§??* ????????

?'?A??'???→??→?+??88??????%??*%+???↨?%??*!↑+¶§??* ?????????'?=????????????BJ??7+¶▼→!??'???→??→?+??8;??????%??*%+???↨?%?

?*!↑+¶§??* ?????????'??→??'???→??→?+??8:??????%??*%+???↨?%??*!↑+¶§??* ?????????'??→??'???→??→?+??85??????%??*%+???↨?%??

*!↑+¶§??* ?????????'??↑??'???→??→?+??84???????↨?%??*!↑+¶§??* ?????????'????'???→???+▲&??+▲&?+????????*)↑▼*????????1??↑→

?§05¶*→¶▼%?????????→?+??8??????*?????%??*???'???????'???→??→?+??;??????*?????%??*???'???????'???→??→?+??:??????*?????%?

?*???'???????'??Scope},

+ CategoryInfo : ParserError: (??????*?????%??...?↨?%??*!↑+¶§??*:String) [], ParentContainsErrorRecordExc

eption

+ FullyQualifiedErrorId : UnexpectedToken

 

 

 

and also certain members of staff are unable to send / receive emails with the message...

There's a problem with the recipient's mailbox. Please try resending the message. If the problem continues, please contact your helpdesk.

 

and...

 

#554 5.2.0 STOREDRV.Deliver.Exception:StoragePermanentException.MapiExceptionJetErrorReadVerifyFailure; Failed to process message due to a permanent exception with message Cannot complete delivery-time processing. 16.55847:581D0000, 17.43559:000000009E020000000000008C00000000000000, 255.23226:00000000, 255.27962:0A000000, 255.27962:0E000000, 255.27962:0A000000, 255.27962:9E000000, 255.17082:06FCFFFF, 0.18273:00000000, 4.21921:06FCFFFF, 255.27962:FA000000, 255.1494:43000000, 255.1238:13800000, 0.32865:0300E965, 6.41897:170104804000060E0F010480, 6.21970:0F01048040008F660201F065, 4.13300:06FCFFFF, 4.2797:06FCFFFF, 0.39635:0F010480, 4.54995:06FCFFFF, 4.50899:06FCFFFF, 1.45987:E924F600, 4.47913:06FCFFFF, 1.45987:04FCC600, 4.47913:06FCFFFF, 4.64297:06FCFFFF, 0.55383:0F010480, 4.5041:06FCFFFF, 4.4465:06FCFFFF, 4.6833:06FCFFFF, 0.50217:0F010480, 4.5093:06FCFFFF, 4.5318:06FCFFFF, 4.10104:06FCFFFF, 1.64937:C06B7DC4, 1.43123:04FCC600, 4.60457:0F010480, 4.4353:0F010480, 0.57449:0F010480, 4.6025:05000780, 4.5257:05000780, 4.4606:06FCFFFF, 255.1750:00000000, 0.26849:06FCFFFF, 255.21817:06FCFFFF ##We are about to turn the server off and use the eseutil to check and/or repair the database. Does anyone else have any input to our situation?

 

Thank you.

[Roberto]

I would start by migrating mailboxes from the suspect database to a new one before doing anything with eseutil. What version of Exchange are you running?

[PHAICT2]

Hi Roberto. We're using Exchange 2010. Ok we will look into this. Could you be so kind as to elaborate on why this would be your preferred method at this stage ?

[Roberto]

Hi Roberto. We're using Exchange 2010. Ok we will look into this. Could you be so kind as to elaborate on why this would be your preferred method at this stage ?

 

Because Microsoft don't support ongoing use of a suspected database (see New Support Policy for Repaired Exchange Databases - Exchange Team Blog - Site Home - TechNet Blogs). I think it makes sense, in light of what they say there, to recover whatever mailboxes you can to a new database *first* (use move-mailbox) then run eseutil on whatever's left then move the recovered mailboxes then shoot the old, suspect DB.

 

If you interpret that policy literally, you could say "I haven't run ESEUtil *yet* and I don't need to evacuate mailboxes prior to doing so" but I think in terms of ensuring you protect and recover mail data as best you can, it makes absolutely no sense at all to use a suspect mail DB for a moment longer than necessary.

Edited February 8, 2016 by Roberto

[PHAICT2]

how long does a process like this usually take on average (250 users)....also do you know why powershell is doing that above or is it just a generic message due to our potentially corrupt db ?

[Roberto]

how long does a process like this usually take on average (250 users)....also do you know why powershell is doing that above or is it just a generic message due to our potentially corrupt db ?

 

I don't think the powershell issue will be down to DB corruption per se. Having given this some more thought, I suspect that both symptoms are pointing to an underlying issue with storage. Normally at this stage, I'd be suggesting contacting Microsoft support. While it's not cheap to do that, it might cost less than *not* calling them in the long run. Are you getting storage errors in the event logs on exchange? Incidentally, you mention restoring Exchange to a new cluster - do you mean you have an Exchange cluster, or do you mean you have an Exchange server on a virtual server cluster?

 

As for length of time for mailbox moves, it depends on all kinds of things - mailbox size (number and size of messages) is more important than number of mailboxes, along with disk and (if relevant) network I/O.

Edited February 8, 2016 by Roberto

[PHAICT2]

Indeed we have exchange server on a virtual server cluster. Thanks for your input so far, much appreciated.

 

So far we've tried a couple of mailbox repair requests on a specific (less vital) user's mailbox and it just gives us

 

Request ID | Mailbox | ArchiveMailbox | Database | Server

 

...but nothing seems to really happen, unless it's cooking in the background?

 

As for storage ill look into the logs event logs again and see if there's anything that I can pick out.

[PHAICT2]

The total capacity of the HDD that the database sits on is 753GB of that, 182GB is free.

 

We've looked at the staff edb file and this is at around 320GB.

 

The folder that contains the database file and log files is clocking in at 557GB

 

I am still looking at the log files however, could this suggest why we're having these issues?

[Roberto]

If the log files are large then this suggests that backups are not working. In Exchange (and other transactional database technologies) the log files are part of the database operation, not what we'd think of as "logs" from a human point of view. A successful "Exchange Aware" full backup of an Exchange database will clear the logs related to that database (and is the only safe way of doing so).

 

If this isn't happening then either:

1) Backups aren't working (they could be failing due to DB corruption of course).

2) Backups aren't configured (self explanatory).

3) The restore you mention has somehow managed to incorporate a lot of extra spurious data that isn't causing a problem as such, other than it's "in the way" on the disk.

 

Personally, I'd seriously think about spooling up another Exchange server and moving mailboxes to a database on that, at this point. If the log files are growing like that without any explanation then something seems very wrong with that Exchange server.