Exchange weirdness

[Jawloms]

We're running Exchange 2010 SP3 (Yes, I know) and it appears that email isn't working for everyone on a particular database. My SLT is working, but my staff can't send or receive emails. The Outlook client says it's connected, but if you send an email it appears to go, then a second later an non-delivery message appears in your inbox, but then disappears instantly. The email you sent is not in your Sent Items folder, and the NDR is nowhere to be found. OWA again appears to work as you can see your mailbox, but if you try and delete anything it says the item could not be found, and trying to send an email just says an error occurred.

 

Restarted the Information Store service and rebooted the server. Now off to delve in to the logs........

 

Errrm..... Help?!

[Jawloms]

In my event log at 20:56 lsat night I have;

 

Information Store (6584) Staff: An attempt to write to the file "C:\Program Files\Microsoft\Exchange Server\V14\Mailbox\Staff\Staff.edb" at offset 323207364608 (0x0000004b40a90000) for 0 (0x00000000) bytes failed after 0 seconds with system error 665 (0x00000299): "The requested operation could not be completed due to a file system limitation ". The write operation will fail with error -1022 (0xfffffc02). If this error persists then the file may be damaged and may need to be restored from a previous backup.

 

For more information, click http://www.microsoft.com/contentredirect.asp.

 

Closely followed by hundreds of;

 

At '18/05/2023 20:56:41' database copy 'Staff' on this server appears to have a serious I/O error. To help identify the specific failure, consult the Event log on the server for other storage and "ExchangeStoreDb" events. Service recovery was attempted by failover to another copy. Failover was unsuccessful in restoring the service. Error: There is only one copy of this mailbox database (Staff). Automatic recovery is not available.

 

My Veeam backup of the entire VM finished at 20:46. I hoping that means the backup is good, and not the cause! Guess I'm restoring......

[psydii]

Oof, a month ago I'd have said call Microsoft and have them hold your hand fixing it. But you're out of support.

 

Take a backup (of the EDB and Log files, don't rely on what VEEAM has at this point), and then use ESUTIL to analyse and 'fix' problems. As soon as it has repaired the DB as best it can, create a new one and migrate everybody off onto the new DB.

https://www.codetwo.com/admins-blog/how-to-use-exchange-extensible-storage-engine-utilities-eseutil-tool/

 

 

If Veeam does have a recent good copy of mailboxes, it should (as an alternative) be possible to restore the mailboxes to a new DB, loosing only changes that occoured since the last good backup.

[Jawloms]

Currently restoring and just watching a progress bar. Thanks for the link. It's not an Exchange aware backup which throws in another curveball.

[psydii]

Just seen the other post. Without and Exchange Aware backup, if eseutil doesn't help you'll need to restore the database and log files. If exchange didn't snapshot and correctly truncate the log files during backup (or you have circular logging), you will almost certainly have to use eseutil to get the restored DB into a safe/healthy state. Even if you are able to bring the restored database back into service - it is best practice to migrate all the mailboxes from the restored database to a new one, because using eseutil to repair a db leaves it in a slightly inconsistent state that will cause you more problems down the line. (or so the MS escalation engineer told me at 2am one morning when something similar happened to me).

 

/edit: my actual job keeps delaying me posting - so some of what I've written is redundant, because you've already got there! Best of luck.

Edited May 19, 2023 by psydii

[Jawloms]

I'm now at that awkward stage where I'm waiting for things to happen, so don't look busy.

[RobD]

I think I'd take the broken database offline first,

 

open powershell as admin, find the folder with the DB in it and run:

 

Dismount-Database Staff

 

Then run a defag and repair (make sure you have plenty of space on the disk first):

 

eseutil /P Staff

 

Cant hurt if its broken already, these might help:

 

https://practical365.com/defrag-exchange-2010-mailbox-database/

 

https://www.stellarinfo.com/blog/how-to-use-eseutil-for-exchange-database-repair/

[Jawloms]

Currently waiting for a defrag to finish...... Lengthy!

[RobD]

There's a chance after the defrag the DB might complain it did a unclean (dirty) shutdown so you might have to run: Eseutil /R

[Jawloms]

It appears the repair and defrag have actually worked! It's mounted again and mail is flowing once more. No errors in the event logs. I'm now going to tell it to move all mailboxes in to a new database.......

 

Thanks to all.

[3s-gtech]

Excellent news. I wouldn’t trust it at all either, and you can use this incident to move things forward (onto a supported Exchange version/Online!)

 

If you do keep onsite Exchange in any form, set up Windows Server Backup on the box. It’s Exchange aware and can do much better DB backups. You can run it alongside Veeam.