Exchange Server mail store failure aaaAAAARGH!!!

[m2d2]

Perfect storm:

 

Power failure during overnight backup makes raid cache loose data which corrupts pages in mailbox store. Came in to find server waiting for key press during POST

 

Before anyone says UPS, it was a power supply in the server that failed. So much for redudant powersupplies.

 

I attempting a restore from last known good backup

 

Wish me luck

 

If anyone doesn't mind being on the end of phone then please PM. It's very lonely in the server room at the moment.

[Michael]

It sounds you have everything in order to me. You've identified the problem and you're fixing it

[m2d2]

It sounds you have everything in order to me. You've identified the problem and you're fixing it

 

Thanks for the vote of confidence but we'll know for sure in a couple of hours.

I don't think staff 'll let me out alive if this doesn't work.

[Dos_Box]

If you find that your mailstore is dirty or corrupt try this article:@

 

Using the Exchange tools ISINTEG and ESEUTIL to Ensure the Health of your Information Store

 

And if you have a TechNet+ subscription it may be time to use of of the 2 free support incidences you get (I had to last year after our mail server killed itself!).

[m2d2]

If you find that your mailstore is dirty or corrupt try this article:@

 

Using the Exchange tools ISINTEG and ESEUTIL to Ensure the Health of your Information Store

 

And if you have a TechNet+ subscription it may be time to use of of the 2 free support incidences you get (I had to last year after our mail server killed itself!).

 

Thanks for the tip Dos_Box. I did think about using ESEUTIL but the timescales Microsoft gave were just not workable. We have mock exams next week and we can't afford to be offline that long.

[m2d2]

Oh bother!!!!!

 

Restore went OK but everybody starts using email again before I was able to replay log files from yesterday.

 

Now what am I going to do? Is it going to be a case of restore to recovery group, replay and then exmerge Thursday's messages?

[Dos_Box]

You didn't isolate the server!!

[m2d2]

Yeah I know. That was a pretty dumb thing to do but I don't get to practice this procedure all that often.

[PiqueABoo]

the timescales

 

This is why I have nothing but fond memories for a couple of mail "servers" I ran a long time ago that used to keep your e-mail in the file system (some still do) i.e. one text file per e-mail. Pointlessly prettified mail kind of breaks it now, but worst case then you could just get mail-addicted users to point notepad or equivalent at a folder.

 

Exchange's big mysterious database with those potentially huge integrity scanning/fixing timescales is definitely something to consider when picking a mail server. Not that I've seen a store issue for a long time, but I've had to run some of those scans in environments where mail was much more credibly business critical and yeah... much too, much too long.. restores are the only practical way out of these scenarios.

 

I don't get to practice this procedure all that often.

 

Very few do... you need all that time, a lab bench with at least one DC, Exchange and a workstation and backup/restore. That's easier now though with VMs and I bet there isn't anyone who's been through an Exchange DR who wouldn't strongly recommend maybe a week configuring and playing with this and writing down all the important steps. DR prep and practice has been a chapter or two in decent Exchange books since they started writing them.

Edited December 12, 2008 by PiqueABoo