Exchange Server Crashed!!

[brahma]

Not sure if you will remember me - I'm the guy that has inherited a dodgy Windows Network. On an earlier post, I mentioned that I have 5 domain controllers one of which was an Exchange Server. Well guess what? Today the server suffered a hard drive failure and, depsite RAID5 being installed on the server, we have lost everything! I spent hours trying to resurrect but have had it! What makes it worse, is that I found out today our backups aren't working! The last completed backup was about 2 weeks ago!

 

I am concerned that some advice on this forum suggested demoting a Domain Controller running Exchange server to a standard member server would be a problematic exercise. Can anyone expand on this for me?

 

Should I rebuild an Exchange box as member server now, or do I have to rebuild it as a domain controller? Was looking forward to a good weekend but I think I know where I'm gonna be!!!

 

Many thanks,

[Gatt]

Did the /disasterrecovery switch not help or ws it that far gone for it?

As for a DC - i think you can install Exchange 2003 on a member server (wasn't it just Exchaneg 2000 that wanted to be a DC?)

[jj99]

You dont need exchange 2003 to be on DC, i would seroiusly suggest against making your exchange server a dc! Exchange likes to be on its own. I would build it just on a member server. Why do you have 5 dc's? could you not demote another dc and have two exchange server is a cluster (has to exchange enterpise) but would mean that your current situation would never happen again?

[jj99]

For your current situation there are some raid 5 raid recovery software on the web as well have you tried them not sure if there any free ones? For example RAID Reconstructor - Recover Data from a Broken RAID Array

[brahma]

The server had the ususal alarm that goes hand in hand with a disk failure but was up and running. Our analysts downed the server and removed the failed drive, replacing it with a good one. When they fired the server up, it failed to start - complaining the boot sector was missing. They put the old disk back in but still no joy. When I got to look at it, a disck consistency check had been run which reported it had "fixed" the problem. When I rebooted the server, still got the missing boot sector message. I was on the phone to Dell for a couple of hours and we tried lots of different things but to no avail!!

 

I now need to rebuild and I am thinking of rebuilding back onto a member server. Does it matter that the previous mail server would ahve been a DC? If I restore what I have got from tape, will it get in a mess? What do I do about the fact that the original install of Exchange would have extended the AD Schema? Will a rebuild recognise this? What about all the user accounts - how will that work?

 

Sorry for loads of questions ....

[jj99]

Hmmm! What is your backup software, can you restore the mailboxes and system state speratly?

 

Its been a while since ive done anything like this but there probably a couple of ways you could do this. Restore the whole of you backup mailboxes and system state on the rebuilt server, and soilder on with the exchange/dc setup. maybe looking at fixing it later

 

Rebuild the server and just restore mailboxes on to it (youll probably have to call it the same as teh old one tho or it might through a wobbler!) You could then rename the server, mailbox stores etc I have doen this one before using Veritas backup and fortunatly it kept all the users mailbox info!

 

You could allways try creating a virtual server (vmware is now free) on another server restore your exchange on here the building a brand new exchange server on the rebuilt box. Then move all mailboxes off virtual server onto your new box, you could then take the chance to tidy up!

 

I hope these help, i dont envy your task!! Hopefully your users will pick up ther mailbox info (how many have you got btw) else you will be maunlly reconnecting :-S!!!

[brahma]

I've got around 1500 user accounts - including teachers and support staff!!!

 

To be honest, I need the simplest solution - I would like to stop the Exchange Server being a DC but if that's too complicated, I will keep it as a DC for now. I'm a bit rusty on rebuilding a DC anyway - haven't done it for a while!! Need to find a good article on the web that explains how to restore DC and Exchange.

[maniac]

AS I see it, it should be fairly simple.

 

First step, install Operating system and add to domain as a member server, ensure it is fully patched to the same level as previous server.

 

Second step, install exchange from origenal media, and ensure it is working 100% properly. Set up a test mailbox and ensure it is sending and receiving properly, you now have a functioning exchange server. The installation of exchange will sort out the extended schema for AD, although this should already be in place as your AD schema will be stored on your other DCs. Setup your mailbox store etc.

 

You should now be able to restore all your mailboxes and mail back into the mailbox store, and hopefully with any luck the associations within AD should still be there and it should work! If you find they don't, you may have to re-associate each mailbox with their relevant AD account, which might have to be done manually, but with anyluck if the new exchange server has the same credentials as the old one, it should work without having to do this.

 

There's a a basic guide on backup and restore of exchange using NT backup on the the msexchange help site, the basics should be the same whatever backup software you're using. Exchange 2003 Backup and Restore with NTBACKUP

 

Mike.

[srochford]

If it's just complaining about a missing boot sector, can you not boot from a floppy (format it on a 2003 server - that puts on the necessary boot files; you might need to edit the boot.ini file to point to the proper location)

 

If you can't boot it like that, can you PXE boot it with WinPE or boot from something like BartPE or UltimateBootCD and then get the data off the discs - I know it's likely to be big but you should be able to ghost it to somewhere and then just put it back on working hardware??

 

If you do have to rebuild from scratch then just don't make the server a DC - there's no problem at all with changing from Exchange on a DC to Exchange not on a DC after a disaster; it's just that the permissions fall to pieces if you dcpromo a machine which is actually running Exchange.

[Lee_K_81]

If you have 1500 user and backups arent working, you should prioritise. Do all staff use webmail or do they use outlook and use offline mailboxes. If they use offline mailboxes then all might not be lost. Check out your c:\documents and settings\%username%\local settings\application data\microsoft\outlook\ directory. Copy the offline files, you might be able to open them seperatly once you have restored exchange and individual mailboxes, and drag the other emails back to the server.

[zag]

ooOOh Disaster recovery, my favourite subject!!

 

To get your place up and running again:

 

- Build a new Member server (not a DC, that's a very bad idea for exchange)

- Name it something different than the previous server, but use the same external IP

- Get mail connectivity, test mail routes. (the exchange 2007 analysis tools are good for this)

- Restore the exchange store and personal mailboxes using your backup software. (lots of whitepapers on the net for this especially if you use backupexec)

 

And for the future...

 

- Don't use RAID5 on your operating system drive, only the data, I've seen so many problems as you describe, most actually caused by the RAID in the first place!

- Don't use exchange on a DC

- Image your exchange server every so often, Acronis is very good at this.

- Backup the data drive(exchange store) mid week, setup email alerts if this fails for any reason.

[brahma]

Hi All. Server hardware has difinitely gone so we are going to see what we can get off of our last restore. I am going to spec up a new server to replace this one - it had one Raid set with system drive, database and logs set up logically on this. I thought ideal config for this was O/S - Raid1, Database - Raid 5 and Logs - Raid1. So server would need to be able to hold 7 drives am I right? I am looking at Dell server for my 1500 users. Are we going in the right direction here?

[maniac]

You might get away with just 6 drives, i.e 3 pairs each in RAID1. 1 Pair for the system, 1 Pair for the Database and 1 Pair for the Logs. Because you can now buy drives in such large capacities, quite often RAID1 with 2 drives will do the job. I'm not sure what the performance of RAID1 compared to RAID5 is, but I don't think there's much in it.

 

The exchange server at my last place was set up so we had 2 drives in RAID1 for the system, and 4 Drives in RAID5 for the database and logs, which was partitioned into 2 logical disks. Not ideal, but it was the only way on that server we could get the size of partition we needed for our database. I'm not sure how the exchange server is set up at my current job, not got that far yet!

 

Mike.

[djm968]

I would go for Exchange 2007, there is lots of information here and you could repurpose Microsoft Exchange Server 2007 the old box with the front end client access server roll.

[jj99]

This is my setup for arround 3000 users: 1 x Del 2950 server as a mail hub transport (email in and out from internet ect and OWA access) then i have two Dell R900 servers set up in a cluster ie. they both have the same info on them thats repicated betwen them using a direct netrowk link. so if one goes down the other takes over.

 

The server R900 have 2 x 146gb sata scsi in a raid 1 for OS and prg files.

 

They have 4 x 300 gb sata scsi in raid 5 for exchange databases and logs, (i keep logs to minimum but backup everyday)

 

If i was in your shoes (not sure of your budget tho) id get one server (mayve a 2950 or r900 if you can afford) with 6 disks, same setup as one of my R900 above, some people recommend not putting logs and databases on raid 5 cos of disk write speeds, but i have setup anumber of excange servers and found it not to be a problem (when compared to the space you gain from raid 5 over raid 1 & 0).

 

Then build a simple member server put a new exchange setup on and restore mail stores on.

 

You could look in the future at adding a front end server maybe (hub transport owa etc) to take strain off.

[brahma]

Thanks for this JJ. Not sure if I can stretch to an r900 but should be able to get a 2950. Wasn't sure whether I should be looking at SAS drives or SATA - I'm more used to SCSI3!! What's the best / difference?

 

I quite like the idea of 6 drives and splitting O/S, database and logs. Although you guys think that I might get away with just a raid1 and a raid5?

[mattx]

- Don't use RAID5 on your operating system drive, only the data, I've seen so many problems as you describe, most actually caused by the RAID in the first place!

 

Been in the business of building servers etc for a long time now - never heard of 'Don't use RAID5 on your operating system drive'

The only problem I have had with RAID5 was on an old IBM720 running LAN Server - it was flagging parity errors on a HD so I changed the faulty drive [ on the fly ] and as the new one was re-building another one failed. Thinking I had lost the entire RAID I booted into the RAID5 options and re-initialized the RAID and booted and hey presto.

 

I would rather use RAID5 with a hot spare than say a Mirror [ personal preference mind you ]

 

The server had the ususal alarm that goes hand in hand with a disk failure but was up and running. Our analysts downed the server and removed the failed drive, replacing it with a good one.

 

This has me puzzled, could you not replace the drive on the fly ? Or don't Dell Raid Cards / Raid chipsets support this ?

Edited June 27, 2008 by mattx

[brahma]

Morning Mattx. Couldnt swap the drive on the fly. The previous IT team had this build on a server that didn't use hot swap drives (the Raid card is a Perc3C) - I think the server is around 8-10 years old!! Why on earth they installed Exchange 2003 on a box that old is beyond me!! I have worked with Perc 5C's and these do allow hot swapping. I always spec servers with hot swap drives - knowing how frequently you can get drive failures, I dont want to keep shutting servers down!

 

Can you throw any light on SCSI v SATA v SAS? Not sure what drives to spec with the new server!!

[mattx]

Can you throw any light on SCSI v SATA v SAS? Not sure what drives to spec with the new server!!

 

Give me a torch then.....

Instead of me ranting on about the differernces I'll let someone else do the talking for me.

[ BTW I was bought up on SCSI drives but have recently built a server with SATA connections - far easier dealing with the cables !! ]

 

SAS

 

Serial Attached SCSI (SAS) is a data transfer technology designed to move data to and from computer storage devices such as hard drives and tape drives. It is a point-to-point serial protocol that replaces the parallel SCSI bus technology and uses the standard SCSI command set. Designed to be used with mission critical systems such as servers, high-end work stations or any other users that require the best performance going to improve their systems performance.

 

SATA

 

These high capacity drives are designed to be used with todays newest connection - Serial ATA. Providing more bandwidth and advanced features such as NCQ, Staggered spin up and much more, these drives are ideal for todays systems which can have upto 8 SATA ports and support various RAID styles which are ideal for power users, people who want data security and much more.

 

SCSI 68 Pin

 

These hard-drives are at the forefront of performance. Ideal for the extreme gaming system or a system where data transfer is a must, the use of SCSI is mainly limited to the Server market but thanks to its high speeds and reliability its slow progressing across to desktop PC's of the highly enthusiastic gamers.

 

SCSI 80 Pin

 

Designed to give the ultimate performance, this range of drives is designed for reliability, speed and high data transfers. These drives have been designed to give the best performance and as such are aimed at the server markets these are ideal for high speed applications such as the server market, mission critical systems or other systems which require the best data performance available.

[brahma]

Hi All. Well at last we've got our server back up!! Slight problem now is that many users have not got all of their emails. I think they have got offline folders in use - if I check in Documents and Settings under the path "\Local Settings\Application Data\Microsoft\Outlook\", I have an outlook0.ost file (761KB) and an and outlook.ost file (33,745KB). I cant get access to the outlook.ost file!! If I try and add another Data File, I can only load a .PST - how do I load the original outlook.ost? If I select Data Files, Settings, Advanced and click the Offline Folder File Settings button, I dont get an option to change location - the browse button is "greyed" out!! Outlook Client is 2007

 

Any help much appreciated!!