Exchange 2003 Disk Space problem and Log Files

[MrWu]

Hi all;

 

I've got a sticky situation at school whereby the Exchange Server (2003) has about couple of weeks life left in it.

 

When I started the job a month ago I discovered that it has a RAID disk fault and the DB hasn't been backed up for almost a year (the Event viewer is also flagging inconsistencies in the DB)

 

Obviously the Transaction Logs have built up (80gb worth !) and is chewing up disk space on the C Drive.

 

My challange being I am unable to back up the DB (I've tested backing up a small file (50mb) on NT Backup and it froze) the server is sluggish and copying say 4 gb worth of data onto D Partition takes 30 minutes. I suspect this is due to the RAID fault

 

So I'm worried the server will crash doing any disk intense activity (ie moving the Transaction log files path, or trying a full backup)

 

My question is thus, I know it's a bad idea to delete log files manually but how likely are log files from say back in April 2011 are still not committed to the DB?

 

I suppose I could dismount the DB and run eseutil to see what log files are needed, but would it remount again?? Damn I can't do a backup either !

 

 

We will be migrating next week to a new Exchange 2010 server but I read in MS that when migrating mailboxes you will get a big increase in log file size at the Exchange Server 2003 side as well as 2010? (in some cases 1:1 they say which means transfering mailboxes would fill up Exchange 2003's log disk !

 

No mans land really lol.

[SYNACK]

Take it offline and copy manually? One thing that I have done to quickly clean up such a mess is to turn on circular logging, it does cut down on the resilience especially if you have no backup but it gets the disk space under control really quick.

 

Microsoft Exchange Server 2003 - Circular Logging of Storage Groups

 

How to remove Exchange Server transaction log files

 

Transaction Logs, The Lifeblood of Exchange

[glennda]

As you have mentioned in you other post you have a virtual host (although an ML150) you could try the Physical to virtual converter and test running it on that - it should remove any Raid problems.

[m25man]

If its really that bad I would try and make it as easy as possible for your self.

 

Ive been involved in several Exchange recovery scenarios where the disk has been allowed to run out of space, here are a few ideas you might want to consider all have some quirks but I dont think we have ever lost any data in the process.

 

1. Depending on how many users you have backup each users mailbox to a PST file. (I have an Exchange Administrators Account for this I log in and open the target users mailbox and export everything to a PST file) We normally use this when archiving a user mailbox that has left the company freeing up the license for the incoming user.

If all else fails this can be used to recover user mailboxes after a clean install.

 

2. Take the store offline and copy everything to another disk. The DB file can be mounted and read by several proprietory recovery tools in the event of a disaster.

 

3. Use NTFS Compression to shrink the size of the log files this can recover a lot of space.

 

4. Install a nice big SATA Disk and move the DB and Log files onto that How to move Exchange databases and logs in Exchange Server 2003

 

5. Do not use the ESEUTIL on your live database always test it on a copy first know what your dealing with first.

 

In some cases its far easier to just use the PST dump method uninstall Exchange connect everyone to the nice shiny new 2010 one and import the old stuff or just leave it as an Archive PST file in each users Outlook, they soon get used to it.

The secret with Exchange is plan ahead work through all of the possibilities and always have a get out of jail free card.

[steve]

Who uses the exchange box? If just staff I'd be tempted to advise management of the problem, and through them make all staff aware of the issues.

 

As part of the advice to staff I'd suggest to them they delete all the old junk they have and backup any important attachments.

 

This should cover your back then if the worst happens - given you've only been there a month you need to make management and staff it might get worse before it get better.

 

On the technical side, my first thought is to try and do a physical to virtual conversion. How easy this will be depends on your RAID set-up. There are some free vmware and hyper-v tools to do this from a boot cd. The major problem here might be the RAID fault that screws it up mid conversion.

 

I'd then run the migration from the VM, first doing DB checks on this maybe using snapshots (not sure how exchange 2003 behaves with snapshots though). If the worst happens at least you'd have the physical machine to revert back to.

[bodminman]

ntbaclup will do bugger all anyway really. Get yourself an evaluation of BackupAssist for now. This should get you a backup and truncate the log files.

[PiqueABoo]

Eh? On a happy system NtBackup will do exchange-aware backup and truncate logs.

 

--

 

You never know, but sometimes on a poorly maintained E2K3 server there are truckloads of *IIS* logfiles (from OWA access) that could be deleted to make some space - I've seen boxes with more than 10GB of those.

 

Other than that, I think the OP has talked themselves into a bit of a corner. If the concern is intensive disk I/O crashing the system then doing *anything* is a risk, but Exchange does some serious disk I/O at the best of times and the box is still running so perhaps that risk isn't that huge? Not enough detail/info on the disk problems with that box, but before doing anything else, as insurance I'd probably want to try getting high value mailboxes (e.g. SMT, key admin) backed up to PSTs on an external HDD and would use Exmerge for that... I'd do it cautiously, try one first, then two more, then four.. watch disk space etc.

[MrWu]

Thank you all for your help, I will look at some options tomorrow and roll with it.

[MrWu]

Eh? On a happy system NtBackup will do exchange-aware backup and truncate logs.

 

--

 

You never know, but sometimes on a poorly maintained E2K3 server there are truckloads of *IIS* logfiles (from OWA access) that could be deleted to make some space - I've seen boxes with more than 10GB of those.

 

Other than that, I think the OP has talked themselves into a bit of a corner. If the concern is intensive disk I/O crashing the system then doing *anything* is a risk, but Exchange does some serious disk I/O at the best of times and the box is still running so perhaps that risk isn't that huge? Not enough detail/info on the disk problems with that box, but before doing anything else, as insurance I'd probably want to try getting high value mailboxes (e.g. SMT, key admin) backed up to PSTs on an external HDD and would use Exmerge for that... I'd do it cautiously, try one first, then two more, then four.. watch disk space etc.

 

Thanks Piqueboo

 

Yes I did clear 5 Gigs of IIS Logs last week, which bought some breathing space (it was now 7 gigs free) Exchange 2003 is generating 500mb of logs per day during work time so it won't last.

 

Talking to the existing techs the Exchange server RAID 10 drives suffered a crash, and the RAID 10 mirror is running on a degraded state, hence slow down, ask why another disk wasn't put in for the rebuild they said it's been done twice and Exchange either takes days to rebuild or even when successful the drive will fail after a few weeks. The Ex Network manager had a consultant in to build an Exchange 2010 server to migrate mailboxes, it wasn't done straight away so that brings us to the present, I have test migrated a few mailboxes but found that it didn't integrate with Frog too well etc and wanted a few more weeks testing before mass migration. But with the disk space issue now I need to move quick (with hindsight I should have spotted this the first week I started)

 

Very sensible idea about backing up VIPs to PSTs :-) I will do that but be mindful of disk space ( exmerge creates Transaction logs too?) and also migrate users this week who just uses an Outlook Client (rather than Frog,Activesync etc)

One thing that still puzzles me is according to microsoft, if you migrate mail 2003>2010, you generate as much log files at source and also destination? dicussed here but maybe refuted: 2003 to 2010 Mailbox Migration - Are log files sized 1:1 with mailbox on the source server or just the destination server?

 

Also, would log files from April last year be possibly comitted on Exchange 2003 already? (or not take that risk) I was thinking moving a small chunk of last year Trans log to the D Partition.

 

Otherwise I think the only way is to run eseutil to see which logs are not needed.

[sukh]

1. Those transaction would have been played in the DB. You can go ahead and remove them. If you want to be safe then copy them to another location if you feel uncomfortable.

2. Reduce the logs down to around the last 4 weeks and then perform a FULL backup.

[bodminman]

Eh? On a happy system NtBackup will do exchange-aware backup and truncate logs.

 

 

Thanks for that. I was told several years ago to not bother with ntbackup on Exchange because it didn't truncate logs. I just took it as gospel as the guy who told me was supposedly sh1t hot on all the Exchange stuff!

 

Live & Learn I suppose.

[SYNACK]

Thanks for that. I was told several years ago to not bother with ntbackup on Exchange because it didn't truncate logs. I just took it as gospel as the guy who told me was supposedly sh1t hot on all the Exchange stuff!

 

Live & Learn I suppose.

 

ntbacup has had an exchange aware agent plugin since about ex2k and is often the first resort to getting a good backup out as it is native (programmed by the MS team) so is usually the most resilient (or at least was) way to get a clean backup out and truncate the logs. It took some time for MS to de-corporate-douchbag themselves and release a plugin for 2007 but hopefully they have learnt their lesson for the time being.

[sukh]

Some catches like, you can only do FULL backups and not INC.

[PiqueABoo]

One thing that still puzzles me is according to microsoft, if you migrate mail 2003>2010, you generate as much log files at source and also destination? dicussed here but maybe refuted: 2003 to 2010 Mailbox Migration - Are log files sized 1:1 with mailbox on the source server or just the destination server?

 

According to MS where? I'm no expert on what MS do, but in a sane universe you only keep DB *changes* in transaction logs. If I send you a 10MB mail then all of that 10MB needs to go in the log because of all that is added to the DB, but if you then read the mail the only DB changes needed might be some relatively little ones re. metadata e.g. updates to last accessed, whom by, other housekeeping. So my theory is that transaction logs on the target server are going to grow to more or less the combined size of the mailboxes moved there, but the source Exchange 2003 transactions logs will grow at a much smaller rate (metadata changes, post-move deletes etc.)

 

PS: If you do any SMT et al PST exports, tell one or more of them you're doing it - might help avoid any potential future misunderstandings about why you've got copies of the orgs most sensitive mailboxes sat around somewhere.

Edited January 8, 2012 by PiqueABoo

[MrWu]

According to MS where? I'm no expert on what MS do, but in a sane universe you only keep DB *changes* in transaction logs. If I send you a 10MB mail then all of that 10MB needs to go in the log because of all that is added to the DB, but if you then read the mail the only DB changes needed might be some relatively little ones re. metadata e.g. updates to last accessed, whom by, other housekeeping. So my theory is that transaction logs on the target server are going to grow to more or less the combined size of the mailboxes moved there, but the source Exchange 2003 transactions logs will grow at a much smaller rate (metadata changes, post-move deletes etc.)

 

PS: If you do any SMT et al PST exports, tell one or more of them you're doing it - might help avoid any potential future misunderstandings about why you've got copies of the orgs most sensitive mailboxes sat around somewhere.

 

This is what confused me:

 

For every gigabyte of data that you move, an additional gigabyte of transaction logs is generated at the source and target server. Verify that you have sufficient free space on your transaction log drives. If you do not have sufficient free space on your transaction log drive for transaction log file generation

 

Moving mailboxes in Exchange Server 2003 (PS I may have read this in the wrong context, this was for EX2000 > EX2003 sorry)

 

I didn't think it made sense either, but your explanation makes more real world sense.

 

Thanks will inform SMTs that I'll be doing an Exmerge, they would probably then ask would it effect their whiteboards lol

[PiqueABoo]

For every gigabyte of data that you move, an additional gigabyte of transaction logs is generated at the source and target server.

 

That's nicely ambiguous, no way to be sure whether that's combined (with no indication to the distribution on either side) or two independent gigabytes. I suppose it's possible that the overhead for the deleting etc. on the source server is big enough per message, for the growth to still be quite significant e.g. move a few big messages and you don't notice much, but move lots of little ones and they add up. Think you'll have to move an initial few mailboxes and let us know the outcome!

 

Note, I think @sukh's comments (1, 2) about deleting the older transaction logs are sound and vaguely recall doing that somewhere once, wasn't confident enough about that recall to tell you to go ahead and do it though.

[sukh]

When you move mailboxes a GB will be at the target if not more, but NOT at the source, I have never seen this in my experience or have seen this stated anywhere.

However, a tiny amount MB if that will be at the source.

[MrWu]

You guys Rock! Thank you...

 

Moved 3 Mailboxes tonight, all under 400mb in size and up to 6000 items, transfered in about 18 minutes. exchange 2003 log files increased no more than 20mb, as predicted Exchange 2010 logfiles increased about 800mb (but we do have 1.6tb free)

 

Now to test usability etc for those users tomorrow and will start migrating for sure this week, thank you guys so much!