We've been hacked...

[Oaktech]

It would appear that someone has brute forced some part of our mail server over the weekend, we arrived this morning to find the mail server on its knees with a full HDD.

 

It appears to be attempting to send thousands of messages from [email protected]. We immediately disabled outbound mail.

 

We've scanned all of our servers and come up with nothing,

We've scanned all active clients and come back with nothing.

We've scanned the mail server itself and come back with nothing

 

We've removed default gateway from the mail server and it continues to fill up,

We've changed the IP of the mailserver, no-one can connect to it either externally or internally except by TSC on the new IP or VMconsole

 

If we allow the services to run (all exchange services, smtp and IIS which it depends on) the queues continue to fill up with crap

 

I've tried to run the aqadmcli.exe script to clear all messages from this sender and it just keeps generating more mail.

 

Where the hell is it coming from and how the bloody hell do I stop it.

[Domino]

If you open the queue viewer up can you see if the messages are coming from a single source IP? It certainly sounds like an internal client may have been compromised

[Oaktech]

I don't think its coming from an internal client as we have isolated the box by changing IP and removing default gateway thus rendering the box not contactable by clients... The messages do not appear to have a client IP, they seem to appear from within the mail server.

[psydii]

I don't think its coming from an internal client as we have isolated the box by changing IP and removing default gateway thus rendering the box not contactable by clients... The messages do not appear to have a client IP, they seem to appear from within the mail server.

 

Call Microsoft they have a team for this.

[ConradJones]

can you wireshark to see if something is connected to the machine via smtp

also try netstat to examine exe's making TCP connections.

 

If i was near i would quite happily help you with this.

[dhicks]

Where the hell is it coming from and how the bloody hell do I stop it.

 

Are you sure your Exchange server isn't set as an open relay? Googling for "exchange open relay" should get you an explanation, some documentation and some external testing tools for you to check with. User @sukh has been very, very helpful in the past with Exchange issues.

[apeman]

Where the hell is it coming from and how the bloody hell do I stop it.

 

Is it that you have had so much mail that exchange has created a lot of logs and these are still getting played into the databases??

[Oaktech]

Are you sure your Exchange server isn't set as an open relay? Googling for "exchange open relay" should get you an explanation, some documentation and some external testing tools for you to check with. User @sukh has been very, very helpful in the past with Exchange issues.

 

I've used mxtoolbox and test smtp and both said not an open relay...

@psydii do you have a number to call, we don't have any kind of support with MS?

[psydii]

@Oaktech: https://support.microsoft.com/oas/default.aspx?&gprid=13965&st=1&wfxredirect=1&sd=gn

 

I'm on the tube so can't give you much more than that at the moment.

 

It will cost £200 + Irish VAT

Edited August 20, 2013 by psydii

[Domino]

I don't think its coming from an internal client as we have isolated the box by changing IP and removing default gateway

 

You've no other machines on the same subnet as the exchange box?

[AngryTechnician]

Personally I would remove the VMDKs from the VM for later diagnosis and restore the mail server from a backup from before the problem started. You'll lose a couple of days mail but you'll have your mail server back a lot quicker. Just be sure to go over patches and firewall with a fine-toothed comb to make sure it isn't immediately re-compromised, and you can then dissect the problem a bit more calmly on an isolated VM with the original disks.

[Oaktech]

Ok, Update.

 

As our normal support channel was looking like costing a bloody fortune to fix this, we've just made a new exchange 2010 server, transferred everyone's mail boxes and we are ironing out the wrinkles with that now.

[Dos_Box]

Is it to late to suggest that you buy a TechNet subscription and use your free support calls? It's a lot cheaper than buying in support.

Microsoft dug out a new Exchange 2007 install years back for me after they made some changes to the .Net framework and hadn't yet realised it's knock-on effect to Exchange. I used a support call from my TN subscription and it took them about 7 hours

[witch]

@Oaktech

Wasn't me, honest!

Hope it is all sorted. So what are you going to do with the old one?Nuke it?

[psydii]

Good to hear that you're stabilised, clean build is usually the best way.

 

I was chatting to a MS chap (trying to get our licensing sorted after Capita botched it, again) and he said: register Technet Plus benefits as the first thing you do... it takes a couple of days for your free support calls become available, and that's two days too long if you are down and haven't pre-registered.

 

Got to say MS Exchange Support are awesome and fast and work round the clock (11pm call back, proper managed handover between the US and India teams - really awesome to watch). Even though I pay for support from a IT Services company, if I were in a 'critical incident' (where the skills needed are deep product knowledge, rather than a holistic understanding of the local configuration) I'd happily drop the cash on a MS call rather than chance my arm with a generalist at the day to day support company.

Edited August 21, 2013 by psydii

[Oaktech]

@Oaktech

Wasn't me, honest!

Hope it is all sorted. So what are you going to do with the old one?Nuke it?

 

we'll archive the VM components, delete them from the VM Host, keep them for a year somewhere else and then delete that too.

@psydii our normal support company are the ones we call in a dire emergency, and they are have full product specialists which is why they were so damn expensive... But yes, I take your point about the technet benefits. I'm going to look into it as we've never used it before.

[plexer]

The only thing with the complimentary support calls provided are these two conditions:

Resolve technical break/fix issues in a non-production environment.

Find a solution to a production issue that can be replicated in a non-production environment.

Although MS may not require you to do the 2nd before helping you out from what others have said.

 

Ben

[Dan_ATR]

Good excuse to install out new 2010 box I say Taking ages for our users to transferee across ... zz,z,,z,,,.z.z.z.z.z

[Dan_ATR]

@Oaktech (Thank Post 460) I wish I got these above my head

 

Might grab a few when your back in tomorrow when your not looking he he he ....

[EduTech]

Hi OakTech,

 

If you want me to have a look at this for you then drop me a PM more than happy to give your old box a once over to try and give you some idea on how it has happened, also if you need a hand ironing out any issues then just let me know. I am pretty free this week so should be able to spare you some time.

 

Regards, James.

[MatthewL]

Just spoke to someone who had the same issue, turned out to be a trojan in a users home directory (SBS server) that was causing it, the AV didn't pick it up.

 

They ran a netstat -p which led them to it.

[Oaktech]

Interesting... we did find 1 dubious file in a users home drive that ESET missed for some reason, but spybot picked up. It didn't give an indication that it would have caused the symptoms we saw, spybot suggested it was more adware than anything, but of course we deleted it - and gave the user a stern talking to about bringing in things from home. It was a copy of something called "unzipit professional" which does what it says it does, and ran somehow even though we have software execution policies in place. so we are examining that now too.

[sukh]

If no open relay then it just says to me that your Exch server was not protected enough by AV/AS?

 

What product was you using?