+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 23
Enterprise Software Thread, We've been hacked... in Technical; It would appear that someone has brute forced some part of our mail server over the weekend, we arrived this ...
  1. #1

    Oaktech's Avatar
    Join Date
    Jul 2011
    Location
    Bournemouth
    Posts
    3,217
    Thank Post
    943
    Thanked 645 Times in 504 Posts
    Rep Power
    288

    We've been hacked...

    It would appear that someone has brute forced some part of our mail server over the weekend, we arrived this morning to find the mail server on its knees with a full HDD.

    It appears to be attempting to send thousands of messages from info3@lce.net. We immediately disabled outbound mail.

    We've scanned all of our servers and come up with nothing,
    We've scanned all active clients and come back with nothing.
    We've scanned the mail server itself and come back with nothing

    We've removed default gateway from the mail server and it continues to fill up,
    We've changed the IP of the mailserver, no-one can connect to it either externally or internally except by TSC on the new IP or VMconsole

    If we allow the services to run (all exchange services, smtp and IIS which it depends on) the queues continue to fill up with crap

    I've tried to run the aqadmcli.exe script to clear all messages from this sender and it just keeps generating more mail.

    Where the hell is it coming from and how the bloody hell do I stop it.

  2. #2

    Domino's Avatar
    Join Date
    Oct 2006
    Location
    Bromley
    Posts
    4,126
    Thank Post
    217
    Thanked 1,353 Times in 826 Posts
    Blog Entries
    4
    Rep Power
    528
    If you open the queue viewer up can you see if the messages are coming from a single source IP? It certainly sounds like an internal client may have been compromised

  3. #3

    Oaktech's Avatar
    Join Date
    Jul 2011
    Location
    Bournemouth
    Posts
    3,217
    Thank Post
    943
    Thanked 645 Times in 504 Posts
    Rep Power
    288
    I don't think its coming from an internal client as we have isolated the box by changing IP and removing default gateway thus rendering the box not contactable by clients... The messages do not appear to have a client IP, they seem to appear from within the mail server.

  4. #4

    Join Date
    Jul 2006
    Location
    London
    Posts
    1,279
    Thank Post
    115
    Thanked 247 Times in 197 Posts
    Blog Entries
    1
    Rep Power
    76
    Quote Originally Posted by Oaktech View Post
    I don't think its coming from an internal client as we have isolated the box by changing IP and removing default gateway thus rendering the box not contactable by clients... The messages do not appear to have a client IP, they seem to appear from within the mail server.
    Call Microsoft they have a team for this.

  5. #5

    Join Date
    Mar 2013
    Location
    west sussex
    Posts
    519
    Thank Post
    74
    Thanked 26 Times in 26 Posts
    Rep Power
    15
    can you wireshark to see if something is connected to the machine via smtp
    also try netstat to examine exe's making TCP connections.

    If i was near i would quite happily help you with this.

  6. #6

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,772
    Thank Post
    1,308
    Thanked 804 Times in 698 Posts
    Rep Power
    247
    Quote Originally Posted by Oaktech View Post
    Where the hell is it coming from and how the bloody hell do I stop it.
    Are you sure your Exchange server isn't set as an open relay? Googling for "exchange open relay" should get you an explanation, some documentation and some external testing tools for you to check with. User @sukh has been very, very helpful in the past with Exchange issues.

  7. #7

    Join Date
    Jul 2010
    Posts
    106
    Thank Post
    0
    Thanked 14 Times in 14 Posts
    Rep Power
    12
    Quote Originally Posted by Oaktech View Post
    Where the hell is it coming from and how the bloody hell do I stop it.
    Is it that you have had so much mail that exchange has created a lot of logs and these are still getting played into the databases??

  8. #8

    Oaktech's Avatar
    Join Date
    Jul 2011
    Location
    Bournemouth
    Posts
    3,217
    Thank Post
    943
    Thanked 645 Times in 504 Posts
    Rep Power
    288
    Quote Originally Posted by dhicks View Post
    Are you sure your Exchange server isn't set as an open relay? Googling for "exchange open relay" should get you an explanation, some documentation and some external testing tools for you to check with. User @sukh has been very, very helpful in the past with Exchange issues.
    I've used mxtoolbox and test smtp and both said not an open relay...
    @psydii do you have a number to call, we don't have any kind of support with MS?

  9. #9

    Join Date
    Jul 2006
    Location
    London
    Posts
    1,279
    Thank Post
    115
    Thanked 247 Times in 197 Posts
    Blog Entries
    1
    Rep Power
    76
    @Oaktech: https://support.microsoft.com/oas/de...direct=1&sd=gn

    I'm on the tube so can't give you much more than that at the moment.

    It will cost £200 + Irish VAT
    Last edited by psydii; 20th August 2013 at 11:06 AM.

  10. #10

    Domino's Avatar
    Join Date
    Oct 2006
    Location
    Bromley
    Posts
    4,126
    Thank Post
    217
    Thanked 1,353 Times in 826 Posts
    Blog Entries
    4
    Rep Power
    528
    Quote Originally Posted by Oaktech View Post
    I don't think its coming from an internal client as we have isolated the box by changing IP and removing default gateway
    You've no other machines on the same subnet as the exchange box?

  11. #11

    AngryTechnician's Avatar
    Join Date
    Oct 2008
    Posts
    3,730
    Thank Post
    698
    Thanked 1,214 Times in 761 Posts
    Rep Power
    395
    Personally I would remove the VMDKs from the VM for later diagnosis and restore the mail server from a backup from before the problem started. You'll lose a couple of days mail but you'll have your mail server back a lot quicker. Just be sure to go over patches and firewall with a fine-toothed comb to make sure it isn't immediately re-compromised, and you can then dissect the problem a bit more calmly on an isolated VM with the original disks.

  12. #12

    Oaktech's Avatar
    Join Date
    Jul 2011
    Location
    Bournemouth
    Posts
    3,217
    Thank Post
    943
    Thanked 645 Times in 504 Posts
    Rep Power
    288
    Ok, Update.

    As our normal support channel was looking like costing a bloody fortune to fix this, we've just made a new exchange 2010 server, transferred everyone's mail boxes and we are ironing out the wrinkles with that now.

  13. #13

    Dos_Box's Avatar
    Join Date
    Jun 2005
    Location
    Preston, Lancashire
    Posts
    9,442
    Thank Post
    701
    Thanked 2,302 Times in 1,063 Posts
    Blog Entries
    23
    Rep Power
    678
    Is it to late to suggest that you buy a TechNet subscription and use your free support calls? It's a lot cheaper than buying in support.
    Microsoft dug out a new Exchange 2007 install years back for me after they made some changes to the .Net framework and hadn't yet realised it's knock-on effect to Exchange. I used a support call from my TN subscription and it took them about 7 hours

  14. #14

    witch's Avatar
    Join Date
    Nov 2005
    Location
    Dorset
    Posts
    11,521
    Thank Post
    1,532
    Thanked 2,639 Times in 1,827 Posts
    Rep Power
    814
    @Oaktech
    Wasn't me, honest!
    Hope it is all sorted. So what are you going to do with the old one?Nuke it?

  15. #15

    Join Date
    Jul 2006
    Location
    London
    Posts
    1,279
    Thank Post
    115
    Thanked 247 Times in 197 Posts
    Blog Entries
    1
    Rep Power
    76
    Good to hear that you're stabilised, clean build is usually the best way.

    I was chatting to a MS chap (trying to get our licensing sorted after Capita botched it, again) and he said: register Technet Plus benefits as the first thing you do... it takes a couple of days for your free support calls become available, and that's two days too long if you are down and haven't pre-registered.

    Got to say MS Exchange Support are awesome and fast and work round the clock (11pm call back, proper managed handover between the US and India teams - really awesome to watch). Even though I pay for support from a IT Services company, if I were in a 'critical incident' (where the skills needed are deep product knowledge, rather than a holistic understanding of the local configuration) I'd happily drop the cash on a MS call rather than chance my arm with a generalist at the day to day support company.
    Last edited by psydii; 21st August 2013 at 12:46 PM.



SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Replies: 2
    Last Post: 9th January 2013, 05:08 PM
  2. Have we been hacked?
    By wesleyw in forum Virtual Learning Platforms
    Replies: 13
    Last Post: 22nd June 2009, 11:11 PM
  3. Microsoft.com Has Been Hacked
    By FN-GM in forum Web Development
    Replies: 2
    Last Post: 26th July 2007, 09:54 PM
  4. Think I've been stung on Ebay
    By park_bench in forum General Chat
    Replies: 18
    Last Post: 4th July 2007, 12:18 PM
  5. We have been offered
    By Elky in forum Our Advertisers
    Replies: 11
    Last Post: 26th March 2007, 02:32 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •