We have about 400 or so winXP machines running on w2k servers.
Back in September of last year we were hit with the SDbot virus which infected all the clients. In addition we discovered a root kit on some of the workstations.
We managed to get the network clean with the exception of the following strange behaviour.
Doing some routine scans we discovered that some of the clients were reporting their DNS names as variations of the following
The -2-142 would be the network part of our ipaddress and the rest resolved to The Carnegie Mellon University in Pittsburg USA.
Looking in DNS we could see that these addresses were in the DNS cache which we cleared (on the server that is). This brought temporary relief but the Gruel addresses soon returned even on clients that were re-imaged leading us to believe that the problem related to DNS itself. Some form of poisoning ???
Despite this worrying trend the network was working OK and I’d attempt to do some research into this whenever I got a spare minute.
However, things began to change about two weeks ago.
ActiveD showed some strange behaviour.
The DHCP scope disappeared without trace one day followed by all the forwarders having been taken out of DNS on another day.
We have some additional machines in the DNS cache now which are in the same EDU.CMU.ANDREW. tree and say they are DDNS.
There are A records which call themselves things like ddns-master with an IP in our range. And NS servers called things like ddns-a100.net.cmu.edu.
Most worrying of all the server decided to stop playing today and no one could connect to network shares and hence were unable to login. The server ground to a halt and it needed a reboot to get things going again. There does not seem to be anything in the event logs worth mentioning. Hardware is OK - I'm thinking something overwhelmed SMB.
Any ideas on how I can clean DNS or even to try and get a better understanding of what is actually going on.
Your network is compromised. You have no idea where or how. Therefore, you cannot trust any of your systems. As such:
1. Backup your user's data (after a virus scan. You might want to miss out executables any other nasties just in case) and any configuration settings you need (GPO's, etc).
2. Restore all your servers from the last known good backups (pre-september).
3. Reimage all your client PCs from a known good image (again, pre-september).
4. Evaluate how the virus got in to prevent it happening again.
I've done security forever and TBH I can't figure out from your post whether your network is seriously compromised or just broken. However if Geoff is right and you do follow his advice, I recommend step 4 *first* otherwise you might find it gets compromised all over again before you've finished reimaging.
Just to Add to Geoff and PiqueABoo ... turn off and remove the network leads of all your workstations and servers and only plug them back onto the network after you have rebuilt them from a good configuration and patched them.
A big job you have there ... best of look with it all. If you are looking for possible vectors of attack I would look at staff laptops, and also make your firewall a little overzealous until you are happy things are back to normal.
When I started here in the summer the staff used laptops with full admin rights and blank passwords to boot. The amount of malware that came in once term started had to be seen to be believed. I put a stop to that by reimaging all laptops and locked them down. Some staff still won't talk to me accusing me of being paranoid.
This job really is between a rock and a hard place.
Just for info, when the virus first kicked off I advised just what Grumbledook suggested but the head overruled me. Then by christmas the network was working fine as far as performance was concerned. It's only recently that server behaviour indicates that its been compromised big time.
I think I'm gonna have to bite the bullet.
Anyone fancy a busmans holiday in York LOL ;-)
Agreed. It's difficult to judge from a distance, but if something sufficiently strange happened on mine I'd take the phone off the hook, pull the WAN link, verify the servers were free of any active malware, change the admin password(s) and then set about trying to figure out what happened.
In this particular case, the right details aren't there so I can't judge what going on with the DNS cache, scans and clients reporting names. Could be a red herring coz if you've got an authoritative reverse zone then I don't think your IP-to name mappings will not appear in the cache.
Forwarders & scopes both disappering is less easy to dismiss i.e. the most likely reason is because someone has access and removed them.
It's the verify no malware which I'm finding a bit tricky. I've run all sorts of stuff - nmap to see whats listening, rootkit revealer etc and they all come up clean. nevertheless, this does not mean to say that it is in fact clean.
Not sure exactly what you mean about the mappings. The cache appears to look normal in my limited experience with exception of these rogue GRUEL entries and whats with the DDNS in the cache?
I could mail details over to someone if they are willing to have a quick look for me and see what they think?
Unless I can actually nail this one and say with a degree of certainty that I know whats going on -then as you guys have said - I can't trust it.