Is this DNS poisoning?

[darkstar]

We have about 400 or so winXP machines running on w2k servers.

Back in September of last year we were hit with the SDbot virus which infected all the clients. In addition we discovered a root kit on some of the workstations.

 

We managed to get the network clean with the exception of the following strange behaviour.

Doing some routine scans we discovered that some of the clients were reporting their DNS names as variations of the following

GRUEL-2-142.PPP.andrew.cmu.edu

 

The -2-142 would be the network part of our ipaddress and the rest resolved to The Carnegie Mellon University in Pittsburg USA.

 

Looking in DNS we could see that these addresses were in the DNS cache which we cleared (on the server that is). This brought temporary relief but the Gruel addresses soon returned even on clients that were re-imaged leading us to believe that the problem related to DNS itself. Some form of poisoning ???

 

Despite this worrying trend the network was working OK and I’d attempt to do some research into this whenever I got a spare minute.

However, things began to change about two weeks ago.

 

ActiveD showed some strange behaviour.

The DHCP scope disappeared without trace one day followed by all the forwarders having been taken out of DNS on another day.

We have some additional machines in the DNS cache now which are in the same EDU.CMU.ANDREW. tree and say they are DDNS.

There are A records which call themselves things like ddns-master with an IP in our range. And NS servers called things like ddns-a100.net.cmu.edu.

 

Most worrying of all the server decided to stop playing today and no one could connect to network shares and hence were unable to login. The server ground to a halt and it needed a reboot to get things going again. There does not seem to be anything in the event logs worth mentioning. Hardware is OK - I'm thinking something overwhelmed SMB.

 

Any ideas on how I can clean DNS or even to try and get a better understanding of what is actually going on.

[Geoff]

Your network is compromised. You have no idea where or how. Therefore, you cannot trust any of your systems. As such:

 

1. Backup your user's data (after a virus scan. You might want to miss out executables any other nasties just in case) and any configuration settings you need (GPO's, etc).

2. Restore all your servers from the last known good backups (pre-september).

3. Reimage all your client PCs from a known good image (again, pre-september).

4. Evaluate how the virus got in to prevent it happening again.

[ChrisC]

That sounds like the best course of action to me.

 

Chris

[PiqueABoo]

I've done security forever and TBH I can't figure out from your post whether your network is seriously compromised or just broken. However if Geoff is right and you do follow his advice, I recommend step 4 *first* otherwise you might find it gets compromised all over again before you've finished reimaging.

[GrumbleDook]

Just to Add to Geoff and PiqueABoo ... turn off and remove the network leads of all your workstations and servers and only plug them back onto the network after you have rebuilt them from a good configuration and patched them.

 

A big job you have there ... best of look with it all. If you are looking for possible vectors of attack I would look at staff laptops, and also make your firewall a little overzealous until you are happy things are back to normal.

[Geoff]

I would look at staff laptops

 

Unencrypted wifi AP's are a favorite of mine atm.

[GrumbleDook]

I was asked at Manglement Meeting tonight why I didn't just tell the VIth formers the WEP thing so they can use their own laptops ...

 

I asked if she wanted them to know what she orders online since we will have no control over the machines at all ...

 

Strangely enough she backed down and wants me to explained things to the VIth formers ...

[darkstar]

yeah

When I started here in the summer the staff used laptops with full admin rights and blank passwords to boot. The amount of malware that came in once term started had to be seen to be believed. I put a stop to that by reimaging all laptops and locked them down. Some staff still won't talk to me accusing me of being paranoid.

This job really is between a rock and a hard place.

 

Just for info, when the virus first kicked off I advised just what Grumbledook suggested but the head overruled me. Then by christmas the network was working fine as far as performance was concerned. It's only recently that server behaviour indicates that its been compromised big time.

I think I'm gonna have to bite the bullet.

Anyone fancy a busmans holiday in York LOL ;-)

[GrumbleDook]

Tempting ... very tempting ...

 

I charge a very reasonable rate ... unfortunately my Head doesn't. Seems to think that if I am out of school the school should be reimbursed.

 

Can it wait until the Easter hols? (stupid question I know ... but have to ask!)

[E1uSiV3]

Im on holiday next week...

[PiqueABoo]

remove the network leads

 

Agreed. It's difficult to judge from a distance, but if something sufficiently strange happened on mine I'd take the phone off the hook, pull the WAN link, verify the servers were free of any active malware, change the admin password(s) and then set about trying to figure out what happened.

 

In this particular case, the right details aren't there so I can't judge what going on with the DNS cache, scans and clients reporting names. Could be a red herring coz if you've got an authoritative reverse zone then I don't think your IP-to name mappings will not appear in the cache.

 

Forwarders & scopes both disappering is less easy to dismiss i.e. the most likely reason is because someone has access and removed them.

[darkstar]

mmmmmmm

It's the verify no malware which I'm finding a bit tricky. I've run all sorts of stuff - nmap to see whats listening, rootkit revealer etc and they all come up clean. nevertheless, this does not mean to say that it is in fact clean.

Not sure exactly what you mean about the mappings. The cache appears to look normal in my limited experience with exception of these rogue GRUEL entries and whats with the DDNS in the cache?

 

I could mail details over to someone if they are willing to have a quick look for me and see what they think?

Unless I can actually nail this one and say with a degree of certainty that I know whats going on -then as you guys have said - I can't trust it.

[Geoff]

You have to rebuild everything, you don't have a choice in the matter. From both a technical and legal point of view.

[darkstar]

Actually, I've been running this past the head of ICT and I think we're gonna flatten everything. This will have to wait till the summer though.

[Geoff]

I'll let the Data Protection Office know.

[PiqueABoo]

Ok I miss-typed that "mapings" comment but basically if I do this:

 

> nslookup 128.2.2.142

 

Then I get all sorts of stuff in my cache which is *perfectly normal*:

 

--forward zone edu.net

NS ac-ddns2.net.cmu.edu.

NS ddns-a100.net.cmu.edu.

NS ac-ddns1.net.cmu.edu.

A AC-DDNS1 128.2.1.15

A AC-DDNS2 128.2.1.16

A DDNS-A100 128.2.1.16

A t-ns1 128.2.4.14

A t-ns2-sec 128.237.148.6

--reverse zone 128.2.2

NS ac-ddns2.net.cmu.edu.

NS ddns-a100.net.cmu.edu.

NS ac-ddns1.net.cmu.edu.

PTR 142 gruel-2-142.ppp.andrew.cmu.edu.

 

The only question is why would something on my network be wanting to look up that IP address.. or alternatively why would it want to look up gruel-2-142.ppp.andrew.cmu.edu?

 

Apart from a few more gruels, does your cache have anything significantly different or not? In particular do any of the IP addresses in the cache belong to your network?

 

Meanwhile 2.142 should not be the network part of your address (i.e. the first two octets) because whois says 2.0.0.0/8 is reserved. So I think you're saying it's the last two octets that match e.g. x.x.2.142 and that doesn't prove anything.

 

I also don't get is what you mean with "scans" and "clients were reporting their DNS names". It's typically the scanner itself that looks up DNS names to fit IP addresses, so what were you running and where from? Could it be buggy?

 

Is there any reason some 128.2.x.x addresses might be floating about on your network. Have you seen any say with Ethereal? Could a small typo on some machine/device result in that?

 

Rootkit Revealer is good.. I'd have run that on all the DCs.. probably followed by Autoruns (with don't display signed MS stuff turned on to reduce the clutter)... and then some AV scanner.. provided everything looked fine I'd then change all the admin passwords. Then I'd scour the event logs for anything unusual concerning DNS & DHCP.

 

Then I'd find a hub, connect it between your network and router, then connect a laptop with Ethereal on it to see if anything on my network is trying to talk to 128.2.x.x addresses (obviously then checking any workstation that are).

 

NMap isn't so useful in this scenario unless you scan the full 65K+ ports on each machine.

[darkstar]

OK – we’ve got something similar but with the addition of a number of gruel entries.

The reason I mentioned DDNS was because ALL records in the cache are ordinary A or NS entries with the exception of those pertaining to CMU. Nevertheless, if it’s normal to have ddns in a cache then all well and good.

 

However, in our cache there are reverse lookups and the only PTR records in the cache are the Gruel ones which can number a good 80 or so.

I can’t see this as being normal.

If I do nbtstat – a, for example 128.2.2.9 I get the correct netbios name ‘FTT10’.

If I ping the same machine with the a switch I get

GRUEL-2-9.PPP.andrew.cmu.edu.

This is despite their being a legitimate A record for the machine in the forward lookup.

Nslookup 128.2.2.9 returns GRUEL-2-9.PPP.andrew.cmu.edu.

 

I am wondering what is the significance of all gruel entries being ptr records?

 

Our network runs on 128.2.1.0 (16) and the corresponding gruel address will always have the host part of our network (I origionaly posted ‘network’ - sorry about that) e.g.

128.2.2.9 will become

gruel – 2-9 ………..

 

It was ethereal which first alerted us to this problem and all our monitoring stuff says the same (we use network supervisor from 3com and we’ve just acquired observer after running a load of trials).

I’ve changed all passwords as you’ve suggested and I continually monitor event logs.

I’m going to have a deeper look at anti-rootkit tools. Your suggestion running ethereal inbetween the network and the gateway sounds like a good idea.

[PiqueABoo]

Our network runs on 128.2.1.0 (16)

 

?!? Uh ?!?

 

All IP addresses beginning with 128.2 belong to Carnegie Mellon University aka CMU. Those strange DNS cache entries of yours are a direct result of using someone else's addresses.

 

You need to find out how that happened and fix it ASAP i.e. change the IP numbering of your network to a range you "own".

[darkstar]

mmmmmm

Our range is private

[E1uSiV3]

As PiqueABoo said, CMU own the 128.2 range. a quick whois on all-nettools brings back:

 

128.2.0.0 - 128.2.255.255

Carnegie Mellon University

Computing Services

5000 Forbes Avenue

Pittsburgh, PA

US

 

You need to use one of the private reserved ranges (or a subset) as outlined in RFC1918

[darkstar]

Yeah

I know it's not strictly correct because it should be in the private range but we are behind that many layers of NAT. The internet is provided by the York council/Kingston/Affinity and they set this network up here long time before I got here.

Other schools in this area also use 128.0.0.0.

As far as the internet is concerned our machines are seen to have one single class C address which is one of Kingstons machines.

.... so, could this point to a configuration issue within our providers systems?

[Geoff]

Yes, they suck and don't understand TCP/IP networking.

[GrumbleDook]

They really need to look at the RFCs to get an idea of what ranges they should be using.

 

Ok ... subnet masks can change to expand your "class 'c'" ranges to give you more than your usually 254 usable addresses, but the use of a public range on a private network, even behind something doing NAT is a no-no.

 

If you are really unhappy with the amount of work you have had to put in because of this you can complain to IANA (or RIPE), who do take a dim view of poor practice like this.

 

I would also have a whinge at Kingston/Affinity/York and explain your problems. If it is a range that they have put in there is no guarantee that they have been pubbering around with things either.

 

I shudder to remember that when EMBC came to set up our connection they did not have a working range for us, they decided to use a temporary range only to find that it was already in use by someone else and they buggered up their connection too.

[john]

mmm, sounds like my past place - they used 131.1.71.* for the range, and I am sure thats not a private range.... My lans all use the 192.168.1/2/3 range or 172.28.96.* ranges

[darkstar]

Ok - its all coming out now. The head of ICT now remembers that the council had set us up with 192.168. blah.

Apparently the school ran out of IP addresses so a guy came in from another school and set the school up with a class B - only instead of 172 he assigned 128 telling all who cared to listen that we were natted so it would be OK.

And it was OK for two years.

I tried to log on to one of our Access Points yesterday and was presented with the portal for the Carnegie Mellon VPN. - highly amusing

 

So it looks like we'll be able to nail this one now.

I can't believe everyone has missed the absolutely blindingly obvious. I think its something like - how can anyone make such an elementary, basic error? Sometimes I suppose it just needs an outsider to point these things out.

Thanks