Trace a Network Fault - Packet Capturing Help...

[enjay]

So, we've got something generating a shed-load of traffic and killing our Internet connection. I've run a packet sniffer, but don't totally understand what I'm looking at, so was hoping for some help.

 

The only broadcasts detected (scanned from my PC and from the DC server, but can try elsewhere as people advise...) are ARP requests to/from the servers and Internet gateway, which are presumably okay. There are LOTS of bad SMB2 and TCP packets between my PC and the server with an "incorrect header checksum" which apparently might be caused by "IP checksum offload" - is that something to be concerned about?

 

I'll try port mirroring our Internet uplink too, since that seems to be where the issue is (that, or our Gig network is too fast for the bad packets/device to have a noticeable impact). That said, our DC server acts as DNS, so doesn't that mean that any Internet traffic would be visible from the server too?

[FN-GM]

Do you have some kind of router or firewall you can see where the traffic is coming from?

[enjay]

The router doesn't log traffic, so I'd need to set the sniffer up alongside it to get anything helpful.

[psydii]

If you are RBC connected, or your ISP is friendly you could ask them to take a look at the traffic leaving your site.

 

A more physical method would be to isolate your LAN from the firewall and bring it back switch by switch, starting with your servers.

 

I don't suppose anyone has brought in a personal computers recently, running say iTunes match?

[bio]

The "incorrect header checksum" problem can be solved by disabling ALL ip offload capabilities on your NIC (the machine you use for sniffing).

 

regards

bio..

[enjay]

The "incorrect header checksum" problem can be solved by disabling ALL ip offload capabilities on your NIC (the machine you use for sniffing).

 

Are incorrect header checksums an issue? I can go round every PC disabling the offloading, but obviously want to be reasonably sure it will solve the problem before going and doing it.

[psydii]

i think bio meant "to eliminate the checksum error disable all offloading functionality on the NIC you are using to sniff the packets" rather than on all NICs on your lan.

[enjay]

Yes, I got that. My question was whether having these checksum errors is a problem, or if I should continue looking for something else which is causing our issues...

[cpjitservices]

are you SURE someone hasnt plugged a router or something in somewhere or a switch and it's causing your network to reconverge ? I saw this at the college stacks of switches and servers offline / not accessible via the network because someone plugged in a little Belkin router in one of the IT rooms and it was a port that there PC was plugged into, the Belkin had DHCP enabled and eventually computers etc etc where requesting IP addresses from this router rather than from the correct DHCP server, the PC's that responded would loose all internet connectivity because they were still logged on but had gotten a new IP but couldnt talk to the right gateway, it's worth double checking!

[enjay]

Thanks cpjitservices, but it isn't that - our issue is that something is pulling our Internet connection speed down. Run the router on its own, and we get 40Mb; plug our LAN into it and it immediately drops to 25Mb and then keeps doing down until it levels out at 8Mb (at which point BT tether our service to that speed, so even when I remove the offending device, the connection doesn't resume, but that's another story). Sometimes, everyone loses Internet connectivity for a few seconds, during which time the router doesn't respond to pings. There's no evidence of this affecting the rest of our network, so I think it is Internet-bound traffic doing this, although that could be that we don't slave our 1Gb network enough to notice a drop in performance across that.

[Domino]

Thanks cpjitservices, but it isn't that - our issue is that something is pulling our Internet connection speed down. Run the router on its own, and we get 40Mb; plug our LAN into it and it immediately drops to 25Mb and then keeps doing down until it levels out at 8Mb (at which point BT tether our service to that speed, so even when I remove the offending device, the connection doesn't resume, but that's another story). Sometimes, everyone loses Internet connectivity for a few seconds, during which time the router doesn't respond to pings. There's no evidence of this affecting the rest of our network, so I think it is Internet-bound traffic doing this, although that could be that we don't slave our 1Gb network enough to notice a drop in performance across that.

 

I had something similar not long ago, and turned out to be faulty ISP equipment - was a job proving it though.

 

Is the network equipment setup for SNMP? or can you? MRTG and NTOP may help you track down if it's a machine using all the bandwidth, or the bandwidth itself shrinking 9may also point you to the offending article)

MRTG - Tobi Oetiker's MRTG - The Multi Router Traffic Grapher

http://www.ntop.org/

[enjay]

Not faulty ISP equipment, as we've swapped other equipment in and the problem remains.

 

I'll give MRTG a try though - that looks more friendly than port-mirroring the router and running Wireshark on it. It may well allow me to identify the source without having to unplug everything too. Thanks.

[cpjitservices]

Dodgy cable(s) perhaps have you replaced all the cabling to rule it out ???

[enjay]

Dodgy cable(s) perhaps have you replaced all the cabling to rule it out ???

 

Not yet - I need to find where the issue is coming from before I start doing things like that...

[psydii]

Is this really a problem for your users? (as an example I used to have a 100Mb feed, but due to the way the RBC controlled the pipe no individual host got more than a few Mb/s at busy times, but they would all get a consistant capped amount - it annoyed those who felt they ought to get 100Mb/s but in reality it didn't affect their ability to teach or learn)

 

Is the problem still there out of hours?

 

Have you tried isolating the fault by half-splitting your LAN? <-- this is really what I meant suggested isolating your router and then bringing things back switch by switch.

[enjay]

Yes, it is a problem - an 8Mb service is insufficient to support a whole class making extensive use of the Internet, and all connectivity dropping out for 10-60 seconds is unacceptable.

 

I haven't yet tried isolating everything in the way you suggest, as I can't take really do that until the Easter holidays, so I was hoping to find something sooner than that.

[psydii]

Fair enough.

 

Have you had a chat with management about it? You may find them supportive of a little downtime to resolve/isolate the long term issue that is impacting on T+L. If there are two of you with radios/phones it shouldn't take more than an hour to isolate the device(s) causing the issue.

 

The trouble will be if the problem is intermittant - in which case you might not be able to tell whether you've isolated it or it just has gone away on its own, which would be a waste of everybody's time, something you are obviously keen to avoid.

[bio]

The next thing i would do is to look for packetloss on your internet pipe. You ISP can give you this information or if you have a decent router then you can check that info on the interface that connects to your ISP.

 

bio..

[TheMinister]

Build yourself a dual homed firewall that will log all traffic.

 

You can use this to segment areas of your network.