Weird SNMP Issue affecting core switch

[ITGURU]

I have a core switch stack where the CPU constantly runs at over 95% ive been working with the manufacturer on this and the cause is SNMP.

I do have snmp monitoring on a server but disabling this has no effect.

However, a wireshark capture shows a workstation making SNMP requests to the core switch IP.

If the machine is OFF or I block port 161 in outbound rules on the workstation windows defender firewall, the switch CPU suddenly drops to 40% which is as expected.

However, using various tools to find open Ports on the machine, it never shows Port 161 being active/open if unblock the rule.

anyone able to advise how else to find what process is causing the SNMP issue as currports, netstat etc do not show up anything!

[Dos_Box]

What is the workstation used for and is there any software unique to it?

[ITGURU]

What is the workstation used for and is there any software unique to it?

 

Its my day to day workstation but no monitoring software on it.

[Dos_Box]

Its my day to day workstation but no monitoring software on it.

 

Ah, so it has 'everything' on it

Another question needed I think, how long has this been going on for?

 

*Trying not to sound like a doctor.

 

ETA: Have you tried looking in Windows Defender Firewall\Allowed Applications to see if anything is lurking in there?

I suspect it could be the management\admin console for a network app possibly, if you have any installed.

Edited April 25, 2024 by Dos_Box

[dmj]

What does wireshark show it doing? a DOS attack, brute forcing the SNMP passwords?

 

Re-image the machine.

[ITGURU]

What does wireshark show it doing? a DOS attack, brute forcing the SNMP passwords?

 

Re-image the machine.

wireshark shows several entries with different OIDs

 

173641 2024-04-25 14:09:17.055594 Source IP Destination IP SNMP 95 get-response 1.3.6.1.2.1.4.22.1.1.520.10.35.144.54

 

and when blocked windows firewall reports as the following in the log:

 

2024-04-25 14:59:51 DROP UDP source ip destination IP 59029 161 0 - - - - - - - SEND 4248

[kmount]

Hi there,

 

I suspect you're not finding 161 in open ports on the workstation is because its not port 161 on the source (workstation/client) end that is open as its not running a service - the core switch will have it open as its running the service.

 

(little bit like you won't find your workstation has port 80 or 443 open (unless you're running a webserver on it) but you're browsing websites on 80 and 443)

 

If you take a look on your workstation in netstat -ano | find /I "161" you'll likely see your workstation IP on the left and the core switch on the right getting smashed.

 

Crucially though, you should hopefully see the process id / pid of the process that is making the connection, then grab that out with tasklist and should hopefully find the culprit that is doing it.

 

Fingers crossed.

 

It'll be something trying to be helpful

[ITGURU]

Hi there,

 

I suspect you're not finding 161 in open ports on the workstation is because its not port 161 on the source (workstation/client) end that is open as its not running a service - the core switch will have it open as its running the service.

 

(little bit like you won't find your workstation has port 80 or 443 open (unless you're running a webserver on it) but you're browsing websites on 80 and 443)

 

If you take a look on your workstation in netstat -ano | find /I "161" you'll likely see your workstation IP on the left and the core switch on the right getting smashed.

 

Crucially though, you should hopefully see the process id / pid of the process that is making the connection, then grab that out with tasklist and should hopefully find the culprit that is doing it.

 

Fingers crossed.

 

It'll be something trying to be helpful

MMC - HyperV management console when have it open!! why?!

[ITGURU]

MMC - HyperV management console when have it open!! why?!

 

That didnt work! logged on fresh this morning and before opening any applications, logged into core switch UI - CPU low as expected.

Went straight into firewall on PC , removed Port 161 block and it peaked it again.

ran the command, only things open this time with process ids are veeam standby service and TCP/IP NETBIOS helper,

Definitely a Port 161 issue on workstation something communicating.

[kmount]

Interesting

 

What Veeam service(s) are on your workstation? It's not a copy of VeeamONE or something is it that can do SNMP scanning.

[ITGURU]

Interesting

 

What Veeam service(s) are on your workstation? It's not a copy of VeeamONE or something is it that can do SNMP scanning.

 

Just the remote veeam management console.

[Davit2005]

Have you tried running netstat with option -anob Edited April 26, 2024 by Davit2005