Excessive ARP traffic and switch crash: malware?

[Lockbits 10]

Hello guys,

We've a customer that is experiencing the following issue. From today some switch are crashing due to a high amount of ARP traffic. They blocked some ports but the problem continued with others ports.

Coincidentally there're some computers that are running EES and have some ARP Poisoning attacks recorded in log. Indeed one of them have such firewall detection every day. We are requesting from them some ESET Log Collector of the computers that have this firewall detection in order to see if there's something suspicious.

In your experience, can a malware cause high amount of ARP traffic that led switch to crash?

Thank you.

[itman 1,659]

MAC Address Flooding attack which in turn leads to a Denial of Service attack. Ref.: http://www.insecure.in/arp_attack.asp

Quote

Once the switch's MAC address table is full and it can not save any more MAC address, its enters into a fail-open mode and start behaving like a network Hub. Frames are flooded to all ports, similar to broadcast type of communicaton.

Now, what is the benefit of the attacker? The attacker's machine will be delivered with all the frames between the victim and another machines. The attacker will be able to capture sensitive data from network.

How to prevent MAC flooding attacks

Cisco switches are packed with in-built security feature against MAC flooding attacks, called as Port Security. Port Security is a feature of Cisco Switches, which give protection against MAC flooding attacks.

http://www.omnisecu.com/ccna-security/what-is-mac-flooding-attack-how-to-prevent-mac-flooding-attack.php

Edited February 4, 2019 by itman

[itman 1,659]

As far as determining what is attacking the switch, you will have to examine your Eset log for what source IP address is shown. If only one internal network address is shown, that is the culprit. If multiple internal IP addresses are shown, it would point to the router/gateway being compromised/malfunctioning. It is also possible that there is some type of network device malfunction that is overloading the switch.

Edited February 4, 2019 by itman

[Lockbits 10]

1 hour ago, itman said:

As far as determining what is attacking the switch, you will have to examine your Eset log for what source IP address is shown. If only one internal network address is shown, that is the culprit. If multiple internal IP addresses are shown, it would point to the router/gateway being compromised/malfunctioning. It is also possible that there is some type of network device malfunction that is overloading the switch.

Hi itman,

Thank you for your feedback. Tomorrow we’ll visit the customer in order to get the logs and necessary data. 

[Lockbits 10]

Hello guys,

Today I got some ESET Log Collector of two suspicious computer. One of them has many ARP poisoning attacks detections in firewall log.

Can you check them? I analyzed the SysInspector logs and all looks pretty normal for me.

Thank you.

ees_logsCGonzalez.zip

[itman 1,659]

Only Eset mods have access to forum posting attachments. You will have to post screen shots, one or two from each device, of the logs from the two devices if you want help from anyone on the forum.

[Lockbits 10]

11 minutes ago, itman said:

Only Eset mods have access to forum posting attachments. You will have to post screen shots, one or two from each device, of the logs from the two devices if you want help from anyone on the forum.

I'm uploading the files and I'm going to send you the URL via a private message.

What's very strange is that almost all firewall alerts came from IPs that starts with 169.....

Edited February 5, 2019 by Lockbits
Added screenshot

[itman 1,659]

I don't know how to read those Eset logs. So I will pass on that.

As far as IP addresses starting with 169.254:

Quote

Definition of: APIPA. APIPA. (Automatic Private IP Addressing) The Windows function that provides DHCP autoconfiguration addressing. APIPA assigns a class B IP address from 169.254.0.0 to 169.254.255.255 to the client when a DHCP server is either permanently or temporarily unavailable.

https://www.pcmag.com/encyclopedia/term/37858/apipa

If a device cannot establish a connection to its designated DHCP server(usually within the router) which results in IP address lease assignment(dynamically assigned), it will temporarily assign a local non-routable IP address in the above APIPA address range. The device will keep trying to acquire a valid DHCP connection/lease periodically. Appears to me a malfunctioning/misconfigured router/gateway is the source of the Eset alerts. Or more likely, they are relying on APIPA addressing for the switch perhaps? Also appears Eset doesn't exclude APIPA assigned addresses from its ARP poisoning detection. 

Edited February 5, 2019 by itman

[itman 1,659]

I also believe we are referring to a Level 3 switch in this network: https://searchnetworking.techtarget.com/tip/Layer-3-switches-explained . As the article explains, a Level 3 switch works identical to a router except it has no WAN connectivity. In other words, it can't connect to the Internet. A Level 3 switch would use local IP addresses, e.g. 192.168.xxx.xxx, just like a router does. Level 2 switches on the other hand, use MAC addresses as discussed previously.

What I think is happening is all these device assigned APIPA IP addresses are hitting the Level 3 switch and it is attempting to build routing data for them. This is overloading the switch causing it to crash. Bottom line - it definitely appears to me there is a DHCP server problem with the router/gateway in this network.

[itman 1,659]

The following posting might also be informative:
 

Quote

Real World Issue - Broadcast storm from 169.254/16 causes CPU DoS on 6500's

We have been having an issue at one of our sites with collapsed core/distribution in 2 6500's doing layer3 switching.  Occasionally a "misconfigured" windows machine will come online on the network with a "auto self assigned" IP address - IPv4 link local - 169.254.x.x/16. It will then begin sending 100s' of thousands of packets per second broadcasts to 169.254.255.255.  We don't have packet captures, but I suspect its netbios udp 137 traffic.  This causes both 6500's in our core to go to 100% CPU - and the network performance goes down the tubes - and we cannot manage the switches via SSH when this happens.  Usually someone finds the culprit and unplugs them.  Today, I didn't have anyone on site.  I had a severely degraded network for about 2 1/2 hours before it went away by itself.

https://learningnetwork.cisco.com/thread/35755?start=0&tstart=0

Edited February 5, 2019 by itman

[Lockbits 10]

1 hour ago, itman said:

The following posting might also be informative:
 

https://learningnetwork.cisco.com/thread/35755?start=0&tstart=0

Thank you for all your help. I sent all the information you provided me to the customer. I really appreciate this. 

[itman 1,659]

One other thing that should be checked out in regards to APIPA network addresses :

Quote

"Autoconfiguration" IP Addresses:

        169.254.0.0 - 169.254.255.255

Addresses in the range 169.254.0.0 to 169.254.255.255 are used automatically by most network devices when they are configured to use IP, do not have a static IP Address assigned and are unable to obtain an IP address using DHCP.

This traffic is intended to be confined to the local network, so the administrator of the local network should look for misconfigured hosts. Some ISPs inadvertently also permit this traffic, so you may also want to contact your ISP. This is documented in RFC 6890.

https://www.iana.org/help/abuse-answers

As noted by IANA, APIPA addresses should never be allowed external network access. ISPs should never accept like IP addresses. If the client's ISP is doing so or a Internet server is forwarding these addresses to the Internet, it might account for Eset's IDS alert activity.

Edited February 6, 2019 by itman