Many people are confused about the way ACI handles ARPs and whether they should enable the ARP Flooding option. This article explains the following fact:
ARP flooding is only required if the following two conditions are met:
- There is a silent host in a Bridge Domain
- There is no IP address configured for the bridge domain in the same subnet as the silent host
The reason for this is because ACI does ARP Gleaning.
This is howARP Gleaning works
Let’s start with a picture of two leaf switches with three hosts attached in the same subnet.
As you can see, two of the hosts are VMs (one on each leaf), and the other is a single attached host on Leaf102.
ACI is configured fairly simply –
- An IP address 192.168.82.1/24 is configured on the Bridge Domain.
- Forwarding is optimised: i.e.
- L2 Unknown Unicasts are sent to the Hardware Proxy
- L3 Unknown Multicasts are flooded
- Multi Destination frames are flooded within the BD
- ARP flooding is disabled
- WebServer VM2 and WebServer BM are on the same EPG, WebServer VM1 is on a different EPG.
- This is just to illustrate that ARP Gleaning works at the Bridge Domain level, not VLAN encapsulation level, and is not restricted to a single switch.
The hosts are all silent Linux boxes running Lubuntu – in other words none of the hosts have sent any packet at the beginning of the scenario.
I’ll begin the test by sending a single ping from the BM host attached to Leaf102 (via eth1/26) to the VM also attached to Leaf102 (via eth1/23) while running Wireshark captures on all three hosts. Remember, the VM has not yet sent a single packet and its MAC address is as yet unknown on Leaf102. This can be seen by looking at the Operational tab of the EPG.
Now if you know a little about ACI, you will know that if a workstation has NEVER sent a packet, it will be unknown to its closest leaf, and therefore unknown to the entire fabric. The question that needs to be addressed is “If ARP flooding is disabled, how can ACI find a workstation if it has never sent a packet?”. To find the answer, read on as I describe what happens when a ping command is issued at the source station.
The ping generates three ARP requests. The following capture taken on the sending PC show the first two ARPs go unanswered, then suddenly an ARP request from the Default Gateway IP turns up. This is the Gleaning ARP – the ARP request sent by the default gateway. Shortly I’ll explain why this Gleaning ARP made it possible for the third ARP request in the capture below to get a reply from the target workstation, and for the subsequent ping packet to get a reply.
To understand why the third ARP request in the capture above got a reply, you’ll have to look at the capture on the target workstation as shown below. Note that before it received the single ARP request from the first workstation, it received three ARP requests from the default gateway IP. There are the Gleaning ARPs sent by the ACI fabric. The purpose of these Gleaning ARPs is simply to “tickle” the target station into sending a packet – not because the gateway needs the MAC address of the target!
So as you can see in the capture above, it is not until the target has responded to the Gleaning ARP that it gets the ARP request from the source station.
I’ll wrap up with a few other points about ARP Gleaning.
- ARP Gleaning ONLY works if the Bridge Domain (or EPG associated with the Bridge Domain) has been assigned an IP address on the same subnet with which it can source a Gleaning ARP.
- The IP address assigned to the Bridge Domain does not have to be the default gateway IP – if you have a router or firewall attached that serves as a default gateway for an EPG and you DON’T want to turn on ARP flooding, assigning any IP address on that subnet to the Bridge Domain will ensure your hosts will find their default gateway.
- ARP Gleaning requests are flooded throughout the Bridge Domain – this is demonstrated by looking at the packet capture of the VM on Leaf101 – it is on the same Bridge Domain but different EPG – yet it still saw the ARP Gleaning broadcast, as shown below:
It is not always necessary to enable ARP flooding on a Bridge Domain in ACI if you have silent hosts – assigning an IP address on the same subnet to the Bridge Domain will enable ARP Gleaning which may reduce the total broadcast count for the Bridge Domain.
Only if you have silent hosts on a subnet and you don’t have an IP address set on the Bridge Domain, will you need to enable ARP flooding.
Dedication: Vineet – this one is for you!