Many people are confused about the way ACI handles ARPs and whether they should enable the ARP Flooding option. This article explains the following fact:
ARP flooding is only required if the following two conditions are met:
- There is a silent host in a Bridge Domain
- There is no IP address configured for the bridge domain in the same subnet as the silent host
The reason for this is because ACI does ARP Gleaning.
This is howARP Gleaning works
Let’s start with a picture of two leaf switches with three hosts attached in the same subnet.
As you can see, two of the hosts are VMs (one on each leaf), and the other is a single attached host on Leaf102.
ACI is configured fairly simply –
- An IP address 192.168.82.1/24 is configured on the Bridge Domain.
- Forwarding is optimised: i.e.
- L2 Unknown Unicasts are sent to the Hardware Proxy
- L3 Unknown Multicasts are flooded
- Multi Destination frames are flooded within the BD
- ARP flooding is disabled
- WebServer VM2 and WebServer BM are on the same EPG, WebServer VM1 is on a different EPG.
- This is just to illustrate that ARP Gleaning works at the Bridge Domain level, not VLAN encapsulation level, and is not restricted to a single switch.
The hosts are all silent Linux boxes running Lubuntu – in other words none of the hosts have sent any packet at the beginning of the scenario.
I’ll begin the test by sending a single ping from the BM host attached to Leaf102 (via eth1/26) to the VM also attached to Leaf102 (via eth1/23) while running Wireshark captures on all three hosts. Remember, the VM has not yet sent a single packet and its MAC address is as yet unknown on Leaf102. This can be seen by looking at the Operational tab of the EPG.
Now if you know a little about ACI, you will know that if a workstation has NEVER sent a packet, it will be unknown to its closest leaf, and therefore unknown to the entire fabric. The question that needs to be addressed is “If ARP flooding is disabled, how can ACI find a workstation if it has never sent a packet?”. To find the answer, read on as I describe what happens when a ping command is issued at the source station.
The ping generates three ARP requests. The following capture taken on the sending PC show the first two ARPs go unanswered, then suddenly an ARP request from the Default Gateway IP turns up. This is the Gleaning ARP – the ARP request sent by the default gateway. Shortly I’ll explain why this Gleaning ARP made it possible for the third ARP request in the capture below to get a reply from the target workstation, and for the subsequent ping packet to get a reply.
To understand why the third ARP request in the capture above got a reply, you’ll have to look at the capture on the target workstation as shown below. Note that before it received the single ARP request from the first workstation, it received three ARP requests from the default gateway IP. There are the Gleaning ARPs sent by the ACI fabric. The purpose of these Gleaning ARPs is simply to “tickle” the target station into sending a packet – not because the gateway needs the MAC address of the target!
So as you can see in the capture above, it is not until the target has responded to the Gleaning ARP that it gets the ARP request from the source station.
I’ll wrap up with a few other points about ARP Gleaning.
- ARP Gleaning ONLY works if the Bridge Domain (or EPG associated with the Bridge Domain) has been assigned an IP address on the same subnet with which it can source a Gleaning ARP.
- The IP address assigned to the Bridge Domain does not have to be the default gateway IP – if you have a router or firewall attached that serves as a default gateway for an EPG and you DON’T want to turn on ARP flooding, assigning any IP address on that subnet to the Bridge Domain will ensure your hosts will find their default gateway.
- ARP Gleaning requests are flooded throughout the Bridge Domain – this is demonstrated by looking at the packet capture of the VM on Leaf101 – it is on the same Bridge Domain but different EPG – yet it still saw the ARP Gleaning broadcast, as shown below:
Conclusion
It is not always necessary to enable ARP flooding on a Bridge Domain in ACI if you have silent hosts – assigning an IP address on the same subnet to the Bridge Domain will enable ARP Gleaning which may reduce the total broadcast count for the Bridge Domain.
Only if you have silent hosts on a subnet and you don’t have an IP address set on the Bridge Domain, will you need to enable ARP flooding.
RedNectar
Dedication: Vineet – this one is for you!
RedNectar's Blog 
Great post. Thank you, Chris
Hey Chirs,
Thanks for the feedback, i remember it now the SPINE will send back the ARP as multicast packets to it leaf switches. beside that i went through that document in ACI forwarding ARP/IP Unknown layer 3 endpoint(glean) Point no.3 – Spine does not have H2 IP, sends special glan packet to all leafs on reserved GIPo with VRF-V1 VNID set in VXLAN header.
so is this the glean packet you referring as multicast packet sent through all leaf switches?
can i comment here.
Hardware Proxy
By default, Layer 2 unknown unicast traffic is sent to the spine proxy. This behavior is controlled by the hardware proxy option associated with a bridge domain: if the destination is not known, send the packet to the spine proxy; if the spine proxy also does not know the address, discard the packet (default mode).
The advantage of the hardware proxy mode is that no flooding occurs in the fabric. The potential disadvantage is that the fabric has to learn all the endpoint addresses.
With Cisco ACI, however, this is not a concern for virtual and physical servers that are part of the fabric: the database is built for scalability to millions of endpoints. However, if the fabric had to learn all the IP addresses coming from the Internet, it would clearly not scale.
———————————
based on the above document, HW proxy enabled and ARP flooding is disabled.
in this case the SPINE also will not learn the endpoint, as per above statement, the SPINE will drop packet, when there is an arp request from from leaf102 which will not suppose send the fabric nodes. how this ARP gleaning still works here and confused on the document what they said and reality in your output is different.
Hi prakin,
Thanks for you comment. Let me make a few observations.
Firtly, I assume the quote you posted is from this post from Thomas De Leon – where he was commenting about layer 2 unicast traffic. Although we could have an academic argument whether ARP is layer 2 traffic or layer 3, the fact is that ACI treats ARP packets more like layer 3 IP packets – if it knows the destination IP sought in the ARP packet, it sends the ARP broadcast frame there.
Now to youe specific points:
HW proxy enabled and ARP flooding is disabled – yes this is the default
in this case the SPINE also will not learn the endpoint, as per above statement, the SPINE will drop packet – again, correct. BUT the Spine WILL, after receiving the ARP packet send a multicast packet to all leaf switches that have ports in the source bridge domain asking them to try and find the destination IP. This is a fundamental part of ARP Gleaning – I guess I didn’t make this step clear in my explanation – see Cisco LIve presentation BRKACI-3101 (login required)
when there is an arp request from from leaf102 which will not suppose send the fabric nodes. – I apologise for the confusion, indeed I was also confused about where the ARP originated at one stage, and I did not make this bit clear. The only arp request from from leaf102 is sent south – not north where indeed (as you say) it would have been dropped. The thing is, BOTH Leaf101 AND Leaf102 send ARP requests after the Spine tells them to, which is why you see the ARP requests from the default gateway IP on both the sending and receiving station in my captures.
I hope this helps, and thanks so much for your comment!
Thanks Chris – much appreciated :-).
A question though, you say “..VM has not yet sent a single packet and its MAC address is as yet unknown on Leaf102.” but in the screenshot the MAC address of VM2 IS present which means the VM must have sent something for ACI to have learnt the MAC address, right? Which leads to another question – if ACI already knows about the MAC then why send the Gleaning ARP request?
Good point, but if you look closely at the screenshot (I know it is very small) you can see that the Learning Source shows vmm only, meaning that the APIC only knows about the VM because vCenter told it. In fact at this stage the VM was powered off.
Got it, thanks for clarifying that point.