Note: This post started as an answer I gave on the Cisco Support Forum. This version is slightly expanded with pictures and examples.

In this post I will examine the roles of three very important protocols that exist in the ACI environment.

I will explain

  • that IS-IS is the underlying routing protocol that is used by the leaves and spines to learn where they sit in the topology in relation to each other
  • how Leaf switches use COOP to report local station information to the Spine (Oracle) switches
  • how BGP and MP-BGP is used ot redistribute routes from external sources to leaf switches.

Let me start with a picture.  Imagine a simple 2leaf/2spine topology with HostA attached to to Leaf1 and with HostB attached to to Leaf2.

  • Leaf1 has a VTEP address of
  • Leaf2 has a VTEP address of
  • Spine1 has a VTEP address of
  • Spine2 has a VTEP address of
  • HostA has a MAC address of A and an IP address of and is attached to port 1/5 on Leaf1
  • HostB has a MAC address of B and an IP address of and is attached to port 1/6 on Leaf2

Enter IS-IS

The leaves and spines will exchange IS-IS routing updates with each other so that Leaf1 sees that it has two equally good paths to reach Leaf2, and Leaf2 sees that it has two equally good paths to reach Leaf1.

Leaf1# show ip route vrf overlay-1
IP Route Table for VRF "overlay-1", ubest/mbest: 2/0
*via, eth1/51.2, [115/3], 6d20h, isis-isis_infra, L1
*via, eth1/52.2, [115/3], 6d20h, isis-isis_infra, L1

For now, that’s all we need to know about IS-IS – it is the routing protocol used by the VTEPs to learn how to reach the other VTEPs.

Now think about the hosts.

This is where COOP comes in.

When Leaf1 learns about HostA because, say HostA sent an ARP request seeking the MAC address of (which you know is HostB, but that’s not relevant at the moment), Leaf1 looks at that ARP request, and just like a normal switch, learns that MAC A is present on port 1/5.  But the leaf is a bit more clever than that, and looks INSIDE the payload of the ARP packet and learns that Host1 also has an IP address of and records all this information in its Local Station Table.

Leaf1#show endpoint interface ethernet 1/5

VLAN/Domain  Encap VLAN  MAC/IP Address  Interface      
65           vlan-2051  a036.9f86.e94e L eth1/5
Tenant1:VRF1 vlan-2051    L eth1/5

AND THEN reports this information to one of the spine switches (chosen at random) using the Council Of Oracles Protocol (COOP).  The spine switch (oracle) that was chosen then relays this information to all the other spines (oracles) so that every spine (oracle) has a complete record of every end point in the system.

The spines (oracles) record the information learned via the COOP in the Global Proxy Table, and this information is used to resolve unknown destination MAC/IP addresses when traffic is sent to the Proxy address.

Note that all of this happens without anything to do with BGP.

But to round off the COOP story, we would assume that at some stage Leaf2 (a citizen) will also learn HostB‘s MAC and IP and also inform one of the spines (oracles) at random of this information using the COOP.

Spine1#show coop internal info repo ep | egrep -i "mac|real|-" 
EP mac : A0:36:9F:86:E9:4E
MAC Tunnel :
Real IPv4 EP :
EP mac : A0:36:9F:61:88:FD
MAC Tunnel :
Real IPv4 EP :

So COOP is used solely for the purpose of distributing endpoint information to spine switches (oracles). As far as I know, spine switches never use COOP to distribute end host information to leaf switches.

So where does BGP fit in?

BGP is not needed until an external router is connected.  So now imagine that Leaf2 has had a router connected and has learned some routes from that external router for a particular VRF for a particular Tenant.

How can Leaf2 pass this information on to Leaf1 where HostA is trying to send packets to one of these external networks?  For Leaf2 to be able to pass routing information on to Leaf1 and keep that information exclusive to the same VRF, we need a routing protocol that is capable of exchanging routing information for multiple VRFs across an underlay network

Which is exactly what MP-BGP was invented for – to carry routing information across MPLS underlay networks.  In the case of ACI, BGP is configured by choosing an Autonomous System number and nominating one of the spine switches to be a route reflector.  MP-BGP is self configuring, you don’t need to do anything to make it work!

(Although you will have to configure your Tenant to exchange routes with the external router.)

Leaf1# show ip route vrf Tenant1:VRF1, ubest/mbest: 1/0, attached, direct, pervasive
*via, [1/0], 04:43:32, static, tag 4294967295, ubest/mbest: 1/0, attached, pervasive
*via, vlan25, [1/0], 03:52:23, local, local, ubest/mbest: 1/0
*via, [200/5], 00:11:41, bgp-1, internal, tag 1

aka Chris Welsh


About RedNectar Chris Welsh

Professional IT Instructor. All things TCP/IP, Cisco or VoIP
This entry was posted in ACI, ACI configuration, APIC, Cisco, Data Center, Data Centre, EPG, L3 Out, L3out and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s