Resolution Immediacy and Deployment Immediacy – ACI Master Class

When configuring ACI, have you ever wondered what those Resolution Immediacy options [Immediate | On Demand | Pre-provision] and the Deployment Immediacy options [Immediate | On Demand] do? Read on to find out.

I always like to start with a picture.  The one below shows two ESXi hosts, one attached to Leaf 101, the other to Leaf 102. A vCenter Applicance has access to both hosts via a management NIC (vmnic0) in each host.  Although vmnic1 in each host is physically connected to an ACI leaf switch, neither host has been configured to use vmnic1, so the ACI leaf switches do not see any MAC addresses or LLDP packets from the ESXi hosts yet.

I have configured an Access Policy Chain as below that includes a VMM Domain called RN:vC-VMM.Dom, but the VMM Domain has not yet been associated with vCenter, so no vDS exists on vCenter or any ESXi hosts.

On the tenant side, my configuration is shown below. Note the Web-EPG has not yet been linked to the RN:vC-VMM.Dom:

It is imprtant to reiterate that the VMM Domain (RN:vC-VMM.Dom) has not yet been configured with the vCenter details. Therefore, the vDS has not been created in vCenter or on the ESXi hosts, so the ACI leaf switches do not see any MAC addresses or LLDP packets yet.  And of course as yet, no policy has been sent to either Leaf 101 or Leaf 102

We can see this by looking at the VRF situation on each leaf. Note that neither leaf even knows about any VRF except the default VRFs:

Resolution Immediacy: Pre-Provision

Now I’m going to associate my Web-EPG in my Tenant with the RN:vC-VMM.Dom and check the Pre-Provision Resolution Immediacy option:

This has now linked my EPG with the VMM Domain, as the picture shows:

Remember, no packets have left the ESXi servers to reach the ACI fabric at this stage, but by specifying Pre-provision for Resolution Immediacy, ACI looks at the Access Policy Chain for the RN:vC-VMM.Dom and sends policy to every Leaf it finds in that chain – in my case Leaf 101 and Leaf 102.  This can be seen by noticing that both leaves now have at least some policy pushed –  they both now see my Prod-VRF:

Note:RedPoint Setting the Resolution Immediacy option to Pre-provision causes policy to be pushed to all switches that are defined in the Access Policy Chain in which the VMM Domain exists. 

Resolution Immediacy: Immediate

So now that I have established that Pre-Provisioned Resolution Immediacy causes policies to be pushed to the leaf switches irrespective of whether hosts are attached or not, I’ll explore Immediate Resolution Immediacy by changing the Domain configuration under the EPG.

Now that the Resolution Immediacy has been changed to Immediate, the VRF information is removed from the leaf switches – in other words the “Pre-provisioned” policies have been removed.

To show when Immediate Resolution Immediacy is applied, I will now configure the VMM Domain with the vCenter credentials.  That will cause the APIC to handshake with vCenter and create a vDS with a name matching the VMM Domain (RN:vC-VMM.Dom). I’ll then configure vCenter so that one of the two ESXi hosts is given an uplink (vmnic1) on the RN:vC-VMM.Dom. This will allow LLDP packets to flow between the vDS and the Nexus 9000 Leaf Switch. Pictorially it will be:

Well, that’s done, so I’ll take another look at the VRF situation on the leaf switches:

And sure enough, policies have been immediately pushed, but ONLY to the leaf switch where the vDS has been given a connection to ACI.  Note the ESXi hosts don’t yet host a single VM – Immediate mans “Immediately the vDS is seen”.

Note:RedPoint Setting the Resolution Immediacy option to Immediate causes policy to be pushed to leaf switches as soon as an LLDP connection is made between the vDS and the Nexus 9000 Leaf Switch. 

Resolution Immediacy: On Demand

Like last time, I’ll back off the Reolution Immediacy and change it from Immediate to On Demand, then seen what happens to the VRF situation on Leaf 101.

No prizes for guessing the result. My RedNectar:Prod-VRF has disappeared:

To show you when On Demand Resolution Immediacy takes place, I’ll continue with the vCenter configuration by adding the second ESXi host to the vDS, and adding a VMs to both ESXi Hosts. But I’ll only configure the VM on ESXi2 with the vDS portgoup assigned to its NIC. Here’s the picture:

And I’m betting you already know that the output of the show vrf commands is going to be just like this:

And as you now doubt predicted, my RedNectar:Prod-VRF has been created only on Leaf102 where the vDS RN:vC-VMM.Dom was assigned an uplink via vmnic1 to the Nexus 9000 Leaf Switch. At this stage the VMs are still powered off, so no packets had to flow for the policy to be pushed to the Leaf Switch.

Note:RedPoint Setting the Resolution Immediacy option to On Demand means that policy is not pushed to the Switches until a VM’s vNIC is assigned to a Port Group on the vDS created by the APIC.

Deployment Immediacy: Immediate and On Demand

Having dealt with Resolution Immediacy, it’s time to look at Deployment Immediacy.  This one is a little more straight forward, and has nothing to do with when the policies are pushed to the switches, and everything to do with when the policies are committed to Contenet Addressable Memory (CAM or TCAM) after being pushed.  As you would expect, Immediate Deployment means that polices will be committed to TCAM as soon as they are pushed to the switches, whether that be Pre-provisioned, when the vDS sees LLDP packets (Immediate) or when a VM is assigned to the vDS (on-demand).

On Demand Deployment Immediacy simply menas that the TCAM resources are not consumed until a packet is seen.

Conclusion and Best Practice

In terms of conservation of resources, using a Resolution Immediacy of On Demand is recommended, although in practice it is probably functionally equivalent to Resolution Immediacy of Immediate because it would not be often that a vDS would be deployed on an ESXi host without any VMs using it. However I can see that it would be possible (perhaps all the VMs have been migrated and no-one has decommissioned the vDS) so my recommendation (with one exception, see below) is to use Resolution Immediacy of On Demand.

Exception: There are times when it is necesary to use a Resolution Immediacy of Pre-Provision. If there is more than one switch hop between the ESXi host and the nexus 9000 Leaf Switch, or there is a switch that does not support LLDP (or CDP at a pinch) then LLDP packets can’t reach the vDS and the Leaf.  In these situations, as will often be the case during migration, use a Resolution Immediacy of Pre-Provision.

Tip: Use a separate AAEP (Attachable Access Entity Profile) for all ESXi attached devices if using the Resolution Immediacy Pre-Provision option. That way you will ensure that only switches to which the ESXi hosts enter the ACI fabric have the policies push to them

Of course you can easily modify the Resolution Immediacy of an EPG as shown in the illustrations above, so if you use Pre-Provision during migration, you can change it after if you wish.

Deployment Immediacy also has some limitations as to when you can use On Demand. For instance, the microsegmentation feature requires resolution to be immediate.

Use On Demand for both Resolution Immediacy and Deployment Immediacy unless:

  1. You don’t have LLDP connectivity between your leaf switch and the ESXi hosts:
    • In which case you should use Pre-Provision for Resolution Immediacy.
  2. You are using the microsegmentation feature:
    • In which case you should use Immediate for Deployment Immediacy.


Further Reading:

Posted in Access Policy Chain, ACI, ACI Tutorial, Cisco, Master Class, Nexus 9000 | Tagged , , ,

ARP Gleaning – ACI Master Class

Many people are confused about the way ACI handles ARPs and whether they should enable the ARP Flooding option.  This article explains the following fact:

ARP flooding is only required if the following two conditions are met:

  1. There is a silent host in a Bridge Domain
  2. There is no IP address configured for the bridge domain in the same subnet as the silent host

The reason for this is because ACI does ARP Gleaning.

This is howARP Gleaning works

Let’s start with a picture of two leaf switches with three hosts attached in the same subnet.

As you can see, two of the hosts are VMs (one on each leaf), and the other is a single attached host on Leaf102.

ACI is configured fairly simply –

  • An IP address is configured on the Bridge Domain.
  • Forwarding is optimised: i.e.
    • L2 Unknown Unicasts are sent to the Hardware Proxy
    • L3 Unknown Multicasts are flooded
    • Multi Destination frames are flooded within the BD
    • ARP flooding is disabled
  • WebServer VM2 and WebServer BM are on the same EPG, WebServer VM1 is on a different EPG.
    • This is just to illustrate that ARP Gleaning works at the Bridge Domain level, not VLAN encapsulation level, and is not restricted to a single switch.

The hosts are all silent Linux boxes running Lubuntu – in other words none of the hosts have sent any packet at the beginning of the scenario.

I’ll begin the test by sending a single ping from the BM host attached to Leaf102 (via eth1/26) to the VM also attached to Leaf102 (via eth1/23) while running Wireshark captures on all three hosts.  Remember, the VM has not yet sent a single packet and its MAC address is as yet unknown on Leaf102.  This can be seen by looking at the Operational tab of the EPG.

Now if you know a little about ACI, you will know that if a workstation has NEVER sent a packet, it will be unknown to its closest leaf, and therefore unknown to the entire fabric.  The question that needs to be addressed is “If ARP flooding is disabled, how can ACI find a workstation if it has never sent a packet?”.  To find the answer, read on as I describe what happens when a ping command is issued at the source station.

The ping generates three ARP requests. The following capture taken on the sending PC show the first two ARPs go unanswered, then suddenly an ARP request from the Default Gateway IP turns up. This is the Gleaning ARP – the ARP request sent by the default gateway.  Shortly I’ll explain why this Gleaning ARP made it possible for the third ARP request in the capture below to get a reply from the target workstation, and for the subsequent ping packet to get a reply.

To understand why the third ARP request in the capture above got a reply, you’ll have to look at the capture on the target workstation as shown below. Note that before it received the single ARP request from the first workstation, it received three ARP requests from the default gateway IP. There are the Gleaning ARPs sent by the ACI fabric.  The purpose of these Gleaning ARPs is simply to “tickle” the target station into sending a packet – not because the gateway needs the MAC address of the target!

So as you can see in the capture above, it is not until the target has responded to the Gleaning ARP that it gets the ARP request from the source station.

I’ll wrap up with a few other points about ARP Gleaning.

  • ARP Gleaning ONLY works if the Bridge Domain (or EPG associated with the Bridge Domain) has been assigned an IP address on the same subnet with which it can source a Gleaning ARP.
  • The IP address assigned to the Bridge Domain does not have to be the default gateway IP – if you have a router or firewall attached that serves as a default gateway for an EPG and you DON’T want to turn on ARP flooding, assigning any IP address on that subnet to the Bridge Domain will ensure your hosts will find their default gateway.
  • ARP Gleaning requests are flooded throughout the Bridge Domain – this is demonstrated by looking at the packet capture of the VM on Leaf101 – it is on the same Bridge Domain but different EPG – yet it still saw the ARP Gleaning broadcast, as shown below:


It is not always necessary to enable ARP flooding on a Bridge Domain in ACI if you have silent hosts – assigning an IP address on the same subnet to the Bridge Domain will enable ARP Gleaning which may reduce the total broadcast count for the Bridge Domain.

Only if you have silent hosts on a subnet and you don’t have an IP address set on the Bridge Domain, will you need to enable ARP flooding.

Dedication: Vineet – this one is for you!

Posted in ACI, ACI Tutorial, Cisco, Master Class | Tagged , , , | 7 Comments

An Epyc new addition to the UCS Family!

Another great post from UCS Guru. Make sure you read the full story on his blog.

Back in February of this year, when I read an article in The Register, announcing that Raghu Nambiar, the then chief technology officer for UCS servers had joined AMD. I didn’t think too much of it, but when I also saw that AMD were, for the first time (in my memory), exhibiting at Cisco Live, My right eyebrow rose in a particular “Roger Moore esque” manner, and I sensed something may well be afoot.

Some of you may well have noticed that even since 2009 there has always been an AMD CPU server qualification policy in Cisco UCS Manager , and several years ago I did bring this up with Cisco, as to why in an exclusively Intel based product would need such a policy, to which, if memory serves, the answer at the time was “never say never”

Well today that “prophecy”  was fulfilled with the announcement of the…

View original post 765 more words

Posted in GNS3 WorkBench

Backup Plan

Found this in a hotel tonight. Always good to have a backup plan

Posted in GNS3 WorkBench | Tagged

Why Easter doesn’t it fall at different times in different time zones

If Easter Sunday falls “on the first Sunday AFTER the first full moon after the vernal equinox”, why doesn’t it fall at different times in different time zones?  This year for example, tonight’s (Easter Saturday March 31 2018) full moon occurs after midnight in places between the International date line and the UTC+11 time zone. So according to the formula, Kiwi kids should have to wait another week before breaking open those chocolate eggs.

Well, it turns out that the formula is not set by the astronomical path of the moon, but by a bunch of men (I’ve no doubt women weren’t invited) who formulated the Ecclesiastical Lunar Calendar so long ago that it was before the split of the Gregorian and Julian calendars. (In 325 AD/CE in fact).

Which means today we actually have two Easters, one for each of the divergent calendars, even though both follow the same formula.

Anyway, in the said Ecclesiastical Lunar Calendar, the vernal equinox is always March 21, irrespective of the position of the earth in regard to its transit around the sun. And Easter is always the Sunday following the Pascal Full Moon. And for the calculation of Easter, the Pascal Full moon is defined as been the 14th day after the Ecclesiastical Lunar new moon – so we are back to the Ecclesiastical Lunar Calendar and its ancient origins.

Now it’s probably a good thing that there is a universal standard or two, it means we only have two variations – the Gregorian and the Julian – of Easter throughout the world, and children in New Zealand, Fiji etc. don’t have to hang out for another week to get their Easter Eggs – oh that’s unless they are following the Julian calendar (as Orthodox Christians do), it which case they will have to wait until April 8 2018!


Posted in blog, opinion

CSMA/CD and full duplex for wireless? It could be coming

A group of researchers at National Instruments have found a way to listen to radio signals while receiving on the same frequency.

The team found a solution that relies on in-band full duplex, so it can sense while transmitting, which potentially eliminates all collision overheads in wireless networks.

This could have huge implications – and even give your home wifi a boost if you have a lot of users – certainly will give the office and cafe wifi hotspots a boost.

The problem with existing wireless communications is that once a device starts transmitting, it doesn’t know if another device has transmitted at the same time (causing a collision) until it has finished transmitting and waited for an acknowledgement from the Access Point.  If no acknowledgement comes, it tries again. This is called Carrier Sense, Multiple Access with Collission Avoidance (CSMA/CA).

Your ancient (1980-c2000) shared Ethernet on the other hand operated in much the same way, a device would start transmitting, but was able to detect if any other device transmitted at the same time, and so stop transmitting immediately. This was called  Carrier Sense, Multiple Access with Collission Detection (CSMA/CD) and is course much more efficient than Collision Avoidance.

But that is not the whole story. Modern wired Ethernet networks use two pairs of wires to transmit, and another to recieve, meaning they can transmit AND receive at the same time. Full Duplex.  If we could do that for wireless, (and this article indicates that they have achieved full-duplex operation albeit with just 6 devices at this stage), then the benefits could be much greater.


Posted in blog, opinion, wifi, wireless network | Tagged ,

Why have WordPress made it soooo hard to follow someone?

WordPress, you have hosted my blog since 2010.  I won’t start a tirade of things you STILL can’t do on WP, but I am going to have a whinge about one feature you have obscured.

Why have WordPress made it soooo hard to follow someone?  I should never have to respond to a reader’s comment such as the one I got today.

I would like to thank you sooooo much for such a awesome ACI blogs, I found things here which are not well documented even in Cisco Docs. You are surely doing a great job. I wish to find a subscriber button on your website and keep up with your great work.

For those who would like to follow my blog, or any other blog, you have to move your cursor to the bottom right-hand corner of the page, and/or scroll up a bit (scrolling is clearly the only option on a mouseless device). You will then get an option pop up giving you the chance to follow or subscribe to my blog.


Posted in opinion, rant, wordpress | Tagged | 2 Comments