When configuring ACI, have you ever wondered what those Resolution Immediacy options [Immediate | On Demand | Pre-provision] and the Deployment Immediacy options [Immediate | On Demand] do? Read on to find out.
I always like to start with a picture. The one below shows two ESXi hosts, one attached to Leaf 101, the other to Leaf 102. A vCenter Applicance has access to both hosts via a management NIC (vmnic0) in each host. Although vmnic1 in each host is physically connected to an ACI leaf switch, neither host has been configured to use vmnic1, so the ACI leaf switches do not see any MAC addresses or LLDP packets from the ESXi hosts yet.
I have configured an Access Policy Chain as below that includes a VMM Domain called RN:vC-VMM.Dom, but the VMM Domain has not yet been associated with vCenter, so no vDS exists on vCenter or any ESXi hosts.
On the tenant side, my configuration is shown below. Note the Web-EPG has not yet been linked to the RN:vC-VMM.Dom:
It is imprtant to reiterate that the VMM Domain (RN:vC-VMM.Dom) has not yet been configured with the vCenter details. Therefore, the vDS has not been created in vCenter or on the ESXi hosts, so the ACI leaf switches do not see any MAC addresses or LLDP packets yet. And of course as yet, no policy has been sent to either Leaf 101 or Leaf 102
We can see this by looking at the VRF situation on each leaf. Note that neither leaf even knows about any VRF except the default VRFs:
Resolution Immediacy: Pre-Provision
Now I’m going to associate my Web-EPG in my Tenant with the RN:vC-VMM.Dom and check the Pre-Provision Resolution Immediacy option:
This has now linked my EPG with the VMM Domain, as the picture shows:
Remember, no packets have left the ESXi servers to reach the ACI fabric at this stage, but by specifying Pre-provision for Resolution Immediacy, ACI looks at the Access Policy Chain for the RN:vC-VMM.Dom and sends policy to every Leaf it finds in that chain – in my case Leaf 101 and Leaf 102. This can be seen by noticing that both leaves now have at least some policy pushed – they both now see my Prod-VRF:
Note: |
Setting the Resolution Immediacy option to Pre-provision causes policy to be pushed to all switches that are defined in the Access Policy Chain in which the VMM Domain exists. |
Resolution Immediacy: Immediate
So now that I have established that Pre-Provisioned Resolution Immediacy causes policies to be pushed to the leaf switches irrespective of whether hosts are attached or not, I’ll explore Immediate Resolution Immediacy by changing the Domain configuration under the EPG.
Now that the Resolution Immediacy has been changed to Immediate, the VRF information is removed from the leaf switches – in other words the “Pre-provisioned” policies have been removed.
To show when Immediate Resolution Immediacy is applied, I will now configure the VMM Domain with the vCenter credentials. That will cause the APIC to handshake with vCenter and create a vDS with a name matching the VMM Domain (RN:vC-VMM.Dom). I’ll then configure vCenter so that one of the two ESXi hosts is given an uplink (vmnic1) on the RN:vC-VMM.Dom. This will allow LLDP packets to flow between the vDS and the Nexus 9000 Leaf Switch. Pictorially it will be:
Well, that’s done, so I’ll take another look at the VRF situation on the leaf switches:
And sure enough, policies have been immediately pushed, but ONLY to the leaf switch where the vDS has been given a connection to ACI. Note the ESXi hosts don’t yet host a single VM – Immediate mans “Immediately the vDS is seen”.
| Note: |
Setting the Resolution Immediacy option to Immediate causes policy to be pushed to leaf switches as soon as an LLDP connection is made between the vDS and the Nexus 9000 Leaf Switch. |
Resolution Immediacy: On Demand
Like last time, I’ll back off the Reolution Immediacy and change it from Immediate to On Demand, then seen what happens to the VRF situation on Leaf 101.
No prizes for guessing the result. My RedNectar:Prod-VRF has disappeared:
To show you when On Demand Resolution Immediacy takes place, I’ll continue with the vCenter configuration by adding the second ESXi host to the vDS, and adding a VMs to both ESXi Hosts. But I’ll only configure the VM on ESXi2 with the vDS portgoup assigned to its NIC. Here’s the picture:
And I’m betting you already know that the output of the show vrf commands is going to be just like this:
And as you now doubt predicted, my RedNectar:Prod-VRF has been created only on Leaf102 where the vDS RN:vC-VMM.Dom was assigned an uplink via vmnic1 to the Nexus 9000 Leaf Switch. At this stage the VMs are still powered off, so no packets had to flow for the policy to be pushed to the Leaf Switch.
| Note: |
Setting the Resolution Immediacy option to On Demand means that policy is not pushed to the Switches until a VM’s vNIC is assigned to a Port Group on the vDS created by the APIC. |
Deployment Immediacy: Immediate and On Demand
Having dealt with Resolution Immediacy, it’s time to look at Deployment Immediacy. This one is a little more straight forward, and has nothing to do with when the policies are pushed to the switches, and everything to do with when the policies are committed to Contenet Addressable Memory (CAM or TCAM) after being pushed. As you would expect, Immediate Deployment means that polices will be committed to TCAM as soon as they are pushed to the switches, whether that be Pre-provisioned, when the vDS sees LLDP packets (Immediate) or when a VM is assigned to the vDS (on-demand).
On Demand Deployment Immediacy simply menas that the TCAM resources are not consumed until a packet is seen.
Conclusion and Best Practice
In terms of conservation of resources, using a Resolution Immediacy of On Demand is recommended, although in practice it is probably functionally equivalent to Resolution Immediacy of Immediate because it would not be often that a vDS would be deployed on an ESXi host without any VMs using it. However I can see that it would be possible (perhaps all the VMs have been migrated and no-one has decommissioned the vDS) so my recommendation (with one exception, see below) is to use Resolution Immediacy of On Demand.
Exception: There are times when it is necesary to use a Resolution Immediacy of Pre-Provision. If there is more than one switch hop between the ESXi host and the nexus 9000 Leaf Switch, or there is a switch that does not support LLDP (or CDP at a pinch) then LLDP packets can’t reach the vDS and the Leaf. In these situations, as will often be the case during migration, use a Resolution Immediacy of Pre-Provision.
| Tip: |
Use a separate AAEP (Attachable Access Entity Profile) for all ESXi attached devices if using the Resolution Immediacy Pre-Provision option. That way you will ensure that only switches to which the ESXi hosts enter the ACI fabric have the policies push to them |
Of course you can easily modify the Resolution Immediacy of an EPG as shown in the illustrations above, so if you use Pre-Provision during migration, you can change it after if you wish.
Deployment Immediacy also has some limitations as to when you can use On Demand. For instance, the microsegmentation feature requires resolution to be immediate.
| RedNectar’s Recommendation |
Use On Demand for both Resolution Immediacy and Deployment Immediacy unless:
|
RedNectar
Further Reading:
RedNectar's Blog 