moquery – Cisco’s Mysterious Obscure ACI query utility

[Edit: I also made a video that explains some of this more clearly.]

You really could be forgiven if you thought that Cisco’s ACI moquery command was an acronym for Mysterious Obscure query. Read on and I’ll try and take some of the mystery and obscurity out of Cisco’s Managed Object query.

The final outcome will be that you will be equipped with the knowledge to write a script you can use to find what EPGs are linked to a particular Interface Policy Group. And to be able to adapt my script to suit your needs.

On the way, I’ll explain:

  • What is moquery? Why would I use it?
  • How to construct a simple query
  • Coming to terms with classes and distinguished names
  • Come and meet the family
  • The rest is up to REST
  • Putting moquery to work
  • Appendix: Some geeky background on moquery

What is moquery? Why would I use it?

If you didn’t already know that moquery was a command line utility for  the Cisco ACI APIC, then you probably wouldn’t be reading this.

And in spite of my cynical introduction, you’ve also probably heard the term managed object, and know that ACI stores information using a Management Information Model (MIM), which can be represented in a hierarchical management information tree (MIT).

moquery is used to extract information from the MIM/MIT and display it on the console

There a lot of good reasons to NOT use moquery – Cisco ACI has a reasonably good set of show commands that allow detailed troubleshooting, not to mention the troubleshooting facilities available in the APIC GUI, and there is a far more friendly GUI equivalent to moquery called visore – and you can access that by right-clicking on any object and choosing Open in Object Store Browser.

But there will be times when you’d like to get some information but can’t just get it from the CLI or GUI, like the person why asked the Cisco Community Forum

How can I list all the EPGs that are associated to a particular Leaf Interface Policy Group on the ACI Fabric?

Or maybe you actually want to build a customised view using the APIC APIs, and need to explore the MIT.

For me, I started exploring moquery because I wanted to get a better understanding of the MIM/MIT.

These are all great cases to use moquery.

How to construct a simple query

You already knew that moquery is a command, but like many Unix commands, it is pretty useless without some parameters. Here’s a command to list all your tenants.

moquery -c fvTenant

The problem is, the output of the command is copious and contains fields that you don’t necessarily want. You can fix that to some degree in one of two ways

  1. use the -o table option (which tabulates the most important fields)
  2. use egrep to filter the output.

So try these as variations:

moquery -c fvTenant -o table
moquery -c fvTenant | egrep "^name "

The first ignores irrelevant fields like modTs (timestamp showing when the object was last modified) but includes some rubbish as well.  The second gives you more control but requires a better understanding of how to parse results using egrep.  (BTW – you could have used moquery -c fvTenant | awk '{if ($1=="name") print $3}' if you wanted to make it really complicated)

From here on, I’ll promise to keep the parsing via grepegrepawk or even sed as simple as possible.  But, in time you will see that it is necessary to get the most out of moquery.

Before I digress too far from the topic, I want to revisit the -c option I used above.  The example I used moquery -c fvTenant listed ALL tenants.  To see the details of just a particular tenant (say the mgmt tenant), I would have used the -d option, like:

moquery -d uni/tn-mgmt

which is a pretty boring output.

But to get anything more out of moquery, you’ll have to get used to a couple of concepts. In particular, classes and distinguished names (DNs). And then you’ll have to learn some of those classes and DNs so you can query them.

Coming to terms with classes and distinguished names

I’ve already mentioned that there are two key ways to use moquery.  You can use moquery to query the MIM about a specific distinguished name or to query every instance of a particular class.  Soon you’ll see that you can combine the two to query the MIM about all the instances of a particular class that exist for a particular DN.

Every device, node, interface, policy, endpoint or even user is represented by an object in ACI. Or more precisely, a managed object and will occupy a place in the Managed Object Tree (MIT). There are physical entities (switches, interfaces,…), logical entities (policy groups, profiles, vlan pools…) and even relationships, which are a little harder to explain.

 Key Point: Every object is an instance of a class, and has a unique distinguished name.

That statement is important. If you haven’t memorised it, keep re-reading it until you can.

For example, a tenant called, say Tenant1 has a relative distinguished name of tn‑Tenant1 and is an instance of the class fvTenant.

And just how did I pull those little gems of wisdom [the tn- prefix and fvTenant] from my brain?

How to find the distinguished name (DN) of an object

If you know how to find a particular object in the GUI, you can find its DN simply by looking at the URL of the page that shows you the object.

Everything to the right of the vertical bar | character in the URL defines the distinguished name.  Like a directory structure, the DN can be several levels deep. Remember, we are talking about a Managed Object TREE after all.

 Note: The distinguished name is made up of a series of relative names separated by /.
So the relative name ap-2Tier_AP in the URL above may appear in another DN, such as uni/tn-Tenant2/ap-2Tier_AP. In other words, different objects can have the same RN (relative name) but the whole DN (distinguished name) will always define a single object.

Armed with a DN, you can now query the MIT using moquery with the -d or –dn options:

admin@apic1:~>  moquery --dn uni/tn-Tenant1
Total Objects shown: 1

# fv.Tenant
name         : Tenant1
dn           : uni/tn-Tenant1
rn           : tn-Tenant1
status       :
uid          : 15374


admin@apic1:~> moquery --dn uni/tn-Tenant1/ap-2Tier_AP/epg-AppServers_EPG
Total Objects shown: 1

# fv.AEPg
name                 : AppServers_EPG
dn                   : uni/tn-Tenant1/ap-2Tier_AP/epg-AppServers_EPG
rn                   : epg-AppServers_EPG
uid                  : 15374
 Note: Oh – that’s curious. Did you notice that both uni/tn‑Tenant1 and uni/tn‑Tenant1/ap‑2Tier_AP/epg‑AppServers_EPG have the same uid? That tells me that both objects were created by the same user. In fact, uid 15374 is always the user admin.

But remember, querying a DN is only going to yield a single result – after all, a DN is defined as the name that distinguishes an object, and is therefore unique. Ergo one result per DN.

Which brings us to a great point to discuss classes

How to find the class of an object

If an object is an instance of a class, there must be a way of finding out which class defines the attributes of an object.   Once you have found the class of an object, you can easily find all instances of that class using moquery. In ACI, there are several strategies to find the class of an object.

Probably the easiest way is to again start with the GUI, where you can right-click on an object and select Open in Object Store Browser.  This will show you the object with its class name shown clearly at the top. Here’s an example using uni/tn‑Tenant1

If you’ve noticed that the class name (fvTenant) looks kind of similar to the line…

# fv.Tenant

…that came from the moquery on the DN of Tenant1, you can guess the second method.  Just remove the dot between the fv and Tenant and you have it.

The third method is rather obscure, and you’ll need your grandma’s glasses to be able to read it, but if you (in the ACI GUI) click the Settings cog wheel in the top RH corner of the screen and select Show Debug Info, you will see some information in the browser footer in print about 2pt font. Let me magnify it a bit and point you to what you are looking for. Note you can also see the DN here, but no point in straining your eyes when the same information is in the URL above.

So great. Now you have a class name. How is that useful?  Well, before, when you had the DN of a single object, you could use moquery to query that single object. Now you can use the ‑c or ‑‑klass option1  to get all objects of a class.  So, a command of  moquery ‑c fvTenant will show ALL tenants (like the very first example I gave above). My example below uses egrep to show only lines that begin with # OR begin with name followed by <space> OR begin with dn OR begin with uid

admin@apic1:~> moquery -c fvTenant | egrep "^#|^name\ |^dn|^uid"
# fv.Tenant
name         : Tenant5
dn           : uni/tn-Tenant5
uid          : 15374
# fv.Tenant
name         : common
dn           : uni/tn-common
uid          : 0
# fv.Tenant
name         : Tenant3
dn           : uni/tn-Tenant3
uid          : 15374
# fv.Tenant
name         : mgmt
dn           : uni/tn-mgmt
uid          : 0
# fv.Tenant
name         : infra
dn           : uni/tn-infra
uid          : 0

The words listed on the left-hand side on the output of moquery (like name and dn) are the attributes of a given object.  The attributes of an object are be defined by the class of an object, and that class may even inherit some of those attributes from a parent class, in which case the objects of that class will also inherit attributes from a parent object.  If that sounds like gobbledygook, don’t worry, there are more practical examples later.

For now, make sure you are familiar with the key concepts that:

  • You can use moquery to query a single object
  • You can use moquery to query return a list of every object of a given class

If I were teaching a course right now, I’d be asking

  • Which moquery option would you use to return a single instance of an object?
    Select this line to see answer: moquery -d
  • Which moquery option would you use to return a all instances of a class?
    Select this line to see answer: moquery -c
  • Is fvAEPg an example of a Distinguished Name (DN), an object or a class?
    Select this line to see answer: fvAEPg is an example of a class

Come and meet the family

In the previous section, I showed examples of querying a DN and a class. That’s about as simple as it gets as far as the construction of the syntax goes, but to actually work with ACI you are going to have to learn some key classes and the structure of some DNs.

Since the objects are arranged in a tree, every object has a class and a parent class (except topRoot), and therefore many classes have child classes, which means it is very much a family affair.

So come and meet the family! Learn these to begin with:

Class/DN Description
topRoot you probably won’t ever use it, but it is the only class that doesn’t have a parent class.
infraInfra wierd name, but is the parent class for all objects in the Access Policy Chain
fvTenant the parent class that defines all Tenant attributes
/uni/tn-tenantName The format of the DN of tenant tenantName
fvAEPg the parent class for Application End Point Groups. Not to be confused with fvEPg
/uni/tn-tenantName/ap-appProfileName/epg-epgName The format of the DN of epg epgName within Application Profile appProfileName
fvEPg an abstract class that includes all varieties of EPGs, including fvAEPg, l3extInstP (L3 EPG), and possibly others you didn’t even know about.

So from the previous section you should be comfortable constructing a query to list all instances of the above classes.  If you practice this now on the classes listed above, you’ll notice that some of these classes have many instances – for instance, a query of …

moquery -c fvEPg

… will list ALL EPGs across all tenants.

But say you wanted to list just the EPGs of one particular tenant?

Well, to do that, you can start combining ‑c and ‑d options, so the following query will list the EPGs for tenant Tenant5

moquery -c fvEPg -d /uni/tn-Tenant5

Get the idea?

Now, if you understand the tree structure of a tenant, you can take this concept a little further.  What do you think the following would list?

moquery -c fvBD -d /uni/tn-Tenant5

And if the above query revealed that the tenant named Tenant5 had two Bridge Domains called App_BD and Web_BD, can you work out what the difference between the following two queries would show?

moquery -c fvSubnet -d /uni/tn-Tenant5/BD-App_BD
moquery -c fvSubnet -d /uni/tn-Tenant5

If you wanted to see what subnet had been configured in the same tenant on an EPG called AppServers_EPG within an Application Profile called 2Tier_AP, do you think you could construct the query?  I’ll give you the start below. [Select the whole line to reveal the answer]

moquery -c fvSubnet -d /uni/tn-Tenant5/ap-2Tier_AP/epg-AppServers_EPG

So far, I’ve kept you within the bounds of the fabric virtualisation space of the tenant, where many objects are child objects of a Tenant, so when you look at a diagram of a Tenant, it’s easy to see that VRFs, Bridge Domains and Application Profiles are child objects of a Tenant, and EPGs are child objects of an Application Profile.

Looking at the diagram, it’s not so hard to relate the path of the DN of say an EPG (like /uni/tn‑Tenant5/ap‑2Tier_AP/epg‑AppServers_EPG) to the object model. It’s also not hard to see a pattern in the names of the classes that sit in the fabric virtualisation space of the tenant.

Things get a bit more interesting though when you start querying the Access Policy Chain, which lives under the parent object infraInfra.  But a query of the infraInfra class is very boring. I won’t even waste space here showing you – try it yourself. moquery ‑c infraInfra

Unlike where you might have many tenant objects, there is only one infraInfra object. What you need to learn if you want to query the Access Policy Chain, is the equivalent classes for the stucture you are probably already familiar with:

If you are already familiar with the diagram above, all you need to do now is learn the classes of each of these elements of the Access Policy Chain, and the format of the DN of each instance. But there is a twist…

Notice that in the diagram there are very few parent-child relationships – most of the relationships are links – and the twist is that the links are also objects. And important ones too.

[Note: I’ve taken a bit of licence with the diagram. The relationship arrows are actually child objects of the object they point away from.  For the official diagrams, refer to the APIC Management Information Model Reference website]

Time Out!  Are you confident that you could use the diagram above to produce a query to:

  • List all Physical Domains for an ACI fabric?
    moquery -c physDomP
  • List all Interface Selectors for the Interface Profile with DN=uni/infra/accportprof-T5:L102_IntProf ?
    moquery -c infraHPortS -d uni/infra/accportprof-T5:L102_IntProf

[Select the blank spaces to see the answers]

OK. If you could not answer the questions above, spend some more time re-reading and experimenting at the command line before continuing.

Here’s a problem for you.

Assume you have figured out that you can list all Interface Selectors for the Interface Profile with DN=uni/infra/accportprof-T5:L102_IntProf  by using a query of:

moquery -c infraHPortS -d uni/infra/accportprof-T5:L102_IntProf

How do you find what Interface Policy Group each Interface Selector is linked to? 

To answer that question, you need to know that the relationship between the Interface Selector and the Interface Policy Group is stored in the class infraRsAccBaseGrp as shown in the diagram.

Just query the DN asking for instances of that class to be listed.  The query is almost the same as above, but with at different class-  infraRsAccBaseGrp

admin@apic1:~> moquery -c infraRsAccBaseGrp -d uni/infra/accportprof-T5:L102_IntProf
Total Objects shown: 1

# infra.RsAccBaseGrp
dn           : uni/infra/accportprof-T5:L102_IntProf/hports-1:20-typ-range/rsaccBaseGrp
rn           : rsaccBaseGrp
tCl          : infraAccPortGrp
tDn          : uni/infra/funcprof/accportgrp-T5:SA.Host_APPG

This presents you with a number of new concepts

Firstly, the class itself has a particular naming structure. The letters Rs in infraRsAccBaseGrp indicate that this class is a Relationship source – so you are querying the Interface Profile T5:L102_IntProf for it Relationship sources for “AccBaseGrp” – in this case “AccBaseGrp” could be either of class infraAccPortGrp or infraAccBndlGrp

And the answer to exactly which object is found for each “AccBaseGrp” is found in the tDn (=target Distinguished name) field, and in the example above that DN is an instance of the class infraAccPortGrp as shown be the value of the tCl (=target Class).  Here they are again in case you too lazy to look back.

tCl : infraAccPortGrp 
tDn : uni/infra/funcprof/accportgrp-T5:SA.Host_APPG

That is a lot of information to digest. And I didn’t even mention that there is another named relationship class called infraRtAccBaseGrp (where the Rt=Relationship target) that is a child object class of  infraAccPortGrp pointing back to the interface selector. Nor did I mention that the Interface Selector has child objects that define the actual blocks of ports that are defined for each Interface Selector!

Must be time for another diagram to encapsulate all that.

Now I have to admit that I’ve skipped one little detail while taking you through the journey from infraHPortS to infraAccPortGrp

Recall I said above

“you need to know that the relationship between the Interface Selector and the Interface Policy Group is stored in the class infraRsAccBaseGrp as shown in the diagram. “

If you are astute, you’d be asking “How did I know that the class was called infraRsAccBaseGrp ?”

If I have a starting point of class of infraHPortS  then there must be some way of querying that class to reveal its child objects, including infraRsBaseGrp (and for that matter, infaPortBlk as well).

The query will start with moquery -c infraHPortS – but the rest of the query, well…, the rest of the query is up to the Cisco REST API.

The rest is up to REST

If you have issued a command of moquery --help you would have seen that one of the options is described as…

  -x [OPTIONS [OPTIONS ...]], --options [OPTIONS [OPTIONS ...]]
                        Extra options to the query

which is not very helpful.

But those -x options are the key to getting the most out of moquery.  But to get a description of what those options are, you will have to go to some documentation for the REpresentational State Transfer (REST) documentation for ACI.

In particular, you’ll want to know about these options:

Filter Type Syntax Description
query‑target {self | children | subtree} Define the scope of a query
rsp‑subtree {no | children | full} Specifies child object level included in the response

So back to the problem – “How did I know that the class was called infraRsAccBaseGrp ?”

I used a query of

admin@apic1:~> moquery -c infraHPortS -x query-target=children
Total Objects shown: 2

# infra.RsAccBaseGrp
  dn           : uni/infra/accportprof-T5:L102_IntProf/hports-1:20-typ-range/rsaccBaseGrp
  rn           : rsaccBaseGrp
  tCl          : infraAccPortGrp
  tDn          : uni/infra/funcprof/accportgrp-T5:SA.Host_APPG
# infra.PortBlk
  name         : block
  dn           : uni/infra/accportprof-T5:L102_IntProf/hports-1:20-typ-range/portblk-block
  rn           : portblk-block

and found the child object classes listed – infra.RsAccBaseGrp and infra.PortBlk. All I had to do was remove the separating period.

Another approach would have been to use the x query‑target=subtree, or x rsp‑subtree=children options, but these options also show the parent class as well.  In theory, I should have also been able to use the same options on a DN, but unfortunately moquery gives only one result when using these options with distinguished names. (Probably due to a bug)

 Key Point: Use the -x query-target=children option to find child classes of a class.


If you have a configured APIC you can query, you should try starting at a Switch Profile (moquery -c infraNodeP) and work your way through until you have mapped the entire access policy chain. (And if you DON’T have access to a configured APIC, book yourself a session with the ACI Simulator at – login required)

Here’s one example of how I put moquery to work, similar to the challenge I’ve given you above.

Putting moquery to work

I mentioned earlier that I had been challenged with a question in the Cisco Community Forum

How can I list all the EPGs that are associated to a particular Leaf Interface Policy Group on the ACI Fabric?

Before I show you how I solved the problem, here’s a picture of what was being asked. I’ve duplicated the diagram below with the class names to make it easier for you to follow my logic.

And now the details:

Task 1:

To begin, I needed to find the child object that showed me which AAEP the Interface Policy Group is linked to.

Similar to my previous example, I used moquery to query the particular policy group and added the -x query-target=children option. On my system, Interface Policy Group names have the format TenantID:Name_APPG (if it is an Access Port Policy Group) or TenantID:Leaves:slot:port_VPCIPG) if it is a VPC Interface Policy Group2.  In my case I used an Access Port Policy Group called T5:SA.Host_APPG, and I looked at the URL of the page that showed me the object to find the DN.

The URL was #c:d|uni/infra/funcprof/accportgrp-T5:SA.Host_APPG so my moquery to find the class of the child object that links was:

apic1# moquery -d  uni/infra/funcprof/accportgrp-T5:SA.Host_APPG -x query-target=children
Total Objects shown: 1

# infra.RsAttEntP
annotation     :
childAction    :
dn             : uni/infra/funcprof/rsattEntP
extMngdBy      :
forceResolve   : yes
isUsingConnSel : no
lcOwn          : local
modTs          : 2020-03-27T13:21:41.043
monPolDn       : uni/fabric/monfab-default
rType          : mo
rn             : rsattEntP
state          : formed
stateQual      : none
status         :
tCl            : infraAttEntityP
tDn            : uni/infra/attentp-T5:HostLinks_AAEP
tType          : mo
uid            : 15374

[About 20+ MORE object should have appeared. I was just lucky that
the one that did appear was the one I wanted]

Now to be honest, I was expecting many more child objects, but due to what I believe is a bug in moquery3, I only got one – and by sheer luck, it happened to be the correct one.  If this had have failed, I would have had to resort to querying the class
moquery -c infraAccPortGrp -x query-target=children

Armed with the information that the class of the link to the AAEP is infraRsAttEntP, I can now create a more specific query to give me ONLY the child object that point to the AAEP, and not the 20+ I SHOULD have got from the above query.  I also started by storing the name of the Interface Policy Group in a variable called ipgName so that the process can be more repeatable.

admin@apic1:~> ipgName=T5:SA.Host_APPG
admin@apic1:~> moquery -d  uni/infra/funcprof/accportgrp-$ipgName -c infraRsAttEntP
Total Objects shown: 1

# infra.RsAttEntP
annotation     :
childAction    :
dn             : uni/infra/funcprof/accportgrp-T5:SA.Host_APPG/rsattEntP
extMngdBy      :
forceResolve   : yes
isUsingConnSel : no
lcOwn          : local
modTs          : 2020-03-27T13:21:41.043
monPolDn       : uni/fabric/monfab-default
rType          : mo
rn             : rsattEntP
state          : formed
stateQual      : none
status         :
tCl            : infraAttEntityP
tDn            : uni/infra/attentp-T5:HostLinks_AAEP
tType          : mo
uid            : 15374

Great! I’d found the target DN for the AAEP

Time Out!  I’m sure you see that the output of the last two commands is the same.  The second one is specific to the precise class I’m interested in. The first one SHOULD have produced a lot more output and would have been a lot harder to parse. And should Cisco ever fix the bug, it will be different.

The final step of this task then is to keep that target DN of the AAEP in a variable.  I used awk to extract the string uni/infra/attentp-T5:HostLinks_AAEP and store it in a variable I named aaepDn4

admin@apic1:~> aaepDn=$(moquery -d  uni/infra/funcprof/accportgrp-$ipgName -c infraRsAttEntP | awk '{if ($1=="tDn") print $3}')
admin@apic1:~> echo $aaepDn

Task 2:

Next, I needed to find the child object that shows me which Domains the AAEP is linked to.  Using the same logic, I found the class of the object that links the AAEP to the Domains is infraRsDomP  – only this time I was not so lucky as to have the query revealing this information from the DN work.  I.e. I SHOULD have been able to use moquery ‑d $aaepDn ‑x query‑target=children but instead had to use moquery ‑c infraAttEntityP ‑x query‑target=children

Again, following the same logic, I created a variable called domainList to store the output of the query on infraRsDomP, only this time, life was a little more difficult because an AAEP can be linked to multiple Domains, and on my system that indeed was the case.  That meant that I could process the list of Domains more easily as an array.

Here’s the query I used to do that

admin@apic1:~> declare -a domainList=($(moquery -d $aaepDn -c infraRsDomP | awk '{if ($1=="tDn") print $3}'))
admin@apic1:~> for i in "${domainList[@]}"; do echo $i; done

Task 3:

Finally, I needed to find the child object that shows me which EPGs the Domains are linked to.  Using the same logic, (i.e., running moquery ‑c physDomP ‑x query‑target=children) I found the class of the object that links a Domain to the EPG is  infraRtDomAtt  by observing the output!

This time of course the process is a bit trickier, because I have to process multiple Domains – and again the result might be multiple EPGs.  I used a simple for loop to process the Domain list, but came up with a small cosmetic problem in the output.  If it doesn’t worry you – you’re done!

admin@apic1:~> for i in "${domainList[@]}"; do moquery -d $i -c infraRtDomAtt | awk '{if ($1=="tDn") print $3}'; done

Note that some of the EPGs are repeated? Since your APIC bash shell has the normal linux command set, I decided to tidy up using Linux’s sort and uniq utilities.

Here’s version #2 of the above:

admin@apic1:~> declare -a epgList=($(for i in "${domainList[@]}"; do moquery -d $i -c infraRtDomAtt | awk '{if ($1=="tDn") print $3}'; done | sort | uniq))
admin@apic1:~> for i in "${epgList[@]}"; do echo $i; done

Task 4: Bonus task

With all that work put into creating the list, you’ll want to be able to use it again. Since the APIC runs Linux, you can turn the above into a bash script that will take any Interface Policy Group name as a parameter and produce a list of EPGs that use that Policy Group.  The script below is very basic, with virtually no parameter parsing, but if you cut and paste it to your APIC, you’ll be able to run it any time you want (until you wipe the APIC).  It also checks to see if the Policy Group is a VPC or PC, whereas my example above only checked for an Access Port Policy Group. The script has some other limitations too which I’ll list in this footnote.5

Here’s the script:

if [ "$1" == "" ] ; then
    echo "Usage: $0 InterfacePolicyGroupName"
    exit 1
aaepDn=$(moquery -d uni/infra/funcprof/accportgrp-$ipgName -c infraRsAttEntP | awk '{if ($1=="tDn") print $3}')
if [ "$aaepDn" == "" ] ; then
    aaepDn=$(moquery -d uni/infra/funcprof/accbundle-$ipgName -c infraRsAttEntP | awk '{if ($1=="tDn") print $3}')
    if [ "$aaepDn" == "" ] ; then
        echo "Interface Policy Group $ipgName not found"
        exit 1
declare -a domainList=($(moquery -d $aaepDn -c infraRsDomP | awk '{if ($1=="tDn") print $3}'))
declare -a epgList=($(for i in "${domainList[@]}"; do moquery -d $i -c infraRtDomAtt | awk '{if ($1=="tDn") print $3}'; done | sort | uniq))
echo "Tenant               Application Profile  EPG"
echo "-------------------------------------------------------------------------"
for i in "${epgList[@]}"
    declare -a j=($(echo $i | sed 's:uni/tn-::; s:/ap-: :; s:/epg-: :'))
    printf "%-20s %-20s %-20s" ${j[0]} ${j[1]} ${j[2]}
    echo ""

To create the script with a name of say , prepare yourself by making sure you have the entire script from above copied into your paste buffer, then start at the apic1# prompt, and enter the emphasised commands below

apic1# bash
admin@apic1:~> touch
admin@apic1:~> chmod +x
admin@apic1:~> vim

When the VIM editor opens, follow these steps precisely

  1. Press the letter i              [You will see — INSERT — in the bottom LH corner]
  2. Paste the buffer
  3. Press the <Esc> key      [– INSERT — will disappear]
  4. Press : [the <colon> key]   [A : prompt will appear in the bottom LH corner]
  5. Type wq
  6. Press <Enter>

Here’s the output of a test run from my system:

admin@apic1:~> ./ T5:SA.Host_APPG
Tenant               Application Profile  EPG
Tenant5              2Tier_AP             AppServers_EPG
Tenant5              2Tier_AP             WebServers_EPG

Enjoy your moquerying – I’ve added an Appendix with some background stuff you may find interesting.


Appendix: Some geeky background on moquery

Did you know that moquery is actually NOT an NX-OS ACI command, but operates from the underlying unix os?

This can be easily seen from the bash prompt, or “Object Model CLI” as it is officially known as.

apic1# which moquery
moquery: aliased to _exec_legacy_cmd "/controller/bin/moquery" "$@"
apic1# bash
admin@apic1:~> which moquery

In fact, ACI didn’t have a NX-OS command line until version 1.2, which his why you will not find any reference to moqurey in the Cisco APIC NX-OS Style Command-Line Interface Configuration Guide.  To get the (totally inadequate) official guide to moquery, you have to find the Cisco APIC Object Model Command-Line Interface User Guide, which has not been updated since ACI v1.1.

The upshot of this is that moquery is not restricted to just the APIC. You can run moquery from:

  • The APIC NS-OX prompt
  • The APIC bash prompt
  • The bash prompt on leaves and spines

Of course, when used on Spines and Leaves, you will only have access to the objects that are configured on the said leaf or spine.

Another interesting titbit is that the moquery program is written in python – you can see it right there on the APIC. Just type (from the bash prompt) cat $(which moquery) to see the source code.

More Mojo, mobrowser, modelete …

There are other mo commands on the APIC too.  Although, like moquery, the documentation is a bit sparse.  There is

mo utility Description
mobrowser My favourtie. Try it.
moconfig Usually in the form moconfig commit to commit changes made by other commands
mocreate Create an object. Better know what you are doing. Must follow with moconfig commit
modelete Also requires moconfig commit
moprint You can use this to turn the output files displayed using the cat command into json or xml
moquery You know this one now
moset You can set MO values directly
mostats Set up periodic sampling

If documentation for the mo-utilities is sparse, the opposite is true for the Manage Information Model itself. You can browse the docs for the entire model for wany version of sotware at and you APIC has a local copy for the installed version, although it can loose diagram at times. You can reach your local copy of course from the APIC GUI by selecting the cog-wheel settings icon and navigating to Documentation > API Documentation.

  1. klass is a deliberate misspelling of the word class to avoid confusion with the reserved word class 
  2. See my post on Cisco ACI Naming Standards for more details 
  3. See my discussion on the Cisco Community forum regarding the moquery bug 
  4. In my answer on the Cisco Community Forum I used egrep and sed instead of awk to get the same result. 
  5. There are a number of shortcomings in this script that I know about. 1. If you have an EPG linked to a Domain that is linke to an AAEP that is linked to the Interface Policy Group you queried on, it will still show up in the list, even if there are no active EndPoints using the initial Interface Policy Group. 2. If you have a) Not enabled the Enforce Domain Validation (System>System Settings>Fabric Wide Settings)  opton, and b) configured a mapping from the AAEP to an EPG, the script will not find that EPG.  Thanks to Sergiu Daniluk for pointing out my shortcomings! 
Posted in ACI, ACI Tutorial, Cisco, Master Class | Tagged , , , , | 1 Comment

How to find powerpoint slides that don’t fit the template

If you have ever copied a Powerpoint presentation from an old compay template to a new one, you will find that any slides that have been altered even slightly from the template cause a new slide layout to be added to your template – it will be give the same name as the original with the characters 1_ prefixed.  You can see what I mean by pressing Cmd+Opt+1 to open slide Master View* and hovering the mouse over the slide layout.

This means that these slides (slides 10,30 plus who knows how many others in my illustration above) are not going to reflect any change made to the original slide master – which in my illustration above was called Two Content Layout.  Often this is now a big deal, but sometime it can cause all sorts of wierd re-arrangements of layout.

I needed a way of quickly identifying which slides had been copied across and not matched the new template.  The Microsoft super-unfriendly way of hovering the mouse over the slide and hoping the list of slide numbers would appear to totally inadequate and extremely inefficient. And just sooo frustrating.

So this is what I did to find the powerpoint slides that didn’t fit the template in four simple steps:

  1. Add a marker slide to the end of the template
  2. Select ALL the layouts after the marker slide
  3. Change the slide background colour
  4. Fix your slides by re-applying the correct layout
  5. Delete the erroneous layouts

Here are the steps in detail and pictures.

Step #1. Add a marker slide to the end of the template

This picture taken from slide master view says it all

Step#2 Select ALL the layouts after the marker slide

When you paste your slides into the new presentation, PP will add all of the imported layouts at the end of the list in Slide Master view.

Use Click -> Shift-click to slelect all the extra layouts.

Step #3 Change the slide background colour

With all the extra layouts selected, right click on one of the layouts to bring up the menu, and choose Format Background – I choose a colour like pink. Don’t make the colour too bright, you won’t be able to read your slides.

Step #4 Fix your slides by re-applying the correct layout

Now with the new background colour, your slides can be easiliy identified when you return to normal view.  Unfortunately, the process of re-applying the master to each slide is still time consuming, but is probably best done one slide at a time because sometimes things don’t go as planned when the template is re-applied, especially if you have had others edit the slides who pay no attention to which layout they use.

Step #5 Delete the erroneous layouts

Once you’ve been through your slides, you can go back to the Slide Master view and select all your coloured background layouts and delete them.


*Windows users may have to use this trick to get quickly to Slide Master view – it works on Macos as well. Stolen from this source

Holding the SHIFT key and clicking on the Normal View icon in the lower right-hand corner of your screen will take you to the Slide Master View of your presentation

Posted in Microsoft, PowerPoint, tutorial | Comments Off on How to find powerpoint slides that don’t fit the template

USB-A for Apple Magic Mouse – Apple, you’ve GOT to be joking!

I have just bought a Magic Mouse 2. I bought my Mac Book Pro in 2016. The instructions in th MM2 say “To Pair you mouse with your Mac, use the Lightning to USB cable that came with your Mouse”.

However, the cable that came with my mouse is a Lightning to USB-A cable, wheras all Mac Books sold since 2016 (or thereabouts) have had USB-C (sold as Thunderbolt) ports.

I believe Apple should supply me with an appropriate cable (Lightning to USB-C) or at the very least an adapter to convert the ancient USB-A cable to USB-C.

I’ve submitted the above to Apple Feedback. But I’m not holding my breath waiting for Apple to send be a suitable cable.

How can Apple be allowed to sell periperals that are incompatible with the products that they are meant to support?

Just another case of Apple not caring about the PC world.


Posted in Apple, rant | Comments Off on USB-A for Apple Magic Mouse – Apple, you’ve GOT to be joking!

Configuring Link Speed on UCS 6545 10/25G ports

Configuring Link Speed on UCS 6545 Fabric Interconnects as 10/25G ports is not at all intuitive.  You would think that right-clicking on a port under Equipment > Fabric Interconncets > Fabric Interconnect A (or B) |> General would give you the option to change the links speed (it does for 40G ports, but not 10/25G ports)

So the secret is to double-click on the port to bring up the Properties window.

From here you find the Show Interface option, and once that is opened, you can find the Admin Speed setting.

Imagine calling the option “Show Interface” rather than “Configure Interface”.



Posted in Cisco, Data Center, Data Centre, Hyperflex, UCS | Tagged , | Comments Off on Configuring Link Speed on UCS 6545 10/25G ports

ACI Inband Mangagment Route Leaking Kludge

When I was challenged with this:

Hi @RedNectar ,

Right now I have a simple contract that allows SSH only:

  • Scope set to global.
  • TCP dst 22.
  • “Both directions” and “reverse port filters” enabled.

This contract is provided by the inband EPG at the “mgmt” Tenant and exported to tenant B. EPG at Tenant B consumes the contract interface. Can’t SSH the APICs or switches from a VM in Tenant B. Am I missing something?

I realised my earlier post didn’t cover this scenario where the management workstation was in another Tenant. So here’s the update.

Much like my earlier post, you will have to create an Access Policy Chain to associate a VLAN ID with the interfaces the APICs attach to.  For my example I used VLAN 2002 and my APIC is attached to Leaf 102 & 102 on port 47.

Begin with the Access Policy Chain

Here’s how I built my access policy chain in pictures. Note that I already had Leaf Profiles and Interface Profiles built for leaf 101 & 102;

I was going to need a VLAN pool to specify my chosen VLAN – VLAN 2002, so I created one:

Fabric > Access Policies > Pools > VLAN >+ Create VLAN Pool

Name: mgmt:inb_VLAN.Pool
Allocation Mode: Static Allocation
(+) Encap Blocks:
Range: VLAN 2002VLAN 2002

The VLAN Pool needed a Physical Domain, so again…

Fabric > Access Policies > Physical and External Domains >+ Create Physical Domain

Name: mgmt:inb_PhysDom
Vlan Pool: mgmt:inb_VLAN.Pool

which needed an Attachable Access Entity Profile of course…

Fabric > Access Policies > Policies > Global > Attachable Access Entity Profiles >+ Create Attachable Access Entity Profile 

Name: mgmt:inb_AAEP
(+) Domains (VMM…Exernal) To Be Associated To Interfaces:
Domain Profile: mgmt:inb_PhysDom

I had to make sure my APICs had LLDP enabled when I made the Access Port Policy Group to link to the AAEP – I already have a suitable policy that would do that as you can see here when I created the Leaf Access Port Policy Group:

Fabric > Access Policies > Interfaces > Leaf Interfaces > Leaf Access Port >+ Create Leaf Access Port Policy Group 

Name: mgmt:APIC_APPG
LLDP Policy:  Enable_LLDP
Attached Entity Profile:  mgmt:inb_AAEP

And to finish the chain, I created interface selectors for each APIC in my existing Interface Profiles.

So that’s the Access Policy Chain done for the APICs

Configuring the mgmt teanant for inband management

Now to set up inabnd Management in mgmt tenant. Apart from the route leaking trick below, most of this is just following my earlier post where I did a more complete description. I began by adding the inband management IP to the pre-defined inb Bridge Domain.

Then created the In-Band Management EPG, which is a special EPG.

Tenants > mgmt> Node Management EPGs >+ Create In-Band Management EPG 

Name: inb_EPG
Encap:  vlan-2002
Bridge Domain:  inb

This raised an annoying problem, described in bug CSCuz59329

So I fixed it using the work around described in the bug report. In other words I created a Management Node Connectivity Group. In the process, I calso created an IP address pool for the group. I’m not sure if I really needed to create an address pool, but I did anyway.

Tenants > mgmt> Managed Node Connectivity Groups >+ Create Managed Node Connectivity Group

I then created Static Node Management Addresses. first for the APICs, which trhew up a warning that had me check (and change) the default preference for management back to oob.  And then added more static addresses for Leaves and Spines, but it’s really the APICs that matter.

Tenants > mgmt> Static Node Management Addresses >+ Create Static Node Management Addresses

Then the warning…

so I fixed that!

System > System Settings > APIC Connectivity Preference

And checked that my IP had stuck using the ifconfig bond0.2002 command (recall I allocated VLAN 2002 to inb managment)

And did a ping test to the default gateway IP to be sure:

So at last, my inband management PEG was set up. It was time to test the challenge given me, which said in part:

Right now I have a simple contract that allows SSH only:

  • Scope set to global.
  • TCP dst 22.
  • “Both directions” and “reverse port filters” enabled.

Filters and Contracts

I already had an SSH filter in the common tenant, so I created a Contract there too ready to do the test.

[Note: I later decided that the Contract would be better created in the mgmt tenant, because haveing the contract in the common tenant will allow ALL Tenants access to the inband management IP network]

Tenants > common> Contracts > Standard >+ Create Contract -> is what I did

Tenants > mgmt> Contracts > Standard >+ Create Contract -> is what I should have done

I configured the inband management EPG to Provide this contract.

But now I was stuck – I needed a tenant to consume the contract. So back to the question:

This contract is provided by the inband EPG at the “mgmt” Tenant and exported to tenant B. EPG at Tenant B consumes the contract interface.

Creating a test tenant

So I created a Tenant, and of course called it TenantB

TenantB needed a Bridge Domain and an EPG, so I created those too, making sure that I checked the Shared Beteen VRFs option for the Bridge Domain when I created the subnet for the BD.  I also created the Application Profile on the way.  I already had a host connected on the subnet attached on interafce 102/1/26, so I added that host to the EPG in the process, and made sure it consumed the common/SSH.Global_Ct

Tenants > TenantB > Networking > Bridge Domains >+ Create Bridge Domain

Tenants > TenantB > Application Profiles >+ Create Application Profile

Fantasic. So now I had completed everything, but I had one little worry that I wanted to check.

Route Leaking conundrum

My worry was about route leaking.  You see, the Consumer EPG is in a different tenant and different VRF to the Provider EPG, so to make route leaking work I must do these two things:

  1. Enable the Shared Between VRFs on the Bridge Domain or EPG Subnet of the Consumer EPG (which I had done )
  2. Enable the Shared Between VRFs on the EPG Subnet of the Provider EPG which is the special mgmt tenant’s Node Mangement EPG for In-Band.

So I went looking to how I could add an EPG Subnet to the mgmt tenant’s Node Mangement EPG for In-Band. I found an option to add a subnet, so I did that, but NOWHERE was I able to click any Shared Between VRFs option.

I thought I’d check leaf 102 to see if any routes had leaked between the VRFs, and as expected, TenantB’s route had leaked into the mgmt tenant, but without the ability to make the subnet shared between VRFs on the EPG, TenantB’s VRF had no knowledge of the inband management subnet.

Here’s the kludge I used to fix it

I knew that what I needed to do was somehow to get the inband management IP subnet into the routing table for TenantB.  And I knew that to do that, I needed to either:

  1. Add an EPG Subnet with the shared between VRFs option set on the EPG,
  2. make the mgmt tenant become the consumer of a contract that was provided by TenantB’s EPG.  I figured this would work because I was at least able to check the shared between VRFs option on the inband management DB.

I tried option 1 first, and created an Application Profile and EPG in the mgmt tenant, added the subnet and checked the shared between VRFs option, and had it also provide the common/SSH.Global_Ct contract.

And sure enough, the routing table was happy.

All that was left to do was to test the validity of the contract form TenantB’s host:

EUREKA! So they say.


I didn’t like the solution.

Because what I had created was a contract in the common tenant that was provided by the inband management tenant, and could therefor be consumed BY ANY TENANT. In other words, I had allowed open access to the management network to any EPG in any tenant that cared to consume the common tenant’s SSH.Global_Ct contract. I’m sure any worthwhile security manager would have something to say about that.

To mitigate this, I considered option 2 above. Make the mgmt tenant become the consumer of a contract that was provided by TenantB’s EPG.  I tried this for fun using the same SSH.Global_Ct contract, and it worked, but didn’t mitigate the problem. Any EPG that wanted to consume the same contract would have access to the inband management subnet.  And I could see that while ever I was using a contract in the common tenant, I wassn’t going to win.

So I had to move the contract from the common tenant to the mgmt tenant, which also meant that I had to export the contract to TenantB, and then in TenantB, consume the contract as a Consumed Contract Interface. I still faced the route leaking problem, and still had to create the Application Profile + EPG + Subnet with the Shared Between VRFs option to make it work, but at least I ended up with something that I was a little happier with.

So, there you have it. That’s how you can configure inband management so a tenant can access ACI management.


Posted in ACI, aci inband management, ACI inband management tutorials, ACI Tutorial, Cisco | Tagged , , , | 2 Comments

Understanding Scope Of Prefixes in L3 Out External EPG in ACI —

Excellent post on unofficialaciguide – hightly recommended.

In ACI the external Routing Peer to the router is done through border leaves with a object called L3Out. L3Out has an object in it called the L3Out InstP also known as the External EPG. 1,669 more words

via Understanding Scope Of Prefixes in L3 Out External EPG in ACI —

Posted in GNS3 WorkBench | Comments Off on Understanding Scope Of Prefixes in L3 Out External EPG in ACI —

Validating IP Address Entries in Excel

Problem:  You are designing a spreadsheet where IP addresses are to be entered. Probably with subnet masks as well.  You want to ensure that the IP addresses and subnet masks entered are valid.

In this series I will explain how this is done, plus a few other IP address manipulating tricks in Excel.

Validating IP Address Entries in Excel

Firstly you need to understand that you can add validation to any Excel cell  by selecting a cell then choosing Data > Validation, (in the Data Tools section of the ribbon).  In this case, use a custom criteria based on a formula.

Now come the tricky bit.  The formula has to be less than 255 characters long, and although there may be more elegant ways of expressing the formula below, I haven’t found one that uses fewer than 262 characters, so you have to choose a compromise.  Even after compromising, the formula is only good for cells that have 2 to 4 characters in the cell reference. In other words, this formula won’t work in cell A1000, or AA100 or AAA10 because the cell reference is too large and makes the formula spill over 255 characters.

Formula for validating IP addresses

The formula below is for cell C8, and has been padded with line breaks and extra spaces for readability.  This version is way past the 255 character limit, so check below for a compromise that suits you.


= AND(
LEN(C8)-LEN(SUBSTITUTE(C8,".","")) = 3,
--LEFT(C8,FIND(".",C8)-1) < 224,
--LEFT(C8,FIND(".",C8)-1) > 0,
--MID(SUBSTITUTE(C8,".","    "),6,5) < 256,
--MID(SUBSTITUTE(C8,".","      "),15,7) < 256,
--MID(SUBSTITUTE(C8,".","      "),22,10) < 256,

And this is how it works:

The AND function is to ensure all the following conditions are met.  The first condition checks to see if there are precisely three “dots” in the IP address by replacing the “.” characters with null strings i.e SUBSTITUTE(C8,”.”,””). If there are just three  “.” characters then the resulting string will be three characters shorter than the whole string with the “dots” included:

LEN(C8)-LEN(SUBSTITUTE(C8,".","")) = 3,

The next condition is to check that the first octet is less than 224 (if you wished to allow multicast addresses, you would check that the first octet is less than 240). So simply extract the characters to the left of the first dot FIND(“.”,C8)-1 and check that the result is less than 224.

--LEFT(C8,FIND(".",C8)-1) < 224

But hold on – what are those two minus signs doing before the LEFT function?

Well, the problem is that the LEFT returns a string value, and if you compare a sting value with a numeric value like 224, the string value will always be larger, so in Excel, a test of:


will always yield a result of FALSE

However, if you perform a numeric operation on the string, like adding zero, or finding the negative value of the string, Excel automagically transforms the string into a number. So, in the case above, the double negative is used to turn the string result into a numeric result so that the comparison works the way you expect.  If the double negative worries you, add a 0 to the result  instead. So instead of –LEFT(C8,FIND(“.”,C8)-1) < 224 use:

0 + LEFT(C8,FIND(".",C8)-1) < 224

and you will get the same result. And it uses the same number of characters.

The next comparison is exactly the same as the previous, except it is checking that the first octet of the IP address is larger than zero.

--LEFT(C8,FIND(".",C8)-1) > 0

So that takes care of the first octet, which we extracted by using the LEFT() function.  But to extract the second and subsequent octets requires a but more lateral thinking.

You could try and extract the second octet by looking for the text between the first and second dots. Which is how we mentally do it. But in Excel this would be the mind-blowingly complicated:


and which would chew up far too many of our precious 255 character limit.

But a far more elegant way (the basic idea for which I stole from one of the references below, many of them use a similar approach) is to expand the ip address into sections by replacing the dots with a number of spaces (four in this case), and then extracting that portion of the string where the second octect must reside.

The SUBSTITUTE(C8,”.”,”    “) part takes care of replacing the dots with four spaces. So IP addresses of and get expanded to (using the ∙ character to represent spaces to make them easier to count):


and if you now extract 5 digits from this string beginning with digit 6, MID(SUBSTITUTE(C8,”.”,”    “),6,5) you will get either 2∙∙∙∙ or ∙∙145.  Using the double-negative trick again turns either of these results into a number that can be checked to ensure it is less than 256, which is the condition that must be met for octets 2-4.

Moving onto the third octet, the logic is almost identical, except more spaces need to be inserted – six in fact.  (The only reason 4 spaces were used for octet 2 was to save 2 characters out of our limited budget). And this time 7 digits are extracted starting with digit 15.


And for the final octet, one more subtle change takes place.  Some of the references I’ve read extract the next nine digits from the above to determine the last octet – but that fails if you enter 4 digits in the last octet, so to be sure to catch the condition where someone types the final check extracts 10 digits beginning with digit 22.


The last condition is to check that no additional spaces or operators have been inserted ISNUMBER(–SUBSTITUTE(C8,”.”,””)). This helps ensures that no-one writes an IP address with negative numbers such as 1.2.3.-4.  However it is not foolproof.  An IP address of say 1.2.-1.2 will be converted to the string “12-12” which Excel with all its smarts sees a number. What number you say? Well, here’s a hint: 12-12 will be seen as 43811 in 2019, and as 44177 in 2020. Got it? 12-12 is seen as 12 December of the current year.

Choose your crutch – which condidtion do you want to remove?

Remember I mentioned that the solution is too long? Here it is again as a reminder, in a slightly different order:

= AND(
LEN(C8)-LEN(SUBSTITUTE(C8,".","")) = 3,
--LEFT(C8,FIND(".",C8)-1) > 0,
--LEFT(C8,FIND(".",C8)-1) < 224, 
--MID(SUBSTITUTE(C8,".","    "),6,5) < 256,
--MID(SUBSTITUTE(C8,".","      "),15,7) < 256,
--MID(SUBSTITUTE(C8,".","      "),22,10) < 256

To use the formula as a validation criteria, you have remove something to bring it below the 255 character limit.

Compromise#1: Ignore more than three dots

If you remove the first condition (LEN(C8)-LEN(SUBSTITUTE(C8,”.”,””)) = 3) then not only will you will be able to enter IP addresses that end in a training dot, you’ll be able to enter addresses like…  As shown below though, you COULD catch this with conditional formatting, and do something about it, such as highlighting the cell in red.

Here’s a cut & pastable version of the testing validation criteria:

=AND(--LEFT(C8,FIND(".",C8)-1)<224,--LEFT(C8,FIND(".",C8)-1)>0,ISNUMBER(--SUBSTITUTE(C8,".","")),--MID(SUBSTITUTE(C8,"."," "),6,5)<256,--MID(SUBSTITUTE(C8,"."," "),15,7)<256,--MID(SUBSTITUTE(C8,"."," "),22,10)<256)

Compromise#2: Allow zeros and negatives

If you remove the second condition (–LEFT(C8,FIND(“.”,C8)-1) > 0) then not only will you allow IP addresses beginning with 0, but also negative addresses in the first octet. (The ISNUMBER(–SUBSTITUTE(C8,”.”,””)) condition attempts to take care of negatives in the other octets).  Personally, I’d remove this condition before removing the check for 3 dots.

Here’s a cut & pastable version of the testing validation criteria without the test for the first octet being greater than 0.

=AND(LEN(C8)-LEN(SUBSTITUTE(C8,".",""))=3,--LEFT(C8,FIND(".",C8)-1)<224,ISNUMBER(--SUBSTITUTE(C8,".","")),--MID(SUBSTITUTE(C8,"."," "),6,5)<256,--MID(SUBSTITUTE(C8,"."," "),15,7)<256,--MID(SUBSTITUTE(C8,"."," "),22,10)<256)

Compromise#3: Allow spaces and negative octets

If you remove the condition (ISNUMBER(–SUBSTITUTE(C8,”.”,””))) then you can insert spaces into the IP address, which probably doesn’t matter much.  It will also allow negative octets too, although even with the test some negatives get interpreted as dates anyway.  I see this as the least useful test, and is my preferred test to omit.

And of course, in cut and pastable form:

=AND(LEN(C8)-LEN(SUBSTITUTE(C8,".",""))=3,--LEFT(C8,FIND(".",C8)-1)<224,--LEFT(C8,FIND(".",C8)-1)>0,--MID(SUBSTITUTE(C8,"."," "),6,5)<256,--MID(SUBSTITUTE(C8,"."," "),15,7)<256,--MID(SUBSTITUTE(C8,"."," "),22,10)<256)

So there you have it, data validation for IP addresses in a single cell.  Next time I’ll show to validate subnet masks that can only contain the values 255,254,252,224,192,128 and 0 in any octet.



This is the one that got me started – Glenn Kennedy’s answer to an IP Address conversion formula and

Both these sites have Ron Rosenfield’s validation for cell I1 as

--(MID(SUBSTITUTE(I1,".",REPT(" ",99)),99,99))<256,
--(MID(SUBSTITUTE(I1,".",REPT(" ",99)),198,99))<256,
--TRIM(RIGHT(SUBSTITUTE(I1,".",REPT(" ",99)),99))<256)

but this has too many false positives. For instance, it would allow an IP address of which is invalid.

Apart from the fact that it didn’t test the last octet for values greater than 255, XOR LX‘s suggestion of:

=SUMPRODUCT(N(LOG(1+MID(SUBSTITUTE(B1,".",REPT(" ",10)),{1,11,21,31},10),2)<=8))

was interesting, but a) you can’t use SUMPRODUCT in cell Validation criteria, b) when expanded to make it work takes more characters than the solution I’ve used and c) was missing the compaison criteria.  It should have been:

=SUMPRODUCT(N(LOG(1+MID(SUBSTITUTE(B1,".",REPT(" ",10)),{1,11,21,31},10),2)<=8))=4

It was interesting because it made use of the fact that the log base 2 of 256 is 8, and less than 8 for any number less than 256.  The SUMPROUCT counts the number of octets that have a log base 2 equal to or less than 8, which of course should be 4.



Posted in GNS3 WorkBench | Tagged , , , , | 1 Comment

Seven things to know to make Hyperflex go – Cisco HyperFlex Best Practices

You have Cisco Hyperflex installed, but not quite sure if there is anything you need to do differently now that want to deploy VMs on the Hyperflex Data Platform (HXDP)

Well, yes there are some things that you need to do differently, and some you should do differently, but most activities you’ve ever done with VMs and ESXi hosts will remain the same.

Here are the seven things you need to know to make Hyperflex perform optimally.

Create new users for the HX Connect GUI from vCenter (Must Do)

Create Datastores using the HXDP utilities, not vCenter standard Datastore creation (Must Do)

Create Snapshots using the HXDP utilities, not vCenter standard Snapshot function (Must Do)

Create multiple Clones using the HXDP utilities (Should Do)

Use HXDP Maintenance Mode, not vCenter standard Maintenance Mode (Must Do)

Upgrade ESXi software using HX Connect or Intersight, not vCenter (Must Do)

Create new VLANs using the HX Installer VM (Should Do)

Create new users for the HX Connect GUI from vCenter (Must Do)

There is a default admin user that can be used to log into the HX Connect GUI, but best practice is to use your vCenter username and password.  If you want to add a read-only or another administrator user for Hyperflex, use the regular method for creating a user in vCenter.

Create Datastores using the HXDP utilities, not vCenter standard Datastore creation (Must Do)

Before you can even begin to use Hyperflex, you must create a Datastore on your HXDP.  Since you will only do this rarely, it is an easy point to forget, which can lead to frustration if you try to do this using the normal Datastore creation method from vCenter, because vCenter will want to assign the Datastore to a single ESXi host, whereas the HX Datastore will be distributed across all HX ESXi storage nodes. In other words, you can’t create a Datastore on your HXDP using the normal Datastore creation method from vCenter.  Creating a HX Datastore can only be done in one of two ways, using Hyperflex Connect (easy) or the vCenter plugin (messy).

Method #1: Using HX Connect:

Click Datastores and then Create Datastore (I said it was easy)

Figure 1 Creating a Datastore in Hyperflex Connect is easy


Method #2: Using vCenter:

Navigate to Global Inventory Lists > Cisco HX Data Platform.  Next, select your cluster in the Navigator then click on the Manage tab. From here, click on the Datastores sub-tab where you will find an icon that will let you create a Datastore.

Figure 2 Creating a Datastore in vCenter is messy


Create Snapshots using the HXDP utilities, not vCenter standard Snapshot function (Must Do)

This one is a bit tricky.  When you create the first Snapshot in one of the two HXDP methods, a special SENTINAL snapshot is created.  This ensures that any future snapshots can trace their pointer-based log-structured file system origins back to the original format.  IF YOU CREATE THE FIRST SNAPSHOT using the VMware standard Snapshot functions, then you are stuck with the VMware Re-do snapshot system and will be stuck with the non-HX aware consolidation process should you wish to consolidate in the future.

The VMware Re-do snapshot system works like this:  The first snapshot is made, and the .vmdk file for that VM is locked. A new .vmdk file is created to record any changes that are made after the original snapshot is made. Similarly, when the next snapshot is made, the second .vmdk file for that VM is locked and a new one created, and so on.  The problem with this method is that not so much that no data is ever deleted, and the size of the snapshot re-do files may become far greater than the original, but that if you find that you need to reclaim the space, VMware now has to revert to the original snapshot and process the Re-do files.  This process can take minutes, hours or even days depending on the size and complexity of the Re-do files – and there is a possibility that the process may exhaust your existing disk space before completion (after all, you are probably doing the consolidation to reclaim some space).

Hyperflex consolidation works like this:  All the pointer-based snapshots are deleted in a matter of seconds, and the redundant chunks of data marked for deletion. Job done. In seconds. And no chance of running out of disk space.

Moral of this story: Use Hyperflex pointer-based snapshots for the first snapshot for all VMs.  And to ensure this happens, why not take the approach of using Hyperflex pointer-based snapshots for all snapshots?

Here are the two ways of creating HXDP pointer-based snapshots:

Method #1: Using HX Connect:

Click Virtual Machines and then from the Actions menu, select Snapshot Now

Figure 3 Creating Snapshots in Hyperflex Connect


Method #2: Using vCenter (must be Flash version of vCenter, not HTML):

Navigate to Global Inventory Lists > Cisco HX Data Platform.  Next, select your cluster in the Navigator then click on the Manage tab. From here, click on the Datastores sub-tab where you will find an icon that will let you create a Datastore.

Figure 4 When creating HXDP Snapshots in vCenter you need to be careful


Create multiple Clones using the HXDP utilities (Should Do)

If you want to create one clone of a VM, it does not matter whether you use the standard VMware cloning option or the HXDP Ready Clones option.  The main thing to remember is that to take advantage of the Hyperflex pointer-based log-structured file system’s inherent de-duplication, you must make sure that the VM you are cloning does not have any VMware Redo snapshots – only HXDP pointer-based snapshots.  However, when you use the HXDP Ready Clones option, you’ll be able to create as many clones as you want without taking any extra disk space because of the Hyperflex pointer-based log-structured file system’s inherent de-duplication.  One thing though, if you want to use Customisation Specifications or Resource Pools, you’ll have to have already created them in vCenter.

Figure 5 Creating HXDP Ready Clones in HC Connect


Figure 6 Creating HXDP Ready Clones in vCenter


Use HXDP Maintenance Mode, not vCenter standard Maintenance Mode (Must Do)

Should you ever need to shut down an ESXi storage host (and you will need to sometime), make sure you use the HXDP Maintenance Mode, not vCenter standard Maintenance Mode. The difference is this:  When you shut down an ESXi storage host using the HXDP Maintenance Mode, the HXDP Controller VM will be shut down cleanly, and the other ESXi hosts will be informed that the HXDP Controller VM for that host is no longer available. To understand the re-percussions of this, you need to be aware of how the HX IOVisor works (which I hope will be the subject of a future blog post).  When you use the standard VMware Maintenance mode, VMware doesn’t know that the HXDP Controller VM needs to be shut down gracefully and the other ESXi hosts need to be informed that the HXDP Controller VM for that host is no longer available, so shuts down the ESXi host regardless.  Now, the HXDP will recover for this mis-hap in time (in most cases), but it is certainly NOT “best practice”

Figure 7 Entering HXDP Maintenance Mode using HX Connect


Figure 8 Entering HXDP Maintenance Mode using HXDP extensions in vCenter


Upgrade ESXi software using HX Connect or Intersight, not vCenter (Must Do)

When the ESXi software needs to be upgraded, remember the ESXi software on your HXDP hosts has been modified by the addition of the IOVisor and the VAAI .vibs (VMware Installation Binaries), so upgrading the ESXi hosts is NOT a simple matter of upgrading using VMware’s released version – you need the ESXi versions released by Cisco with the appropriate .vibs installed.  The simplest way to do this is to make sure you do the upgrades using Hyperflex Connect, or better still (if you have more than one site) Cisco’s Intersight SaaS platform ( which can upgrade multiple HXDP sites in a single click! (Pause while I sip come more Kool-Aid). BTW, even if you have only one site, you should still connect your cluster to Intersight, but I’ll talk about that in a future post.

Figure 8 Upgrading using HX Connect


Create new VLANs using the HX Installer VM (Should Do)

If you even need a new VLAN on the HX data platform, you need to make sure that VLAN is available to each ESXi host and each Fabric Interconnect, and your upstream switches.  Since Hyperflex was designed to run on ROBO versions of VMware and above, standard vSwitches are maintained in each ESXi host, so Cisco has provided a utility that allows you to quickly create a new VLAN on all hosts plus the Fabric Interconnects in one step, a task that is quite tedious if you are not using vSphere Distributed Switches.  Of course, if you have a version of VMware that supports VMware VDS, you probably won’t want to use this feature (because it configures Standards vSwitches, not vSphere Distributed Switches).  You’ll be prompted to enter usernames and passwords for UCSManager, vCenter and the ESXi Hosts, so it is a bit tedious, but simpler and safer than adding the VLANs manually to each ESXi host and the FIs.  Here’s a sample session adding one VLAN – in this case it was a stretched cluster, so the VLANs were added to six hosts and four Fabric Interconnects in one hit!

root@HyperFlex-Installer:~# post_install --vlan
Logging in to controller
HX CVM admin password: **************
Getting ESX hosts from HX cluster...
vCenter URL:
Enter vCenter username (user@domain): administrator@vsphere.local
vCenter Password: **************
Found datacenter HX-DC
Found cluster HXCLUS01

post_install to be run for the following hosts:

 Enter ESX root password: **************
 Attempting to find UCSM IP
Site A - UCSM IP:
Site A - UCSM Username: admin
Site A - UCSM Password: **************
Site A - HX UCS Sub Organization: HX
Site B - UCSM IP:
Site B - UCSM Username: admin
Site B - UCSM Password: **************
Site B - HX UCS Sub Organization: HX
 Port Group Name to add (VLAN ID will be appended to the name): TestVLAN
 VLAN ID: (0-4096) 104
 Adding VLAN 104 to FI
 Adding VLAN 104 to vm-network-a VNIC template
 Adding VLAN 104 to FI
 Adding VLAN 104 to vm-network-a VNIC template
Adding TestVLAN-104 to sitea-esxi01.mynet.local
Adding TestVLAN-104 to sitea-esxi02.mynet.local
Adding TestVLAN-104 to sitea-esxi03.mynet.local
Adding TestVLAN-104 to siteb-esxi01.mynet.local
Adding TestVLAN-104 to siteb-esxi02.mynet.local
Adding TestVLAN-104 to siteb-esxi03.mynet.local
Add additional VM network VLANs? (y/n) n
Posted in Best Preactices, Cisco, Data Center, Data Centre, ESXi, Hyperflex, VMware | Tagged , , , | 2 Comments

MS Community Robot Finds Word Tables Offensive

I posted this question on the Micsoft Community board and in the course of the following discussion, I posted this picture of a MS Table with the Language incorrectly set.

But suddenly, it was removed from my discussion.  Replaced by the dreaded missing image icon, and the only indication of what had happened was in the History of the post:

Now I know many things about MS Tables are offensive, the fact that I can’t change the Language setting for a Table Stlye for one – but a picture of a table? Come on MS, what’s the deal? Did you think my arrow was too pointy? Or was the angle-of-the dangle at the wrong angel? Or was it that I posted a picture of a …. “box” (Snigger, snigger)

Whatever the reason, I’m pissed off that I can’t add reasonable pictures to a post.

Posted in Microsoft, opinion, rant | Comments Off on MS Community Robot Finds Word Tables Offensive

RedNectar’s Hyperflex Pre-Install Checklist (Updated)

Completing Cisco’s Pre-installation Checklist for Cisco HX Data Platform will capture all the information you need, but not necessarily in the order you need it, and for a single site only. So, I decided to write a version that gets the information in the order you need it and in a format that’s easier to expand into a multi-site or stretched-cluster deployment. If you are planning to install a Stretched Cluster, make sure Site#1 and Site#2 have identical configurations where indicated.

Logical Planning for the installation

Task 1:               Planning

Before embarking on the checklist, it is important that you know how your Hyperflex installation is going to integrate into your existing network, and what new resources (VLANs, IP subnets you will need to find).

VMware vCenter

Hyperflex cannot operate without VMware vCenter. You may use an existing vCenter installation or vCenter can be included as part of the Hyperflex installation. If it is to be included in the Hyperflex installation process, best practice is that vCenter be installed on an ESXi host that is NOT part of the Hyperflex cluster (to avoid having the management application hosted on the managed system). If absolutely required, vCenter can be installed on the Hyperflex cluster. Recommended options are bolded.

Fabric Interconnects

Typically, Hyperflex clusters are installed using an independent pair of Fabric Interconnect switches. If the Hyperflex cluster is to be installed and integrated with an existing UCS deployment, the Fabric Interconnect switches will be rebooted during the Hyperflex installation.

VLAN/vSwitch and Subnet planning

Your Hyperflex Installation requires that you plan for additional subnets and VLANs. Depending on the final design, this could be as many as four new subnets and four new VLANs. The following diagrams offer two slightly different approaches to VLAN planning:

HX Install Design #1: Separate OOB and HX Mgmt VLAN/Subnets

HX Install Design #2: Combined OOB/HX Mgmt VLAN/Subnets (recommended)

To determine the number of IP addresses and new VLANs required, use the following guidelines. In all the following calculations, n is the number of Nodes in the Hyperflex Cluster.

vCenter Subnet is assumed to be an existing subnet. If a new vCenter is being installed as part of this installation, IP addresses will need to be allocated for the vCenter appliance and its default gateway.

Total IPs required= 2

·       1 x Default GW IP

·       1 x vCenter Appliance VM

Installer VM Subnet is also assumed to be an existing subnet. If practical, make provision for the HX Installer VM to be allocated an IP address on the hx-inband-mgmt subnet. This ensures that the installer has full access to the ESXi hosts, and (in design #2) to UCS Manager. The Installer MUST be able to access vCenter, UCS Manager and the ESXi hosts.

Total IPs required= 2

·       1 x Default GW IP

·       1 x HX installer VM

UCS OOB Subnet is the external UCS OOB management – you probably have a VLAN set aside for this already and is where the UCS Manager Primary, Secondary and Cluster IPs live. These are configured on the Fabric Interconnects during the initial setup. Also, this subnet is where the ext-mgmt IP Pool for OOB CIMC is sourced, which will require at least n addresses. This subnet is a mandatory requirement.

Total IP addresses required=4 + n

·       1 x UCS Manager Primary (FI-A)

·       1 x UCS Manager Secondary (FI-B)

·       1 x UCSM Cluster IP

·       1 x Default GW IP

·       n x CIMC Pool addresses (one per HX Server)

Hyperflex inband management VLAN and Subnet is shown in the diagram as hx-inband-mgmt subnet/VLAN/vSwitch. You may already have a management VLAN that you wish to use in this situation. Indeed, it may even be the same subnet as the UCS OOB Subnet (as shown in design #2 above), in which case the number of subnets required is reduced by one. This VLAN MUST be configured before the install on the UPSTREAM SWITCHES and on the Port Channel between the upstream switches.

Total IP addresses required=2 + 2 x n

·       1 x Management Cluster IP

·       1 x Default GW IP

·       2 x n (vmnic0 and Controller VM per ESXi host)

·       [Optional] Consider reserving an IP address for the Installer VM on this subnet.

Hyperflex Storage VLAN and Subnet is shown in the diagram as hx-storage-data subnet/VLAN/vSwitch. This is an isolated Subnet/VLAN/vSwitch. Supplying a default gateway is optional unless your backup system accesses the storage directly. This VLAN MUST be configured before the install on the UPSTREAM SWITCHES and on the Port Channel between the upstream switches and requires MTU of 9000.

Total IPs required=2 + 2 x n (or 1 + 2 x n if no default gateway IP required)

·       1 x Data Cluster IP

·       1 x Default GW IP

·       2 x n (vmnic1 and Controller VM per ESXi host)

VMotion VLAN (and subnet) is shown in the diagram as hx-vmotion VLAN/vmk. No IPs are required at installation, but can be added post install, and in fact is recommended. This VLAN MUST be configured before the install on the UPSTREAM SWITCHES and on the Port Channel between the upstream switches.

Total IPs required= n (or 1 + n if default gateway IP required)

·       1 x Default GW IP (Optional)

·       n (vmnic2 per ESXi host)

VM Network VLAN is a VLAN that will be added to a standard VMware vSwitch which will be called vswitch-hx-vm-network.  This item exists more or less as a place-marker to remind you that you will need to plumb the VLANs used by the guest VMs to the HX Cluster. One VLAN is configured during the install process for this purpose, and no doubt you will be allocating IPs to VMs as they are added later. Additional VLANs can be easily added post-install and a section is provided later in this document to define any additional VLANs you might want to make available to your VMs.

Task 2:               Fabric Interconnects (FIs) – initial console config

The first thing to configure is the UCS Fabric Interconnects via a console cable. The following information will be required to be able to complete the initial configuration.

·       You will need to enter the items marked * again later, so remember them.

The UCS System Name is used to name the Fabric Interconnects, and the suffixes A and B will be added by the system. If you use OurDC.UCS.FI as the System Name, the first Fabric Interconnect will automatically be named OurDC.UCS.FIA and the second OurDC.UCS.FIB

Configuration Item




UCS Admin Password*

UCS System Name
(Recommend ending with letters FI)

FI-A Mgmt0 IP address
(Recommended a.b.c.7)

Mgmt0 IPv4 netmask (Common for all IPs)

IPv4 address of default gateway

UCS Cluster IPv4 Address*
(Recommended a.b.c.9)

DNS IP address*
(Comma separated, maximum two)

Domain name (optional)

UCS FI-B IP address
(Recommended a.b.c.8)

Task 3:               Fabric Interconnects – firmware upgrade

You may wish to assign the NTP server address to the Fabric Interconnects during the Firmware Upgrade.

Configuration Item




NTP Server IP*

Task 4:               Server Ports, Uplink Ports, FC Uplink ports on FIs

If using 6200 Fabric Interconnects, AND you plan on connecting Fibre Channel storage to the FIs now or in the future, remember these will have to be the high numbered ports and increment in steps of 2. So, for a UCS 6248, ports 31 and 32 will be the first FC ports. For a UCS 6296, ports 47 and 48 will be the first FC ports.

For UCS 6332-16UP FIs, the order is reversed. Only the first 16 (unified) ports are capable of 4/8/16Gbps FC – but they progress from left to right in pairs, so the first two FC ports will be ports 1 & 2.

The UCS 6332 doesn’t support FC, but both the UCS 6332 and the UCS 6332-16UP FIs have 6 dedicated 40Gbps QSFP+ ports – these are the highest numbered ports on the respective FI (27-32 on the UCS 6332, 35-40 on the UCS 6332-16UP)

RedNectar’s Recommendation:

Allocate server ports in ascending order from the lowest port number available.

Allocate uplink ports to the highest numbered ports.


If attaching using 10Gbps servers to UCS 6332-16UP reserve the lowest numbered ports for current and future FC connections.

The configuration of the Ethernet uplink ports is also a consideration. The original design should indicate whether you are using:

·       Port-channelled uplinks

·       Unbundled uplinks

Configuration Item




Server1 port number: (E.g. 1/1)
(Other servers should use incremental ports)

Number of server ports to configure:
(At least as many as there are nodes)

Port range for Ethernet uplinks: (E.g. 1/31-32)

Uplink Type




Port range for FC uplinks: (E.g. 1/1-4)

Task 5:               The Installer VM and Hyperflex configuration items

The Installer VM is required to be running to complete the installation from this point on. Ideally there will already a vCenter deployed and running on which the Installer VM can be deployed or has already been deployed. This VM needs access to the UCS Manager IPs defined in Task 2: and to the vCenter Appliance that will be used to manage the Hyperflex Nodes, as well as the IP addresses assigned to the ESXi Hosts and the Controller VM defined later in this task. The Installer VM also needs access to the DNS server used to resolve the IP addresses and names used for the Hyperflex ESXI hosts (both forward and reverse)*. For a full list of TCP port numbers required, see the Cisco HyperFlex Systems Installation Guide for VMware ESXi. For multi-site installs, the same Installer VM can be used for all installs if connectivity permits.


Technically it is vCenter that needs to be able to resolve ESXi host names in each direction, but it is generally easier to test from the installer VM than vCenter, and if both are using the same DNS server, the test should be valid.

Configuration Item




Will the Installer VM be already deployed in a vCenter before the installation begins?

Yes No

Yes No

Yes No

Installer VM IP Address
(Recommended a.b.c.100)

Installer VM IP Default Gateway




UCSM Credentials/config

For convenience during the install, re-enter items you’ve already defined in Task 2:

Configuration Item




UCS Manger hostname: (=Cluster IPv4 Address)

User Name





(UCS Admin Password)




Site Name:
(Only required if Stretched Cluster install)

vCenter Credentials

·       The vCenter password requires a special character, but should NOT contain special characters like <>[]{}/\’`”* – to be honest I don’t know the complete list, and I’m guessing at the list above based on my knowledge of how hard it is to pass some of these characters to a process via an API, which is what the Installer does to log into vCenter.

Configuration Item




If configuring a Stretched Cluster configure only ONE vCenter


vCenter Server (IP or FQDN)

vCenter username
(Typically username@domain)

vCenter Admin password (Please change if it contains special characters <>[]{}/\’`”*)

Hypervisor Credentials

The ESXi hosts in the cluster all come with a default user called root with a password of Cisco123 – you should NOT change the username, but you MUST change the password.

·       Like the vCenter password, this password requires a special character, but should NOT contain special characters like <>[]{}/\’`”* – furthermore, this password requires one Uppercase letter (two if the first character is Uppercase), one digit (two if the last character is a digit) and of course a lowercase character and has to be between 6 and 80 characters long.

Configuration Item




Recommend keeping consistency for Stretched Cluster


Installer Hypervisor Admin user name




Installer Hypervisor Admin user password




New Hypervisor Admin (root) user password (Min 6 Chars Upper/lower/special/digit mix)




Combined UCSM and vCenter VLAN/IP Config items

You will need at least four VLAN IDs, but not all have to be unique. Here are my recommendations:


Refer to the design diagrams above while completing these items

If you have a VLAN/subnet already planned for the UCS OOB Management, consider using that VLAN for the hx-inband-mgmt. (See design diagrams)

The VLAN ID for HX Storage traffic (hx-storage-data) does not need to be seen any further Northbound than the upstream switches that connect to the FIs. So, it is safe to use the same VLAN ID at every site. This will typically a new VLAN not used anywhere else.

    • You may well already be using a VLAN for vMotion. It’s OK to use the same one here.  In fact, if you are moving VMs from an existing environment, it’s a good idea.
    • One VM Network VLAN ID is required so you can deploy the first VM to the HX Cluster. More can be added later in the Post-Install task.  This vlan will be added as a port-group to a vSwitch called vswitch-hx-vm-network that is created on every ESXi host.
  • Each cluster should have its own MAC address range.  MAC addresses always begin with 00:25:B5: so that that much is filled in for you. Add just one more 2 digit hextet  to the prefix given. All HX ESXi hosts will be allocated MAC addresses using this prefix.

The hx-ext-mgmt Pool for OOB CIMC should have enough IPs to allocate one to every HX ESXi server now and in the future and be from the same subnet that the UCS OOB Management IPs were allocated. Refer to the design diagrams


Configuration Item
[Defaults in brackets]




If configuring a Stretched Cluster all values must be identical for both sites except MAC and IP Pools


VLAN Name for Hypervisor and HX Mgmt
[ hx‑inband‑mgmt]

VLAN ID for Hypervisor and HX Mgmt

VLAN Name for HX Storage traffic
[ hx‑storage‑data]

VLAN ID for HX Storage traffic

VLAN Name for VM vMotion
[ hx‑vmotion]

VLAN ID for VM vMotion*

VLAN Name for first VM Network
(More can be added later)

VLAN ID(s) for first VM Network
(More can be added later)

MAC Pool Prefix
(Add 1 more hextet e.g AA)




ext-mgmt IP Pool for OOB CIMC
(E.g. a.b.c.101-165)
For Stretched Cluster, use same subnet but don’t overlap

ext-mgmt IP subnet Mask

ext-mgmt IP Gateway

iSCSI Storage and/or FC Storage

If you plan to give the HX cluster access to remote iSCSI or FC storage, it will be simpler to configure it during the install.

Configuration Item
[Defaults in brackets]




iSCSI Storage






FC Storage

WWxN Pool










Advanced Items

UCS Manager stores configuration information for each server in a construct called a Service Profile then organises these along with other relevant policies in a construct called an Organisation. This is a required item.

·       If a Hyperflex Cluster Name is given here it will be added as a User Label to Service Profiles in UCS Manger for easier identification. Don’t confuse it with the Cluster Name required later on for the Storage Cluster. Except in the case of Stretched Cluster that may have “stretched” over two sites, I’d recommend a different Hyperflex Cluster name per Site.

·       Org Name (Organisation Name) the can be the same for each site but is probably better to have a naming plan. Organisation Names are used to separate Hyperflex specific configurations from other UCS servers in UCS Manager. I recommend using a consistent name for identical clusters – e.g. Stretched Cluster.

Configuration Item
[Defaults in brackets]




Hyperflex Cluster Name
[HyperFlex cluster]

Org Name
(E.g. HX-VDI)

Hypervisor Configuration

This is the bit where the DNS configuration is important. If your installer cannot resolve the names given here to the IP Addresses (and vice-versa) then the Servers will be added to the configuration using IP addresses only, rather than the names.


Refer to the design diagrams above while completing these items

Other advice:

The Server IP addresses defined here will be in the HX Mgmt VLAN (hx_inband_mgmt).

The subnet assigned to the HX Mgmt VLAN (hx_inband_mgmt) is used for management traffic among ESXi, HX, and VMware vCenter; must be routable

    • If using more than one DNS Server (maximum two), separate with commas
  • Always use sequential numbers for IP addresses and names – allow system to automatically generate these from the first IP Address/Name – so make sure your first hostname ends in 01

Configuration Item




Subnet Mask



ESXi Server1 IP address
(Recommended a.b.c.11)

ESXi Server1 Hostname**
(xxx -01)

** Verify DNS forward and reverse records are created for each host defined here. If no DNS records exist, hosts are added to vCenter by IP address instead of FQDN.

Cluster IP Configuration

The Hypervisor IP addresses and Storage Controller IP addresses defined for the HX Mgmt VLAN (hx_inband_mgmt) must be in the same subnet as the Host IP addresses given in the previous step.

The Hypervisor IP addresses and Storage Controller IP addresses defined here for the Data VLAN (hx‑storage‑data) must be in a different subnet to the addresses assigned to the HX Mgmt VLAN (hx_inband_mgmt). This subnet does not really need to be routable (so gateway IP is optional), although that may change when synchronous replication is supported.

·       Always use sequential numbers for IP addresses – allow system to automatically generate these from the first IP Address.

Configuration Item




Management VLAN – make both IPs on same subnet

Hypervisor 1 IP Address (Already = ESXi Server1 IP address in last step)

Can’t change

Cant’ change

Can’t change

Storage Controller 1 IP address
(Recommended a.b.c.71)

Management Cluster IP address
(Recommended a.b.c.10)

Management Cluster Gateway

Data VLAN – make both IPs on same subnet

Hypervisor 1 IP Address
(Recommended d.e.f.11)

Storage Controller 1 IP address
(Recommended d.e.f.71)

Data Cluster IP address
(Recommended d.e.f.10)

Data Cluster Gateway

Storage Cluster and vCenter Configuration

·       The Cluster Name is the name given to the Storage Cluster. Use a naming convention like HXCLUS01. It will also be used as the vCenter Cluster Name by default too. I recommend keeping it this way unless you already have a vCenter Cluster you wish to use.

·       The controller VM is the VM that manages the storage cluster – and it must be 10 characters long (at least). It also requires the Upper/Lower/Digit/Special Character combo too, but should NOT contain special characters like <>[]{}/\’`”*

·       Cisco recommends using Replication factor 3 for Hybrid Clusters and either 2 or 3 for All Flash Clusters. RedNectar recommends FR2 for All Flash Clusters unless a heightened level of redundancy is desired.

·       If the vCenter Datacenter and/or Cluster (case sensate) exists already, it will be used. Otherwise it/they will get created during install.

·       If multiple DNS and/or NTP servers are used, use commas to separate the list. Typically, these will be consistent throughout the install, but I’ve seen some sites that use different addresses in different places.

Configuration Item




Cluster Name (E.g. HXCLUS01)


Replication Factor (2 or 3)


Controller VM password (required) (10+ chars Upper/lower/digit/special mix)

vCenter Datacenter Name
(E.g. HX-DC)

vCenter Cluster Name
[Default=cluster name above]

DNS Server(s) (Maximum two, comma separated)

NTP Server(s) (Maximum two, comma separated)

Time Zone:
(E.g. AEST +10)

Connected Services

  • Cisco recommends that connected services be enabled. If you wish connected services to be configured, enter the detail below. Enable Connected Services in order to enable management via Cisco Intersight.

Configuration Item

Global value

Enable Connected Services?

Yes No

Email for service request notifications

vCenter Single Sign-On (SSO)

If using SSO, the SSO URI may be required (assumed to be the same for all sites). This information is required only if the SSO URI is not reachable. The SSO Server URL can be found in vCenter at vCenter Server > Manage > Advanced Settings, key config.vpxd.sso.sts.uri

Configuration Item

Global value

SSO Server URI

Stretched Cluster Witness

  • If you are installing a Stretched Cluster, a Witness VM is required, preferably at a third location (RTT to either site 200ms).

Configuration Item



IP address of Witness VM

Task 6:               Post installation task

Once the Hyperflex cluster have been installed, you will need to run a post-install script where additional features can be enabled, including adding additional VM Networks. Your vSphere licence must support any features asked for.

    • Enabling HA/DRS on cluster enables vSphere High Availability (HA) feature per best practice.
    • Disabling SSH warning suppresses the SSH and shell warnings in the vCenter. SSH must remain enabled for proper functioning of the HyperFlex system.
  • Configure ESXi logging onto HX datastore allows the creation of a HX Datastore to hold the ESXi log files.

Adding vMotion interfaces configures vMotion interfaces per best practice. Requires IP addresses and VLAN ID input. Refer to the design diagrams.

    • You can add additional guest VLANs to Cisco UCS Manager and within ESXi on all cluster hosts during this process.
    • Enabling NTP on ESXi hosts configures and enables NTP on ESXi hosts.
  • If SMTP mail server and Auto Support parameters are configured, a test email is sent to ensure that the SMTP relay is working.

Configuration Item




If configuring a Stretched Cluster, Site#1 must = Site#2 except vmk interface IP (but use same subnet)

Enable HA/DRS on cluster?

Yes No

Yes No

Yes No

Disable SSH warning?

Yes No

Yes No

Yes No

Configure ESXi logging onto HX datastore?

Yes No

Yes No

Yes No

Datastore Name for HX ESXi logging?
(E.g. HX-ESXiLogs)




Add vMotion interfaces?

Yes No

Yes No

Yes No

ESXi Server 1 vMotion vmk interface IP
(Recommended x.y.z.11)




VLAN ID for vMotion
(As recorded above)




Add VM network VLANs? (Record VLAN names in table below)

Yes No

Yes No

Yes No

Enable NTP on ESXi hosts

Yes No

Yes No

Yes No

Send test email?

Yes No

Yes No

Yes No

VM Networks List

By adding additional VLANs at this point, the VLAN Names & IDs you define here will be created by UCS Manager on both Fabric Interconnects.  Additionally, the VLAN Names & IDs specified here will be added to the vSwitch called vswitch-hx-vm-network that is created on every ESXi host. (Actually, the VLAN Name on the vSwitch will be as specified here with the VLAN ID appended to the name)

Configuration Item




If configuring a Stretched Cluster, recommend Site#1 = Site#2

VM Network (=VLAN name)




VM Network ID (=VLAN ID)




VM Network (=VLAN name)




VM Network ID (=VLAN ID)




VM Network (=VLAN name)




VM Network ID (=VLAN ID)




VM Network (=VLAN name)




VM Network ID (=VLAN ID)




That completes my Installation Checklist. But it is not enough to have just a checklist of items without validating them. So, …

Before beginning Hyperflex installation…

After the logical Planning for the installation has been completed, you need to validate it.

Here is a checklist of a few things that you should make sure are completed before arriving on each site for the install. Having these items complete will greatly help make the Hyperflex installation go smoothly. If doing the install for a customer, have them complete this as well as the pre-installation above.



Task 1:               The Physical environment

a.     Do you have the correct power chords for all devices that have to be racked?

b.     Do you have the 10G/40G uplinks cabled to the rack where the Hyperflex Clusters is to be installed?

c.     Are the 10G/40G uplinks physically connected to the upstream switches?

d.     If bringing FC to the FIs, do you have the FC fibre uplinks physically connected from the FIs to the upstream FC switches?

e.     Do you have 2 x RJ45 connections to the OOB Management switch that the Fabric Interconnects will connect to?

The two FI’s have ports labelled L1 and L2. Two regular RJ45 Ethernet cables are needed to connect L1 to L1 and L2 to L2. Ideally, these will be ~20cm in length to keep patching obvious and neat.

f.      Do you have the L1 & L2 ports for both FIs in all locations connected via 2 x regular RJ45 Ethernet cables?

HX Clusters use single-cable management so that all management functions are carried out from UCS Manager.


g.      Have you ensured that no cables are attached to the on-board 1Gbps and CIMC interfaces of the nodes?

Task 2:               The Inter-Fabric Network

h.     Are the four VLANs defined in the Pre-Install Checklist configured on the upstream switches that the FIs will be connecting to?
(Refer to the design diagrams above)

i.     Have jumbo frames been enabled on the upstream switches that the FIs will be connecting to? (Refer to the design diagrams above)

Task 3:               The Management Switch/VLAN

The FI’s need 1G Ethernet connectivity to a management switch/VLAN.

j.      Have the IP addresses defined as default gateway addresses in the Pre-Install Checklist been configured on a router/Layer3 switch?

Plug a laptop into the ports assigned to the FI Management ports in the racks where the FIs are to be installed (i.e. as in e above). Assign your laptop an IP address in the appropriate range

k.      Can the laptop ping the default gateway IP?

l.      Can the laptop ping the NTP server defined in the Pre-Install Checklist?

Task 4:               The Installer

The installer is a .ova file (Cisco-HX-Data-Platform-Installer-vxxxxx.ova) – a vCenter (v6.5) needs to be set up with an ESXi Host and the .ova installed on the ESXi Host.

Note:            If all else fails, you can run the installer from a laptop using VMware Fusion or VMware Workstation.

When the installer VM boots, it needs to obtain an IP address and DNS information via DHCP or be allocated the IP details defined above.

The following tests are to verify that the Installer VM has been given a suitable IP address, has access to DNS, and that the DNS has been configured fully.

The installer VM username/password is root/Cisco123.

m.     Has the Installer VM been given an IP address via DHCP?
(Use the command ip addr show eth0 or ifconfig eth0 to check)

n.      Has the Installer VM has been configured with the correct DNS address?
(Use the command cat etc/resolv.conf to check)

o.   Can the Installer VM resolve forward and reverse names using the following commands?
nslookup <insert IP of first HX-ESXi host>
<repeat for all HX-ESXi hosts in the cluster>
nslookup WhateverDNSNameYouUseForESXiHost-01
<repeat for all HX-ESXi hostnames in the cluster>

p.     Can the Installer VM ping the NTP server?
ping <insert IP NTP Server>

q.     (Stretched Cluster) Can the Installer VM ping the NTP server Witness VM?
ping <insert Witness IP>

Posted in Cisco, Hyperflex, UCS, VLANs | Tagged | 1 Comment