I had a Tenant with a statically configured bare metal hosts attached to interface 1/16 on both leaf 101 and leaf 102, but came up with a “invalid-path;invalid-vlan” error on the faults page for the EPG that was being configured for leaf 102. The host attached to Leaf 101 was working, had no errors and was configured in exactly the same way!

I checked:
In the tenant, the EPG had been linked to the correct Physical Domain
In the tenant, the EPG had been linked to the correct leaf/port/vlan

In the Fabric Policies;
The Leaf Profile defined the correct leaf, and was linked to the correct Interface Profile
The Interface Profile had an Access Port Selector for to the correct port (1/16), and the Access Port Selector was linked to an Access Port Policy Group
The Access Port Policy Group was linked to the correct Attachable Access Entity Profile
The Attachable Access Entity Profile was linked to the same Physical Domain as the EPG showing the error
The Physical Domain was linked to a VLAN Pool that included the VLAN ID being used in the EPG for the static mapping.

So I was stumped.  I have never seen an  “invalid-path;invalid-vlan” error before that could be solved by checking the above, so in desperation I checked thing from the CLI:

apic1# show run leaf 102 interface ethernet 1/16
# Command: show running-config leaf 102 interface ethernet 1/16
# Time: Wed Mar  1 04:17:41 2017
  leaf 102
    interface ethernet 1/16
      # Policy-group configured from leaf-profile ['T5:L102-LeafProf'], leaf-interface-profile T5:L102-IntProf
      # policy-group T5:1G.CDP.LLDP-APPG
      lldp receive
      lldp transmit
      cdp enable
      vlan-domain member T5:MappedVLANs-PhysDom type phys
      switchport access vlan 2050 tenant Tenant5 application 2Tier-AP epg WebServers-EPG
      speed 1G
      negotiate auto
      link debounce time 100
      exit
    exit

“That looks a bit strange”, I thought.  “I don’t normally see the lldp and cdp policies etc”.  But there was nothing in the config that was wrong, none the less, I thought I’d compare with the same port on the other leaf.

apic1# show run leaf 101 interface ethernet 1/16
# Command: show running-config leaf 101 interface ethernet 1/16
# Time: Wed Mar  1 04:18:02 2017
  leaf 101
    interface ethernet 1/16
      # Policy-group configured from leaf-profile ['T5:L101-LeafProf'], leaf-interface-profile T5:L101-IntProf
      # policy-group T5:1G.CDP.LLDP-APPG
      switchport access vlan 2050 tenant Tenant5 application 2Tier-AP epg AppServer-EPG
      exit
    exit

Now this looks much like I expect. And at this stage, this is the only indication that the configuration on 102/1/16 is not quite “normal”. So what I tried next was to see if I could remove the “extra” lines of config on leaf 101. Since there is no default interface command in ACI-NX-OS, I tried manually removing the cdp, lldp etc config:

apic1(config)# leaf 102
apic1(config-leaf)# default ?
apic1(config-leaf)# default inter
Command entered is not APIC NX-OS style CLI.Trying shell command…

apic1(config-leaf)# interface ethernet 1/16
apic1(config-leaf-if)# shut
apic1(config-leaf-if)# no lldp receive
apic1(config-leaf-if)# no lldp transmit
apic1(config-leaf-if)# no cdp  enable
apic1(config-leaf-if)# no vlan-domain member T5:MappedVLANs-PhysDom type phys
apic1(config-leaf-if)# no speed 1G
apic1(config-leaf-if)# no negotiate auto
apic1(config-leaf-if)# no link debounce time 100
apic1(config-leaf-if)# no shutdown

Better see if that worked!

apic1(config-leaf-if)# show run leaf 102 interface ethernet 1/16
# Command: show running-config leaf 102 interface ethernet 1/16
# Time: Wed Mar  1 04:28:16 2017
  leaf 102
    interface ethernet 1/16
      no lldp receive
      no lldp transmit
      no cdp enable
      speed auto
      no negotiate auto
      link debounce time 100
      exit
    exit

Clearly that didn’t work as intended. And by now I’d removed the interface selector for interface 1/16 from the interface profile for Leaf 102 as well, so there should have been no association with any lldp, cdp etc – except for one thing – I’d forgotten that when you do anything in the CLI, it automatically starts creating pesky objects with names beginning with __ui and I could see these in the GUI – but I knew how to get rid of those thanks to this post.

Note:

Unless Daniel has updated his blog, you will see that one command I used was different to the one in the link above: Daniel’s blog says to use a moconfig delete command, when in fact it should be moconfig commit

And that’s what I did!

apic1# cd /mit/uni/infra
apic1# for i in `find *__ui*`
for> do
for> echo "removing $i"
for> modelete $i
for> done
removing attentp-__ui_l102_eth1--16
This command is being deprecated on APIC controller, please use NXOS-style equivalent command
removing attentp-__ui_l102_eth1--16/mo
This command is being deprecated on APIC controller, please use NXOS-style equivalent command
.......

apic1# moconfig commit
This command is being deprecated on APIC controller, please use NXOS-style equivalent command
Committing mo 'uni/infra/lldpIfP-__ui_l102_eth1--16'
Committing mo 'uni/infra/hintfpol-__ui_l102_eth1--16'
Committing mo 'uni/infra/cdpIfP-__ui_l102_eth1--16'
Committing mo 'uni/infra/attentp-__ui_l102_eth1--16'

All mos committed successfully.

Re-checking the CLI config showed:

apic1(config-leaf-if)# show run leaf 102 interface ethernet 1/16
# Command: show running-config leaf 102 interface ethernet 1/16
# Time: Wed Mar 1 04:42:46 2017
 leaf 102
 interface ethernet 1/16
 exit
 exit

Brilliant! It certainly looks “correct” although I have no idea why this config should be any more “correct” than what I saw earlier, except…
… when I re-created the Interface Selector for access port 1/16 on leaf 102, and reassigned the same interface to the EPG in the tenant config (in other words restored the previous config) – all errors disappeared and the config worked!

Now it could have been that running the script to remove the pesky __ui items actually removed some other junk that was causing a problem, but whatever caused the error is a mystery!

One of the strangest mysteries I have encountered in the ACI lab so far.

RedNectar