When I was challenged with this:

Hi @RedNectar ,

Right now I have a simple contract that allows SSH only:

• Scope set to global.
• TCP dst 22.
• “Both directions” and “reverse port filters” enabled.

This contract is provided by the inband EPG at the “mgmt” Tenant and exported to tenant B. EPG at Tenant B consumes the contract interface. Can’t SSH the APICs or switches from a VM in Tenant B. Am I missing something?

I realised my earlier post didn’t cover this scenario where the management workstation was in another Tenant. So here’s the update.

Much like my earlier post, you will have to create an Access Policy Chain to associate a VLAN ID with the interfaces the APICs attach to.  For my example I used VLAN 2002 and my APIC is attached to Leaf 102 & 102 on port 47.

Begin with the Access Policy Chain

Here’s how I built my access policy chain in pictures. Note that I already had Leaf Profiles and Interface Profiles built for leaf 101 & 102;

I was going to need a VLAN pool to specify my chosen VLAN – VLAN 2002, so I created one:

Fabric > Access Policies > Pools > VLAN >+ Create VLAN Pool

Name: mgmt:inb_VLAN.Pool
Allocation Mode: Static Allocation
(+) Encap Blocks:
Range: VLAN 2002 – VLAN 2002

The VLAN Pool needed a Physical Domain, so again…

Fabric > Access Policies > Physical and External Domains >+ Create Physical Domain

Name: mgmt:inb_PhysDom
Vlan Pool: mgmt:inb_VLAN.Pool

which needed an Attachable Access Entity Profile of course…

Fabric > Access Policies > Policies > Global > Attachable Access Entity Profiles >+ Create Attachable Access Entity Profile 

Name: mgmt:inb_AAEP
(+) Domains (VMM…Exernal) To Be Associated To Interfaces:
Domain Profile: mgmt:inb_PhysDom

I had to make sure my APICs had LLDP enabled when I made the Access Port Policy Group to link to the AAEP – I already have a suitable policy that would do that as you can see here when I created the Leaf Access Port Policy Group:

Fabric > Access Policies > Interfaces > Leaf Interfaces > Leaf Access Port >+ Create Leaf Access Port Policy Group 

Name: mgmt:APIC_APPG
LLDP Policy:  Enable_LLDP
Attached Entity Profile:  mgmt:inb_AAEP

And to finish the chain, I created interface selectors for each APIC in my existing Interface Profiles.

So that’s the Access Policy Chain done for the APICs

Configuring the mgmt teanant for inband management

Now to set up inabnd Management in mgmt tenant. Apart from the route leaking trick below, most of this is just following my earlier post where I did a more complete description. I began by adding the inband management IP to the pre-defined inb Bridge Domain.

Then created the In-Band Management EPG, which is a special EPG.

Tenants > mgmt> Node Management EPGs >+ Create In-Band Management EPG 

Name: inb_EPG
Encap:  vlan-2002
Bridge Domain:  inb

This raised an annoying problem, described in bug CSCuz59329

So I fixed it using the work around described in the bug report. In other words I created a Management Node Connectivity Group. In the process, I calso created an IP address pool for the group. I’m not sure if I really needed to create an address pool, but I did anyway.

Tenants > mgmt> Managed Node Connectivity Groups >+ Create Managed Node Connectivity Group

I then created Static Node Management Addresses. first for the APICs, which trhew up a warning that had me check (and change) the default preference for management back to oob.  And then added more static addresses for Leaves and Spines, but it’s really the APICs that matter.

Tenants > mgmt> Static Node Management Addresses >+ Create Static Node Management Addresses

Then the warning…

so I fixed that!

System > System Settings > APIC Connectivity Preference

And checked that my IP had stuck using the ifconfig bond0.2002 command (recall I allocated VLAN 2002 to inb managment)

And did a ping test to the default gateway IP to be sure:

So at last, my inband management PEG was set up. It was time to test the challenge given me, which said in part:

Right now I have a simple contract that allows SSH only:

• Scope set to global.
• TCP dst 22.
• “Both directions” and “reverse port filters” enabled.

Filters and Contracts

I already had an SSH filter in the common tenant, so I created a Contract there too ready to do the test.

[Note: I later decided that the Contract would be better created in the mgmt tenant, because haveing the contract in the common tenant will allow ALL Tenants access to the inband management IP network]

Tenants > common> Contracts > Standard >+ Create Contract -> is what I did

Tenants > mgmt> Contracts > Standard >+ Create Contract -> is what I should have done

I configured the inband management EPG to Provide this contract.

But now I was stuck – I needed a tenant to consume the contract. So back to the question:

This contract is provided by the inband EPG at the “mgmt” Tenant and exported to tenant B. EPG at Tenant B consumes the contract interface.

Creating a test tenant

So I created a Tenant, and of course called it TenantB

TenantB needed a Bridge Domain and an EPG, so I created those too, making sure that I checked the Shared Beteen VRFs option for the Bridge Domain when I created the subnet for the BD.  I also created the Application Profile on the way.  I already had a host connected on the 192.168.80.0/24 subnet attached on interafce 102/1/26, so I added that host to the EPG in the process, and made sure it consumed the common/SSH.Global_Ct

Tenants > TenantB > Networking > Bridge Domains >+ Create Bridge Domain

Tenants > TenantB > Application Profiles >+ Create Application Profile

Fantasic. So now I had completed everything, but I had one little worry that I wanted to check.

Route Leaking conundrum

My worry was about route leaking.  You see, the Consumer EPG is in a different tenant and different VRF to the Provider EPG, so to make route leaking work I must do these two things:

1. Enable the Shared Between VRFs on the Bridge Domain or EPG Subnet of the Consumer EPG (which I had done )
2. Enable the Shared Between VRFs on the EPG Subnet of the Provider EPG which is the special mgmt tenant’s Node Mangement EPG for In-Band.

So I went looking to how I could add an EPG Subnet to the mgmt tenant’s Node Mangement EPG for In-Band. I found an option to add a subnet, so I did that, but NOWHERE was I able to click any Shared Between VRFs option.

I thought I’d check leaf 102 to see if any routes had leaked between the VRFs, and as expected, TenantB’s 192.168.80.0/24 route had leaked into the mgmt tenant, but without the ability to make the 192.168.99.0/24 subnet shared between VRFs on the EPG, TenantB’s VRF had no knowledge of the inband management subnet.

Here’s the kludge I used to fix it

I knew that what I needed to do was somehow to get the inband management IP subnet into the routing table for TenantB.  And I knew that to do that, I needed to either:

1. Add an EPG Subnet with the shared between VRFs option set on the EPG,
2. make the mgmt tenant become the consumer of a contract that was provided by TenantB’s EPG.  I figured this would work because I was at least able to check the shared between VRFs option on the inband management DB.

I tried option 1 first, and created an Application Profile and EPG in the mgmt tenant, added the 192.168.99.0/24 subnet and checked the shared between VRFs option, and had it also provide the common/SSH.Global_Ct contract.

And sure enough, the routing table was happy.

All that was left to do was to test the validity of the contract form TenantB’s host:

EUREKA! So they say.

But…

I didn’t like the solution.

Because what I had created was a contract in the common tenant that was provided by the inband management tenant, and could therefor be consumed BY ANY TENANT. In other words, I had allowed open access to the management network to any EPG in any tenant that cared to consume the common tenant’s SSH.Global_Ct contract. I’m sure any worthwhile security manager would have something to say about that.

To mitigate this, I considered option 2 above. Make the mgmt tenant become the consumer of a contract that was provided by TenantB’s EPG.  I tried this for fun using the same SSH.Global_Ct contract, and it worked, but didn’t mitigate the problem. Any EPG that wanted to consume the same contract would have access to the inband management subnet.  And I could see that while ever I was using a contract in the common tenant, I wassn’t going to win.

So I had to move the contract from the common tenant to the mgmt tenant, which also meant that I had to export the contract to TenantB, and then in TenantB, consume the contract as a Consumed Contract Interface. I still faced the route leaking problem, and still had to create the Application Profile + EPG + Subnet with the Shared Between VRFs option to make it work, but at least I ended up with something that I was a little happier with.

So, there you have it. That’s how you can configure inband management so a tenant can access ACI management.

RedNectar