Change from US to UK date format in the vSphere Client — Define The Data Centre

If you’ve found yourself double guessing and still not being sure what date you’re actually looking at in the vSphere client then this post is for you. We’re going to change the US date format MM/DD/YYY to DD/MM/YYYY in Chrome but it also works for Firefox and Edge too. The default setting in when logged […]

Change from US to UK date format in the vSphere Client — Define The Data Centre
Posted in GNS3 WorkBench | Leave a comment

HyperFlex’s new snapshot mechanism matters. Here’s why, and what Cisco forgot.

It all become possible when in ESXi v7.0U2, VMware introduced a new parameter for VMs called snapshot.alwaysAllowNative. Then, with HyperFlex v4.5(2), Cisco took advantage of this new parameter to remove the biggest bugbear of all HyperFlex installations. The SENTINEL snapshot.

The short story

Before HyperFlex Data Platform (HXDP) v4.5(2), HyperFlex used VMware APIs to create an initial snapshot in native format because this format is much more efficient when combined with HyperFlex’s pointer-based log structured file system.  This snapshot was always given the special name SENTINEL, but it took some time to create and always consumed a little extra space if it was not deleted later. And if it was deleted, there was always a chance that a VMware REDO snapshot would be taken, and the efficiencies of the HyperFlex pointer-based log structured file system would be compromised.

Now, with with HyperFlex v4.5(2), coupled with ESXi v7.0U2 the HXDP only needs to set a parameter on the VM – the snapshot.alwaysAllowNative parameter. Much more efficient and far less prone to error.  It is much faster now for backup software to take snapshots and there is no potential residual space wasted.

But the old problem of a VMware REDO snapshot being taken still exists, and unfortunately, when Cisco adapted this new approach, they dropped the ball when it came to management options in the HyperFlex Connect management app, the VMware HyperFlex plugin, the HyperFlex CLI and on the Intersight SAS platform.  None of these management systems has any ability to allow users to see which VMs have the snapshot.alwaysAllowNative parameter set, and none of them has any ability to set the parameter.

I’m calling on Cisco to add these options ASAP.  Previously it was relatively simple to see if a VM had a SENTINEL snapshot – you just needed to look at the Snapshots tab for a VM. Now you have to navigate at least six mouse-clicks to check for the snapshot.alwaysAllowNative parameter in a long list of other parameters.

In the meantime, I’ve created a bunch of PowerCLI scripts you can use to:

  • List all the VMs that have the snapshot.alwaysAllowNative parameter set to TRUE
  • List all the VMs that have the snapshot.alwaysAllowNative parameter set to FALSE
  • List all the VMs that do not have the snapshot.alwaysAllowNative parameter at all, with an option to
    • Set the snapshot.alwaysAllowNative parameter to TRUE on these VMs

These scripts are found at the end of this article.

The full story. Let’s start with: Why does it matter?

Before I can explain WHY it matters, you need to understand a little more about VMware snapshots, and in particular, the Native snapshot, and how it is different to the regular REDO snapshot.

First of all, snapshots matter because all backup and replication software do their work by first taking a snapshot of any running VM – it’s pretty obvious that you can’t backup a running VM while it is potentially writing to disk. The solution is to take a snapshot, copy the snapshot then delete it. Simple.

Secondly, you need to understand that VMware has not always created snapshots the same way.  Today, VMware snapshots are based on a collection of deltas from an initial base, known as REDO files. But in the distant past a snapshot was simply a copy of the .vmdk file that was the VM.  Hence the name the native snapshot. This snapshot of course doubled the amount of disk space required for the VM in the VMware NFS file system, but it turns out that this is an ideal format for snapshot files using HyperFlex’s log-structure pointer based file system which, instead of making a copy of a VM when a native snapshot is taken, makes a copy of the pointers instead. No additional disk space needed! Very cool. Very efficient. And very much suited to a Hyper-Converged Infrastructure (HCI).

But there’s a problem.

The writers of the HyperFlex Data Platform (HXDP) had to come up with a way of forcing VMs to create snapshots in the original native format rather than the normal REDO format.  And the way they did this originally was when a snapshot was created via the HXDP rather than using the normal VMware methods, HyperFlex made use of APIs to create the first snapshot in native format and gave it a special name – SENTINEL.

Once a VM had one snapshot in the original native format, VMware would create any future snapshots in native format as well, for compatibility.

Now the problem is that if a VM has had a normal VMware REDO snapshot taken before the SENTINEL had been created, or after the SENTINEL had been removed, the HXDP can’t take a native format snapshot.  And that problem still exists today, even with the new snapshot.alwaysAllowNative parameter.

Missing in Action – Cisco Management Tools

It has been common practice for HyperFlex users to create an initial SENTINEL HyperFlex snapshot as soon as they deploy a VM to ensure that their integrated backup software would be able to make use of the more efficient pointer-based log-structured files system when backing up VMs.

But there was a disadvantage to this approach – over time, the SENTINEL would contain data that belonged to the originally deployed VM that was now out of date, which was fine should you ever wish to revert to the original state, but if you were getting short on space, it meant that at least a small amount of space was been reserved for an unlikely event – remembering you’d likely have a backup of the original file should you need it.

The advantage though was that you could easily check to see if a VM had a SENTINEL snapshot, you just had to click on the Snapshot tab for the VM

But with the new more efficient snapshot.alwaysAllowNative parameter, checking if the parameter is set is much harder.

 

That’s 8 clicks and two scrolls by the time you are done! And the visual challenge of finding the parameter in that long list is just not easy.

Why didn’t Cisco add a Native Snapshots column to the VM list in HyperFlex Connect, or perhaps better still a symbol like a * in the Snapshot column to indicate that the VM had been configured for native snapshots?

And why is there no Action option to set a VM (or group of VMs) snapshot.alwaysAllowNative parameter to TRUE?

And I’d expect these options to be also available in Intersight, on the right-click menu in vCenter (via the plugin) and even would be nice to have some vm options in the HyperFlex Connect Web CLI – such as stcli vm list (which I’d expect to list the snapshot.alwaysAllowNative parameter among other useful information) and stcli vm [(--id ID | --name NAME)] set snapshot.alwaysAllowNative

Now I’m hopeful that Cisco cares enough about their User Interface to actually repair this oversight. And I was prepared to forgive the initial release. But the urge to write this article came when I recently upgraded our lab cluster to HXDP v5.0.  I really expected these features would have been attended to. But I was sadly disappointed.

So while waiting for Cisco to actually fix this faux pas, I’ve written some PowerShell CLI commands that will help you in the meantime. Feel free to cut and paste from below.

RedNectar

The PowerCLI Scripts

To use PowerCLI scripts you need to have installed PowerShell Core for your OS (Windows usually comes with PowerShell installed) then from the Powershell CLI, install the VMware PowerCLI Powershell modules like this:

Install-Module -Name "VMware.PowerCLI" -Scope "CurrentUser"

Next, you connect to your HXDP vCenter – but if you don’t have valid certificates on your vCenter, fist do this

Set-PowerCLIConfiguration -InvalidCertificateAction:Ignore

And when you connect to vCenter, it should look something like this:


PS /Users/rednectar> Connect-VIServer -Server vca.your.domain.dns
Specify Credential
Please specify server credential
User: admin@your.domain.dns
Password for user admin@your.domain.dns: *********
Name                Port User
----                ---- ----
vca.your.domain.dns 443  YOUR.DOMAIN.DNS\admin

And now you can issue the following command to perform the following functions:

  • List all the VMs that have the snapshot.alwaysAllowNative parameter set to TRUE

Get-VM  |
 where {$_.Name -notlike "vCLS*"} |
 where {($_.ExtensionData.Config.ExtraConfig |
 where {$_.Key -match "snapshot.alwaysAllowNative"} |
 where {$_.value -eq $true})} |
 select @{N="VMs using Native Snapshots";E={$_.Name}}

  • List all the VMs that have the snapshot.alwaysAllowNative parameter set to FALSE

Get-VM  |
 where {$_.Name -notlike "vCLS*"} |
 where {($_.ExtensionData.Config.ExtraConfig |
 where {$_.Key -match "snapshot.alwaysAllowNative"} |
 where {$_.value -eq $false})} |
 select @{N="VMs set to NOT use Native Snapshots";E={$_.Name}}

  • List all the VMs that do not have the snapshot.alwaysAllowNative parameter at all

Get-VM |
 where {$_.Name -notlike "vCLS*"} |
 where {!($_.ExtensionData.Config.ExtraConfig |
 where {$_.Key -match "snapshot.alwaysAllowNative"})} |
 select @{N="VMs NOT configured at all for Native Snapshots";E={$_.Name}}

Set the snapshot.alwaysAllowNative parameter to TRUE on these VMs


Get-VM -PipelineVariable vmname |
where {$_.Name -notlike "vCLS*"} |
where {!($_.ExtensionData.Config.ExtraConfig |
where {$_.Key -match "snapshot.alwaysAllowNative"})} |
New-AdvancedSetting -Name snapshot.alwaysAllowNative -value TRUE -Confirm:$false -force |
select @{N="VMs Converted to use Native Snapshots";E={$vmname.Name }}

Posted in Cisco, Hyperflex | Tagged , | Comments Off on HyperFlex’s new snapshot mechanism matters. Here’s why, and what Cisco forgot.

Finding text within a paragraph or word in MS Word

When using wildcards to search for text between two specific markers, MS Word will find those markers and that text no matter how far apart they are.

If you want to restrict the search to finding that text/markers within a word or a paragraph, you need to be a bit clever about how you search, and think in the negative.

Find within a word

In my case, I wanted to find any place where I had used two carat characters (^) in the same unbroken string of characters, and prepend the first of those characters with a backslash. Turns out the replacement was not so easy either.

FWIW – this was the context I in which I was searching. I’ve highlighted the ^ characters for you.

fabric 1101,1201-2202 show lldp neighbors | egrep "Node|-|apic|^Spine|^Leaf"

Spoiler: the result looked like this – which will find text within any word that has two carat characters (^)

So let’s unpack this. The find pattern above is

([!^ ]@^^)([!^ ]@^^)

Firstly, I’ll remove the parentheses – they come into play for the replacement part.  That leaves us with

[!^ ]@^^[!^ ]@^^

The [! ] sequence says NOT a space character.  The @ symbol says “Any number of … non-space characters

Because the carat character is considered a special character in word search and replace, ^^ is actually searching for a single ^ character

So the whole sequence reads:

Find… any number of non-space characters followed by a ^ followed by any number of non-space characters followed by a ^

That’s the “find” part sorted. But why the parentheses?

The thing is, I want to insert a backslash before the first ^ character, so I have to group by search with () – the first group being all the characters up to, but not including the ^ character. ([! ]@) and the second group being the rest. (^^[! ]@^^)

Which gives us the find pattern as shown above. ([! ]@)(^^[! ]@^^)

The next problem is the replacement part.  For that I use the special MS Word tags \1 and \2 which correspond to the first and second search groups respectively.  But that causes another problem – I actually want to USE a backslash character in my replacement string – and of course it too is a special character. So you’d think, using the logic that if ^^ finds a single ^, then ^\ (or even \) would do in the replacement section to insert a backslash.

But also no. Instead, you need to flash back to when you memorised the set of ASCII characters and remember that the \ character is character number 92

So now I can specify that my replacement string is

\1^92\2
Nerd tip: If you want to find repeating patterns in MS Word, you can use the \1 construction in the search for box to, so searching for ( the)\1 would find all occurrences of the word sequence ” the the”

OK. So that will work fine so long as my two carat characters don’t have any spaces between them.  But what about if my source text was a little less concise, with spaces between the target carat characters like:

fabric 1101,1201-2202 show lldp neighbors | egrep "Node|-|apic|^Spine| ^Leaf"

Find within a paragraph

The logic is exactly the same, except this time I need to search for “anything that is NOT a paragraph marker.

MS Word uses the special combination of ^p to mean “paragraph marker” – but it is NOT available when using wildcards!  Instead, I have to go back to that wonderful ASCII set and remember that a CR (carriage return) character is number 13.

So now I can search for “any number of non-CR characters followed by a ^ followed by any number of non-CR characters followed by a ^” using the following sequence.

[!^13]@^^[!^13]@^^

RedNectar

You don’t need to read why I was doing this.

I’m in the process of converting some documents to asccidoc.

In asciidoc, the ^ characters are used to delimit superscript, so my original line of

fabric 1101,1201-2202 show lldp neighbors | egrep "Node|-|apic|^Spine|^Leaf"

is rendered in asciidoc format as

fabric 1101,1201-2202 show lldp neighbors | egrep "Node|-|apic|Spine|Leaf"

I needed to “escape” the leading ^ character for it to render correctly.

I this site a great help when figuring this out https://wordmvp.com/FAQs/General/UsingWildcards.htm

RN

Posted in Microsoft, Microsoft Word, MS Word Tips | Tagged , , , | Comments Off on Finding text within a paragraph or word in MS Word

Are Interface Descriptions in ACI using magic?

When you add a description to an interface in ACI, the eye of Sauron blinks and unexpected things happen. But is this magic?

I fell into this rabbit hole (or should I say hobbit hold?) when answering this question on the Cisco Community forum, and now I’ll give you the short(er) story.

In a nutshell

There are three ways to put a Description on and Interface in ACI

  1. You enter directly it under the Fabric > Inventory path
  2. You create an Interface Override for the interface and add it there – it automagically appears on the Physical Interface description under the Fabric > Inventory path
  3. You add a description to the Access Port Block for the Interface Selector for the relevant Interface Profile, where again it mysteriously appears in the Physical Interface description under the Fabric > Inventory path

It turns out

  • 1. and 2. are closely related. Change one, and you change the other.
    • If you delete the override object, the description on the interface is deleted also
    • If you EDIT the override object, the description on the interface is edited also
    • If you create your own override object with a Description, the description also appears on the interface
  • IF a Description is assigned to an Access Port Block for the Interface Selector for the relevant Interface Profile 
    • AND there is no relevant Interface Override for that port
    • AND the Interface Selector is linked to an Interface (Access Port) Policy Group
  • THEN the description is copied to the Physical Interface description

If you want more detail, read on.

To Begin…

If a user adds a description to an Interface when viewing it from Fabric > Inventory >> Podn > Leafn > Interfaces > Physical Interfaces > eth1/n then it appears as you view that screen, as you would expect.

But something else unexpected happens too. A NEW object is created and buried under  Fabric > Access Policies >> Interfaces > Leaf Interfaces > Overrides > nnn_eth_1_n

Wow. What’s going on here?

The crux of the matter

The Overrides configuration is really just the beginning of the weird background stirrings that go on, but to get to the root of the problem, you need to look more closely at the original screen. In particular, the Distinguished Name (DN) of the interface, which in my case was 

topology/pod-1/node-1201/sys/phys-[eth1/2]

Right-clicking on the interface to view the interface in the object-store browser (visore) reveals that the object belongs to the l1PhyIf class…

…and if you examine the inbuilt documentation for that class, you’ll find…

…that objects of this type are NOT configurable!

So what happens behind the scenes is that when you edit this non-configurable object, ACI hides it away in a place that is configurable – namely (in my example) uni/infra/hpaths-1201_eth1_2/rsHPathAtt-[topology/pod-1/paths-1201/pathep-[eth1/2]] which, as you can guess is the DN of the Override I illustrated earlier. 

It turns out that:

  • If you delete the override object, the description on the interface is deleted also
  • If you EDIT the override object, the description on the interface is edited also
  • If you create your own override object with a Description, the description also appears on the interface

But this is only HALF the story. And here is where the real weird stuff happens.

The real weird stuff

If the interface has NO override, and therefore no description appearing on the Physical Interface, then a description added to the Access Port Block for the Interface Selector for the relevant Interface Profile can also appear as the description on the physical interface – BUT NOT SO FAST.

The description DOESN’T appear on the Physical Interface until an Interface Policy Group is assigned.  Once an Access Policy Group has been allocated to the interface, the description then magically appears on the Physical Interface Description.

Duel to the death!

Now if there are TWO ways of getting a description into the Physical Interface Description, which one wins?

The only way to find out is to set up a duel!  Perhaps not quite Éowyn vs Witch King proportions, but let’s find out.

LOTR Éowyn vs Witch King of Angmar

Image from Lord of the Rings: The Return of the King. GIF image from Streamerclips.com

Round 1:

  1. Create an Interface Description via the Access Port Block 
  2. Edit the description via the Physical Interface page under the Inventory

Result:

  1. A NEW Override object is created to hold the edited description
  2. The Interface Description under the Access Port Block remains unaltered

Round 2:

  1. Delete the new Override object created in Round 1,
  2. and observe the results

Result:

  1. The Physical Interface description under Inventory is deleted
  2. The Interface Description under the Access Port Block remains unaltered

Round 3:

  1. Disassociate the Interface (Access Port) Policy Group from the Interface Selector
  2. Re-associate the Interface (Access Port) Policy Group with the Interface Selector
  3. and observe the results

Result:

  1. The Physical Interface description under Inventory is updated with the Access Port Block description
  2. The Interface Description under the Access Port Block remains unaltered

It appears that the Override object is playing Éowyn’s part for this duel, while the Access Port Block represents the hapless king.

But the good new is that even if the duel is lost, the description in the Access Port Block remains even if the other description is changed or removed.

Or more simply:

  • IF a Description is assigned to an Access Port Block for the Interface Selector for the relevant Interface Profile 
    • AND there is no relevant Interface Override for that port
    • AND the Interface Selector is linked to an Interface (Access Port) Policy Group
  • THEN the description is copied to the Physical Interface description

Summary

There are three ways to put a Description on and Interface in ACI

  1. You enter directly it under the Fabric > Inventory path
  2. You create an Interface Override for the interface and add it there – it automagically appears on the Physical Interface description under the Fabric > Inventory path
  3. You add a description to the Access Port Block for the Interface Selector for the relevant Interface Profile, where again it mysteriously appears in the Physical Interface description under the Fabric > Inventory path

It turns out

  • 1. and 2. are closely related. Change one, and you change the other.
    • If you delete the override object, the description on the interface is deleted also
    • If you EDIT the override object, the description on the interface is edited also
    • If you create your own override object with a Description, the Description also appears on the interface
  • IF a Description is assigned to an Access Port Block for the Interface Selector for the relevant Interface Profile 
    • AND there is no relevant Interface Override for that port
    • AND the Interface Selector is linked to an Interface (Access Port) Policy Group
  • THEN the description is copied to the Physical Interface description

RedNectar

Posted in Access Policies, ACI, ACI API, Cisco | Tagged | Comments Off on Are Interface Descriptions in ACI using magic?

Hyperflex Post Install script fixer

UPDATE: HyperFlex v5.01b HAS FIXED THIS PROBLEM – Do not run this fixer if you are installing 5.01b or later. Somebody at Cisco must have read this blog!

I was shocked the other day to learn that the hx_post_install script that is used during the Cisco HyperFlex install process does NOT work the way it should.

In fact, the validation option is a complete waste of time (if working with M5 servers, which is probably 90% of installations), as I reported here.

To fix it, I could create a new copy of the script and give that to you, and you could copy that to your HyperFlex Storage Controller VM, but that’s a pain. Instead, I’ve decided to give yo a few commands to run that you can cut and paste into a command shell to fix the problem – or at least work around the problem until Cisco fixes it. To ease the pain, all you need do is cut-and-paste the following into your ssh session on the storage controller – IF you have blind faith in my skills. Otherwise, you might want to go through it step-by-step, so you understand it.

Non-nonsence cut-and-paste answer

Cut-and-paste the following into your ssh session on the storage controller. (Note: With  HX v4.5(2) and later, run this script from the Hyperflex Installer VM – it can’t be run from the storage controller VM because Cisco has restricted access to the root system on the SCVM in 4.5(2+)

cp $(which hx_post_install) .
sed -i 's/vmnic1/rednectar4/g' hx_post_install
sed -i 's/vmnic2/rednectar1/g' hx_post_install
sed -i 's/vmnic3/rednectar5/g' hx_post_install
sed -i 's/vmnic4/rednectar2/g' hx_post_install
sed -i 's/vmnic5/rednectar6/g' hx_post_install
sed -i 's/vmnic6/rednectar3/g' hx_post_install
sed -i 's/rednectar/vmnic/g' hx_post_install
sed -i 's/and args.validate//' hx_post_install
sed -i "s/Select post_install/***RedNectar's Updated hx_post_install script M5 modifications have been applied.***\\\nSelect post_install/" hx_post_install
sed -i 's/SCRIPT_VERSION = "4.0"/SCRIPT_VERSION = "4.1 RedNectar"/g' hx_post_install
echo "To run the modified script, type: ./hx_post_install --validate"

Full-blown answer

The first step after establishing an ssh session to a storage controller VM (or installer VM – esp v4.5(2+) ) is to locate the hx_post_install script

admin@hxscvm1:~$ which hx_post_install
/bin/hx_post_install

Using the result of the output above, copy the script to your admin home directory (where you land when you start your ssh session) and check that it exists.

admin@hxscvm1:~$ cp /bin/hx_post_install .
admin@hxscvm1:~$ ls -lh
total 92K
-rwxr-xr-x 1 admin springpath 92K Sep 17 10:42 hx_post_install

Notes:

  • Don’t miss the period at the end of the first line.
  • If you wanted to be fancy, you could combine step 1 &2 with:
    cp $(which hx_post_install) .

Now comes the bits where you manipulate the copy of the file using sed.  Basically, you have to swap the vnic names from the order used in the old M4 servers to the new order used by the M5 servers according to the table below:

vSwitch M4 vmnics used M5 vmnics used
vswitch-hx-inband-mgmt vmnic0 vmnic1 vmnic0 vmnic4
vswitch-hx-storage-data vmnic2 vmnic3 vmnic1 vmnic5
vswitch-hx-vm-network vmnic4 vmnic5 vmnic2 vmnic6
vmotion vmnic6 vmnic7 vmnic3 vmnic7

The problem is of course that if you replace say vmnic1 with vmnic4, when you later replace vmnic4 with vmnic2, you’ll be replacing the things you just replaced, so you need a double pass over the file.  Since I’m pretty sure the word rednectar does not occur in Cisco’s script, I’ll use that character pattern as a temporary placemarker for the word vmnic and then replace all occurrances of rednectar with vmnic at the end.

admin@hxscvm1:~$ sed -i 's/vmnic1/rednectar4/g' hx_post_install
admin@hxscvm1:~$ sed -i 's/vmnic2/rednectar1/g' hx_post_install
admin@hxscvm1:~$ sed -i 's/vmnic3/rednectar5/g' hx_post_install
admin@hxscvm1:~$ sed -i 's/vmnic4/rednectar2/g' hx_post_install
admin@hxscvm1:~$ sed -i 's/vmnic5/rednectar6/g' hx_post_install
admin@hxscvm1:~$ sed -i 's/vmnic6/rednectar3/g' hx_post_install
admin@hxscvm1:~$ sed -i 's/rednectar/vmnic/g' hx_post_install

Now that should take care of the bug – but there is one more annoying flaw with the script that I’d like to clean up too.  And that is the fact that if you run the script without using the –validate option, it still asks you if you want to run a health check – BUT THEN DOESN’T DO THE MTU check.

So, to make the script ship-shape, add one more change to remove the logic that skips the test if the –validate argument was not specified:

admin@hxscvm1:~$ sed -i 's/and args.validate//' hx_post_install

Great, but you’ll also want to know you are running a version of the script that has been updated, so finish with:

admin@hxscvm1:~$ sed -i "s/Select post_install/***RedNectar's Updated hx_post_install script M5 modifications have been applied.***\\\nSelect post_install/" hx_post_install
admin@hxscvm1:~$ sed -i 's/SCRIPT_VERSION = "4.0"/SCRIPT_VERSION = "4.1 RedNectar"/' hx_post_install

And you are ready to run, BUT you’ll need to be careful that you run the copy that you’ve just edited, so in the same directory, instead of issuing the command hx_post_install, you’ll need to put the location path (i.e. ./) as part of the command – so enter:

admin@hxscvm1:~$ ./hx_post_install
***RedNectar's Updated hx_post_install script M5 modifications have been applied.*** Select post_install workflow- 1. New/Existing Cluster 2. Expanded Cluster (for non-edge clusters) 3. Generate Certificate Note: Workflow No.3 is mandatory to have unique SSL certificate in the cluster. By Generating this certificate, it will replace your current certificate. If you're performing cluster expansion, then this option is not required.

And of course, from now on you can just use the modified script by typing ./hx_post_install at the admin@hxscvm1:~$ prompt.

WARNING: If you started your session to the cluster IP address, then you need to remember which controller VM actually serviced your session, and make sure you have a session with the same controller VM before you try the ./hx_post_install version of the command.

Happy HX Installing

RedNectar

 

Posted in Cisco, Hyperflex | Tagged | Comments Off on Hyperflex Post Install script fixer

Webex multi-screen support – where is it Cisco?

This is a reprint (with pictures) of an idea I submitted to Cisco – please support and vote for it after clicking this link.

Many Webex users have multiple screens, yet Webex fails to make use of this beyond the ability to share one of those screens – at least in Webex (Teams) and Webex Meetings – last time I checked in the obsolete Webex Training not even that was available.

The takeaway

I’d like Cisco to move to a default two window model when screen sharing is active. For the presenter, one “window” would be the screen being shared. And ALL the pesky panels in a SINGLE window that can be managed as a single unit and remember where it lives when screen sharing stops. For the viewer, one window for the screen being shared and one for the collection of other panels.


In this discussion, I am writing from the Webex Meetings experience, but probably the ideas are applicable on other variations. I’m also writing from the point of view of a macOS user – there may be some variations tot Webex behaviour in other versions. Now there are MANY ways and instances where this could be implanted, but I wish to fist make the distinction between a Presenter who is sharing a screen, and a Participant, who is juggling trying to view that screen while keeping track of chats, Q&A etc.

The Presenter – the person SHARING a screen.

For the Presenter, when I share my screen, I need to option to create a panel window. – or the option to NOT use a panel window and put up with what we have now – floating windows covering your shared screen until you move them. (I have 3 screens, some colleagues have more)

This is my shared screen. I want to move ALL those overlay panels into a single window

The panel window should show ALL the other panels: the participants video feeds, the chat, the Q&A etc ALL in a single window that can be maximised (or not) and NOT appear over the top of every window in every space (currently, if i open say the chat window and move it to my second screen, it sits in from to all other content on that window, and EVEN WHEN I SWAP TO ANOTHER SPACE it STILL sits on top of the windows on THAT other space (Windows users may not understand spaces, but macOS users will).

My second screen cluttered with multiple pesky panels

If I wanted that window to be in all spaces, I’d CHOOSE to make that window available on all desktops!

So please don’t force your screen onto every desktop unless I choose!

But I digress – back to the proposed panel window

This Panel Window should remember its settings, so it the presenter STOPs sharing a screen, but later resumes sharing, the Panel Window should remember how it was set up last time. I envisage that the panel window would have many options for showing, hiding, focusing on speakers etc

So to recap – I’d like all my floating panels in ONE SINGLE window, and only appear on one desktop (unless I choose to show on all desktops). Something like this:

This is my mock-up of how a second window might look.

This is a mock-up of how I MIGHT arrange the panels on my second screen, I’d envisage that the second secret would be something like the current Participant’s screen but without the shared screen.

AND I’d like Webex (Meetings) to remember this layout should I stop sharing and then start sharing again.

If another person is talking, I’d particularly like that person’s image (or video if they are using it) to dominate (and show their name) something like above (where I’ve had to ADD the name under the picture – I want the name there even if the video is on)

The Participant – the person viewing the shared screen

Now Cisco has made some great improvements with the experience for the viewer in terms of the options for layouts. But still no support for a second screen.

Why can’t a participant move the presenter’s shared screen to another monitor?

Why can’t a participant move the bit that is being shared to another screen, and have Webex support two windows, like what I’ve described above for the presenter?

Cisco – please improve your support for multi-screen layouts. We have moved to a world of working from home where MANY people have to put up with this day in and day out. It is So frustrating being forced to use Webex when there are so many limitations.

RedNectar

Posted in Cisco, Webex | Tagged , , | Comments Off on Webex multi-screen support – where is it Cisco?

Cisco has re-vamped their ACI Docs pages. Here’s what I think.

If you have upgraded your ACI to verison 5.1 or the recently released 5.2, you’ll notice a big change if you should ever venture to that rather obscure menu item Settings > Documentation > API Documentation

Settings > Documentation > API Documentation

What’s the difference?

What you used to get was a basic but wholesome view of the Cisco APIC Management Information Reference Model, with a list of all the Classes, Types, Events, Faults etc listed on the left side, with clickable links associated with each Class etc.

Clicking on one of the links opened a Pandora’s Box of information in the viewing pane. One of my favourites is fv:Tenant

Old APIC Management Information Model Reference

I’ve taken a bit of an extreme here – fv:Tenant is probably the largest class by far of the whole ACI Object model. For fun, I copied the information (text only) of the information pane above and pasted it in MS Word. I now have a 4140 page (556818 words) document that I can browse!

But if you knew how to navigate the page (there are a few handy shortcuts at the top) and use your browser’s find function, you could generally find what you wanted. Although, I must admit, the shortcuts at the top of the page may not work until the thousands of lines of content have loaded – which may be many seconds.

The new opening screen now presents a very fine search function – I has only to type the letters fvte before the search had located the fvTenant object. Happy to see the search is NOT case sensitive.

New APIC Object Model Documentation

But there are a couple of other subtle improvements too. There is a toggle on the right-hand side that (by default) restricts your search to configurable objects. And you can only search all Objects or Faults via tabs at the top left-hand side. I think everyone always used the All group in the old system, so I’m happy with this improvement.

Having found my object, clicking on it opens a sub-window on the right-hand side, which has a second link that I must click to actually see the information I need.

I have to click a second time to get any useful information

At first, I was annoyed at having to click twice, [edit:2021.06.19 Turns out you can just double-click the name] but in other contexts where you have a list of objects, you’ll find the information window stays open as you click on each object, making it quite a useful feature. However, it does reveal the absurdity of some of the object descriptions which were probably cooked up in a hurry for release 1.0. For instance, the description for a Tenant object includes this:

For example, you can create a tenant with contexts and bridge domains shared by other tenants.

Oh really? Good luck trying to do that!!!!!

Tip for Cisco: Time to review the object descriptions in the Object Model.

Anyway, back at the fvTenant object, the old Pandora’s Box of information is still mostly there (I sadly miss the old Diagram section), some of it less clear than before, some of it more clear.

For instance, the old system had a great section on Naming Rules – nicely formatted with links to other name formats (you can see them underlined in the picture below. As you can see on the right, the new format is a) not formatted at all, and b) is missing the links.

Old and new Naming Rules styles

Tip for Cisco: Keep the old style – codes like to see indents and good mon-spaced fonts, and coloured text always helps. Take a look at any coders editor for goodness sake! And keep the links.

One of the great features of the Old Style listing of the Naming Rules is that I could click on the word name above, and I’d be sent to the part of the page that shows the rules for tenant name, again well formatted and very clear to read.

Name object in the old MIM

From here I could easily see that a Tenant name has a maximum length of 63 characters and consists of only upper and lowercase letters, digits and the characters underscore, period and dash.

The good news is that the same information is not too hard to find in the new system either. With the fv:Tenant object still opened, I have several tabs I can navigate. The first one past the default Overview tab is the Properties tab and clicking on name gives me the same information. Not as succinctly, or as neatly as above, but there all the same:

Easy to find the Validator information in the Properties tab

And I really do have to call Cisco out on the choice of font again here. On the screen above I see the word:

WTF is lId?

Now – if any human can tell if the last 3 letters are double-l-d or double-I-d or Ild or lId then good for you – sure, the context probably reveals it, but this is a reference document.

Tip for Cisco: Stick with non-ambiguous fonts designed for coders when specifying names. It really does make a difference.

I do have one gripe with the Properties table in the new system. The old system also gave me a list of Constants that will be used – and I can’t find this list in the new system.

The old system also gave me a list of Constants that will be used – and I can’t find this list in the new system.

And this is important – without this information it would not have been possible to work out why a filter for TCP port 22 suddenly started allowing ALL traffic through! You can read about that disaster here.

Moving across the tabs, the Relationships tab has some key information right there, and this time with clickable links to the related object.

Relationships Tab

This is much more consumable than the older system, which did have the same information right under the diagram, but with the Relations separated from their corresponding object as shown above.

Relationships and MO Containers in the old view

In the new system, Managed Object (MO) containments get thier own tab – and again, much more consumable, and with the list of Managed Objects shown in a more manageable (still almost never-ending) vertical list, but really, why someone decided to change MO (Managed Object) to Mo (a state in the USa) I can’t understand!

Containment Tab

The remaining tabs (Faults, Events and Stats) are also presented slightly more nicely than the older version. In Events, for instance, the event Code is shown, whereas on the older version, you had to click the hyperlink on the event to discover the Event ID.

So what else is missing?

The most obvious omission in the new system is the massive diagram that accompanied the Object definition in the old system. For the sake of brevity, I’ve chosen one of the more manageable objects. Note that for each box in the diagram there is a clickable link under the diagram. The new system has the same MO information, but NOT the visual representation.

Old MIM Diagram for infra:AccPortGrp

The other sections that missing (apart from the Constants mentioned above) are probably not as important. Those that need to delve deeper into the programming side of ACI may disagree, but the old system also had sections for Containers Hierarchies, Contained Hierarchy, and Inheritance. In some cases, (such as fvTenant above) the Contained Hierarchy list was thousands of entries.

My verdict?

I could find more missing pieces if I dug deeper, but I think I’ve covered the major items. But at the end of the day what Cisco has done is given us not just a prettier version, but in many cases more useable too. There are some important pieces missing, but I hope they will be added back in a future update.

Key advantages of the new UI

  • Ability to filter on Configurable Only
  • The search function is fast. Schmick!
  • Tabbed interface is much neater and manageable than the old huge-html-page approach
  • The pop-up window that appears when an object is clicked makes it easy to quickly browse through many objects/attributes and see the contained information.

Key disadvantages of the new UI

  • Lack of attention to detail when it comes to presenting programming information. There are many typefaces/fonts designed specifically for programming, Cisco should use one of them.
    • Another example of the lack of attention to detail is the sudden translation of MO to Mo – it does make a difference. There could well be other examples too.
    • The CONSTANTS section needs to be shown for each relevant attribute
    • I’d like to see the Diagram section return, but I must admit I rarely used it.

RedNectar

Posted in ACI | Comments Off on Cisco has re-vamped their ACI Docs pages. Here’s what I think.

ACI and the HyperFlex Hiccup Cure

In my previous post, I explained how to regain access to a HyperFlex controller when ACI fails to update the IP to MAC mappings in the endpoint table by enabling the IP Aging option.

In this post I’ll show you how I reduced that failover time to about one minute.

To see if I could reduce the failover time, I turned to one of the best documents Cisco has ever produced for ACI – the ACI Fabric Endpoint Learning White Paper

And sure enough, I found that:

First-generation leaf switches cannot reflect IP address movement between two MAC addresses on the same interface with the same VLAN to the endpoint database. This sort of IP address movement may occur in a high-availability failover scenario in which GARP typically is used to update IP to MAC relation on upstream network devices. This behavior is resolved by enabling the GARP-based EP Move Detection option

And since my HyperFlex nodes are indeed connected to 1st generation ACI N9K-C9336PQ switches, this is exactly what I tried next:

GARP EP Move detection

The curious thing about this option is that appears under the L3 Configurations tab but ONLY if ARP Flooding is enabled under the General tab.

Time to set up a test to see how much faster the failover is with the GARP Based Detection option enabled for the Bridge Domain

Test Plan

For the record, my test platform is running HyperFlex Data Platform v4.0(2d) (the current recommended latest version) and connected to ACI N9K-C9336PQ switches running v14.2(4i) . The APIC is running v4.2(5n).

Recall, my physical setup is like this:

Physical

As I write this, the SCVM that has taken on the 172.16.19.30 management IP address is 172.16.19.31 with MAC address 00:0C:29:90:F4:70

apic1# show endpoints | grep "172\.16\.19\.3[0-3]"
 00:0C:29:82:4F:B2  172.16.19.33    learned       101    eth1/32   vlan-119   not-applicable
 00:0C:29:90:F4:70  172.16.19.31    learned       101    eth1/32   vlan-119   not-applicable
 00:0C:29:90:F4:70  172.16.19.30    learned       101    eth1/32   vlan-119   not-applicable
 00:0C:29:A9:B7:0D  172.16.19.32    learned       101    eth1/32   vlan-119   not-applicable

Armed with the information that the MAC address bound to the Mgmt IP address is shared 172.16.19.31 (the SCVM IP on ESXi host #1), my plan is to put ESXi host #1 into HX Maintenance Mode to force the election of another Mgmt SCVM and measure how long my Mgmt PC looses connectivity to the Mgmt IP address.

To do this I have set up:

  • A continuous ping from my mgmt PC to 172.16.19.30 – I’m using PowerPing to do this so I get timestamps
  • tcpdump sessions on the SCVMs capturing only ARP packets so I can see the Gratuitious ARP requests and replies.
  • an endless loop issuing the command vsh_lc -c "show system internal epmc endpoint ip 172.16.19.30" on the ACI APIC
    • The purpose of this command was to see when ACI’s COOP database was updated to show a different second IP address on the same host as 172.16.19.30

What I expected to happen is that once the two remaining SCVMs discover that 172.16.19.30 has failed, they will elect another SCVM to host the 172.16.19.30 address, and that VM will send gratuitous ARP requests to ensure ACI updates its endpoint table and my management IP will be able to gain access to the management IP again.

Test Results

Here’s the timeline of what happened. It wasn’t quite like I expected

Time Action
14:18:40 Initiate HyperFlex Maintenance Mode for ESXi Host#1
14:19:32 SCVM#1 answers ARP request for 172.16.19.30 from 172.16.19.33 so is still online
14:19:33 SCVM#1 answers ARP request for 172.16.19.30 from 172.16.19.32 so is still online
14:19:44 Last ping reply recieved from 172.16.19.30 on the Mgmt station, indicating HX Mgmt IP is offline from this point
14:19:56 SCVM #3 starts sending contunuous ARPs for 172.16.19.30 to FF:FF:FF:FF:FF:FF
14:20:17 SCVM #2 also starts sending contunuous ARPs for 172.16.19.30 to FF:FF:FF:FF:FF:FF
14:20:39 SCVM #2 starts replying to ARPs for 172.16.19.30 , first to a specific MAC address, then…
14:20:40 SCVM #2 starts sending contunuous ARP replies for 172.16.19.30 to FF:FF:FF:FF:FF:FF
14:20:40 COOP database starts showing endpoint 172.16.19.30 is now shared with 172.16.19.32 indicating that the leaf switch has updated the COOP database on receipt of the ARP reply to FF:FF:FF:FF:FF:FF
14:20:41 Mgmt Station gets replies from 172.16.19.30
14:20:59 SCVM #2 starts sending GARP requests to/from 172.16.19.30 to destination MAC FF:FF:FF:FF:FF:FF

What I expected would have happened is that the GARP requests would have been sent at about 14:20:40 – rather than a string of ARP replies. However, it seems the ARP replies had the same effect.

Total failover time based on last ping reply received from SCVM#1 to first reply from SCVM#1: 14:20:41-14:19:44=00:57 – just inder one minute, which is far better than the 12 minutes I achieved last time.

Conclutions:

  • ACI treats Gratuitous ARP replies just as you would expect GARP requests to be treated – in other words, ACI learns L2/L3 info from ARP replies sent to MAC FF:FF:FF:FF:FF:FF.
  • In ACI, by enabling
    • IP Aging in System Settings > Endpoint Controls, and…
    • …in the the ACI BD where 1st generation switches are used
      • ARP Broadcasting, and
      • GARP based detection for EP Move Detection Mode
  • HyperFlex management IP address failover when used in conjunction with ACI can be reduced to approximately one minute.

RedNectar

Postscript

While preparing to write this, I recorded my steps – it’s on YouTube but the transition to YouTube quality makes it almost impossible to see clearly. But if you have 7 mins to spare (Tip: play it back at double speed and on a 34″ monitor if you have one) the link is here: https://youtu.be/OxCEOAyKcSw

Posted in ACI, Cisco, Hyperflex | Comments Off on ACI and the HyperFlex Hiccup Cure

ACI and the HyperFlex Hiccup

Darn! We’ve lost connectivity to the Cisco HyperFlex Controller AGAIN. What could possibly be wrong?

Well, the problem relates to how Cisco HyperFlex uses a floating IP address across multiple Storage Controller VM MAC addresses, and how ACI maintains the IP to MAC address table.

Let’s start with the physical picture.

Physical

The focus is on the three Storage Controller VMs (SCVMs) in the above picture.  The heavy purple lines show the resolved path to the default gateway. Each SCVM has a its own MAC address and ONE of the SCVMs shares that MAC address with the HyperFlex Management IP address (172.16.19.30).

So the ACI fabric sees the IP to MAC resolution like this:

apic1# show endpoints | grep "172\.16\.19\.3[0-3]"
 00:0C:29:82:4F:B2  172.16.19.33    learned       101    eth1/32   vlan-119   not-applicable
 00:0C:29:90:F4:70  172.16.19.31    learned       101    eth1/32   vlan-119   not-applicable
 00:0C:29:A9:B7:0D  172.16.19.32    learned       101    eth1/32   vlan-119   not-applicable
 00:0C:29:A9:B7:0D  172.16.19.30    learned       101    eth1/32   vlan-119   not-applicable

Note how as far as ACI is concerned, the MAC address 00:0C:29:A9:B7:0D is shared by both 172.16.19.32 and 172.16.19.30.

The problem we are having has been caused by the fact that the floating HyperFlex Management IP address has actually “floated” to another node (172.16.19.31).  Any traffic that needs to go to 172.16.19.30 now needs to go to MAC 00:0C:29:90:F4:70.  But ACI hasn’t learned this, and never will unless it sees a packet FROM 172.16.19.30 sourced with MAC 00:0C:29:90:F4:70.

Packets from my management PC (172.16.5.102) addressed to 172.16.19.30 reach ACI, and ACI routes them to the correct subnet, but sends them to MAC 00:0C:29:A9:B7:0D. Here’s a few ICMP packets I captured on the 00:0C:29:A9:B7:0D host that prove this.

root@hxscvm2:~# tcpdump -i eth0 -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:16.187163 IP 172.16.5.102 > 172.16.19.30: ICMP echo request, id 1, seq 26, length 40
11:48:20.959581 IP 172.16.5.102 > 172.16.19.30: ICMP echo request, id 1, seq 27, length 40

Now let’s be clear – this is a problem caused by HyperFlex using multiple MAC addresses for a single IP address more than it is that ACI won’t forget the old entry.  The same problem could occur if a normal router was used such as if a normal topology like this was used.

traditional_layout

The difference here is that on the router, there would be an ARP cache that would hold the mapping of IP 172.16.19.30 to MAC 00:0C:29:A9:B7:0D that would time out after some time – typically 4 hours.  But ACI doesn’t do anything like this by default.  So long as the MAC says alive, and it will so long as 172.16.19.32 keeps sending packets, ACI keeps the entry. By default.  But there is a fix. Kind of.

In ACI, there is an option for IP Aging created specifically for this kind of scenario.  To configure IP Aging (and I consider this would be best practice to ALWAYS enable IP Aging) you need to navigate to the System > System Settings >> Endpoint Controls >| [IP Aging] tab.

ACI Config IP_Aging

Once IP Aging has been enabled, as explained in the ACI Fabric Endpoint Learning White Paper,

“IP aging policy tracks and ages unused IP addresses on an endpoint. Tracking is performed by using the endpoint retention policy, which is configured for the bridge domain to send ARP requests (for IPv4) and neighbor solicitations (for IPv6) at 75 percent of the local endpoint aging interval. When no response is received from an IP address that IP address is aged out.”

In our case, the default endpoint retention policy was in use, so the aging time was at 15 minutes. And sure enough, 12 minutes (≅ 75% of 15 mins) after enabling the IP Aging option, the SCVM currently hosting the floating IP received an ARP request from the default gateway IP:

12:09:45.833118 ARP, Request who-has 172.16.19.30 (ff:ff:ff:ff:ff:ff) tell 172.16.19.1, length 46
12:09:45.833134 ARP, Reply 172.16.19.30 is-at 00:0c:29:90:f4:70, length 28

And so Happy HyperFlex days were here again from this point onwards.  I was able to access the HX Management IP address (172.16.19.30) from my management PC.

BTW – if you are experiencing this problem and you don’t want to wait the 12 minutes for the IP to be re-mapped by ACI, you can issue the following command at the APIC CLI to clear the IP immediately:

apic1# fabric leaf_id clear system internal epm endpoint key vrf vrf:name ip 172.16.19.30

I have done this a few times in the past because in our lab environment where we do unusual things all the time, this is a common occurrence.  Today I decided to work out exactly what was going on.

Floating IP addresses are used in a number of load-balancing situations.  In some cases, like VRRP, a special virtual MAC address is assigned to the IP and the MAC floats along with the IP.

What I haven’t explored yet is what exactly goes on when a new SCVM takes on the floating IP address.  If best practices are followed, the new SCVM SHOULD send a gratuitous ARP request using it’s new MAC address – in which case both the traditional router scenario AND the ACI topology should respond by updating their mappings.  If this did indeed happen, then clearly (in our ACI setup anyway) ACI is not updating its mapping as it should.

I’ll explore this further in my next post!

RedNectar

 

Posted in ACI, Best Preactices, Cisco, GNS3 WorkBench, Hyperflex | Tagged , , | 3 Comments

Rednectar’s Rules for writing Lab Guides

I wrote these as a guide for lab writers whose work I get to review and are in the context of writing lab guides using the frustrating wordprocessor known as Microsoft Word. It is meant to be a set of instructions for writers to follow BEFORE passing them on to me.


Before saving ready to be check-formatted, take these simple steps

Page breaks, paragraphs, tabs and spaces

 

    1. Remove all page breaks. Page breaks are determined by grouping paragraphs together that need to stick together by using the “Keep with next” paragraph attribute.  [A “keep with previous” would be SO much better… please upvote this https://word.uservoice.com/forums/304924-word-for-windows-desktop-application/suggestions/33552385-keep-with-previous]
      1. This will save me from having to do my first task in every review, which is to search and replace all instances of page breaks with nothing.
      2. And while on the topic – make sure you apply “Keep with Next” to every cell in a table EXCEPT the last row.  [A “keep with previous” would be SO much better…]
    2. Remove all empty paragraphs.  Spacing between paragraphs is determined by the style. If you don’t like the amount of space between paragraphs, let me know which style you’d like to change. Remember that this will change ALL paragraphs of that style, that’s why we use styles. I reserve the right NOT to agree.
      1. This will save me from having to do my second task in every review, which is to search and replace all instances of two CRs with one CR
    3. Remove all double spaces except after full stops. Use <tab>s to space items if necessary, or create a table.

 

Graphics

    1. In general, every Graphic is to be either:
      • Placed inline, so that text flows around it, something like
        Press the gearicon icon; or
      • Given an entire paragraph to itself, like those above.  If you have two graphics that have to go one after the other or side by side, find a graphics program like Preview and combine the two graphics into one.  Don’t paste them as separate graphics and expect that they will stay side by side (they won’t) or on the same page (they might if you are lucky.)
      • In the paragraph where the graphic lives, don’t add any tabs or spaces.
    2. Do NOT use MS Word shapes, or if you do, they follow the same rules as graphics. One per paragraph.  If your graphic needs a circle or arrow super-imposed, use a graphics program to compose it, and paste the picture. Powerpoint is a convenient choice if you love the MS style shapes so much that you have to use them. Preview also does a good job.
      • If you DO use MS graphic shapes, there is no guarantee that they will appear on the page you meant them to be on. That’s just life with MS Word.

Other rules

  • We click buttons – we Don’t press them or push them or “go to” them
  • We don’t “go to” menus or tabs.  We navigate menus and click on tabs or select tabs. You can select menu items too. Using “Navigate to” combines a “Navigate” plus a “Select”
  • Every Step MUST require the user to take an action.  The following is NOT a step.

Step 1: The GET request failed because the API Key has not been added

  • The following IS a step

Step 1: Observe that the GET request failed because the API Key has not been added

  • We check boxes, we don’t tick them. Sometimes we clear or (ugh) uncheck them. We never untick them.  If you must use the work tick, make sure you are referring to a small insect. Oh, and when a box is checked or cleared, it is to be accompanied with a little symbol indicating this:

This checkbox is checked: 

This checkbox has been cleared: 

I’ll update this document if I think of any more!

RedNectar

icon

Posted in Microsoft, Microsoft Word, MS Word Tips | Tagged , | Comments Off on Rednectar’s Rules for writing Lab Guides