Nested ESXi Virtual Appliances

A logo for a software company

Description automatically generated

In the field of home labs, nested virtualization is the must to be able to create vSphere environments in a short time to test or try new features even in the EUC environment.

To help those who want to use nested virtualization, William Lam has been providing Nested ESXi Virtual Appliances for some years now.

Until a few days ago nested ESXi virtual appliances were available to download from William’s website (https://williamlam.com/) and since yesterday they have been available on the VMware Flings Community (Accessible with a free Broadcom community account that can be created here)

The convenience of using these virtual appliances is the ability to also create automatic scripts to be able to create nested environments such as vSphere clusters, vSAN and VCF (VMware Cloud Foundation) environments in a short time, which can then be used to test Omnissa’s EUC solutions as well.

Here is William’s official post

A screenshot of a phone

Description automatically generated

Nested ESXi Virtual Appliances

Approach to updating a horizon infrastructure

When approaching the upgrade of an infrastructure in the EUC world (as with most technologies in the IT world) it is necessary to define a roadmap of activities and follow it carefully. In many cases, IT technology vendors already have update procedures in place that should be followed carefully. When I started working as a consultant, the documentation was very scarce (we are talking about the end of the 20th century and the beginning of the 21st…) and the procedures were poorly documented and only those who took courses or had experience could approach with a certain “tranquility” updates of production environments.

Going back to EUC infrastructures and focusing on the VMware by Broadcom world (still for a while….given the transfer of the technology in question) we have a precise update sequence, especially if we talk about + technologies that interact with each other, and the need to verify the interoperability between the various technologies.

For example, we have this KB that gives us the upgrade sequence of a Horizon 8 infrastructure:

Update sequence for Horizon 7, Horizon 8, and compatible VMware products (78445)

A diagram of software

Description automatically generated

And the ability to use the interoperability portal:

https://interopmatrix.vmware.com/Interoperability

A screenshot of a computer

Description automatically generated

In my ten-year experience in updates and maintenance of vSphere and Horizon infrastructures, it has often happened that I have had to intervene and manage post-upgrade problems, where in most cases the problems were generated by the fact that I did not perform the update in the correct order or even did not complete all the upgrade steps.

For example, I have experienced situations where, following upgrades, the copy and glue to and from VDI sessions no longer worked correctly in a Horizon infrastructure.

In the end, the problem was solved by also performing the update step of the Horizon ADMX templates in Active Directory, something that the customer or whoever had done the update for him had not done.

Approach to updating a horizon infrastructure

VMware Pre Broadcom vs VMware Broadcom – Primi dati reali

Attenzione è una mia valutazione….quindi non sparate sul sistemista

Broadcom ha acquisito VMware, ormai lo sappiamo tutti.
La situazione di incertezza aleggia ovunque soprattutto sul mondo vSphere e sui costi (con tanti competitor che provano a ritagliarsi la loro fetta di mercato togliendole al leader indiscusso di questi anni)


Finalmente in questi giorni incomincio ad avere i primi dati effettivi (Prezzi ecc…) su cui iniziare a fare i primi ragionamenti.
!!Attenzione non voglio dare giudizi ma voglio solo paragonare due offerte fatte allo stesso cliente che abbiamo dovuto rivedere a seguito del nuovo listino (E parliamo di prezzi di listino.. senza eventuali scontistiche)!!

Ragioniamo su un cluster vSphere con 3 nodi da 2 processori ciascuno da 16 core.

Con le precedenti licenze e il vecchio listino nello scenaro ipotizzato dovevamo considerare:

  • Licenza VMWARE VCENTER SERVER 8 STANDARD
  • Licenza VMWARE VSPHERE 8 STANDARD FOR 1 PROCESSOR
  • Support/Subscription
  • Support/Subscription

Con il nuovo listino e le nuove tipologie di licenze invece dobbiamo considerare:

  • VMWARE VSPHERE STANDARD per core (Che comprende la licenza di vCenter)

Entrambe le soluzioni con 5 anni.

Da una prima analisi le prime valutazioni sono:
Con il nuovo listino si viene a pagare circa 30-35% in meno.
Ho una semplificazione nella quotazione (una sola voce rispetto alle 4 precedenti)

Ovviamente:

  • Non abbiamo le licenze perpetue (comunque chi non vuole il supporto sul proprio ambiente di produzione o la possibilità di effettuare aggiornamenti?)
  • é una prima offerta e la quotazione può dipendere da vari fattori e i prezzi potrebbero nuovamente cambiare
  • Le funzionalità all’interno dei bundle possono essere leggermente differenti (il link per vedere le funzionalità presenti nei nuovi bundle VMware vSphere® Product Line Comparison)
  • Posso aver sbagliato i calcoli 🙂
  • Possono avermi dato dei prezzi sbagliati 🙂 spero di no per il cliente 🙂

ma aspettavo di avere due informazioni reali per fare le mie prime considerazioni.

L’unica cosa che posso dire è di valutare con attenzione il cambio …. (io sono il primo che accetta nuove sfide..) ma attenzione a tutti i prezzi nascosti e valutate bene!

P.S. se qualcuno ha delle esperienze in merito … condividiamole.

VMware Pre Broadcom vs VMware Broadcom – Primi dati reali

Check TCP port 443

During the maintenance and updating of the Horizon Connection server components, one aspect is the necessary wait for the connection servers to correctly resume responding on TCP port 443.
During one of the many activities on Horizon my customer created a simple and effective door test script.
Even though it is very simple and intuitive, I want to share the code with you:

do {
    $check = netstat -ano | findstr 0.0.0.0:443
    "Waiting 5 seconds and retry" 
    sleep 5
} while (!$check)
$check

Check TCP port 443

DRS and HPE SimpliVity

In recent days, a customer reported an anomaly on an HPE SimpliVity cluster hosting instant clone Horizon VDIs. In detail:

  • vSphere with seven hosts present, two were always at 98% CPU utilization and 90% RAM utilization.
  • Continuous vMotion generated by the VM DRS to and from those two HOSTS.

After a careful analysis, we identified that there were no problems at the vSphere infrastructure level.
The issue was due to a Simplivity feature called IWO.

By disabling IWO and keeping DRS active (Full automatic) I have an optimal balance of CPU and RAM load between hosts at the expense of a slight increase in I/O trip times

Scenario – Even VM Load Distribution

I want even VM load across my cluster in terms of CPU and memory. Data locality and I/O performance are not top priorities. Most applications are CPU and memory intensive, and adding 1ms to 2ms to I/O trip times will not impact application performance.

In this scenario, IWO can be disabled thus ensuring no DRS affinity rules are populated into vCenter server. Suppressing DRS affinity rules will allow VMware DRS or allow you to directly distribute VMs across the cluster as desired to ensure all VMs are adequately resourced in terms of CPU and memory. The ‘Data Access Not Optimized’ alarm can be suppressed within vCenter server.

More information:

https://community.hpe.com/t5/around-the-storage-block/how-vm-data-is-managed-within-an-hpe-simplivity-cluster-part-3/ba-p/7033153

DRS and HPE SimpliVity

Configure ControlUp for VMware Horizon Instant Clone VDI monitoring

In this guide, we will analyze how to configure ControlUP COP (ControlUP on-Premise) to monitor a VMware Horizon 2309 infrastructure with Instant Clone Desktop Pools (we will not cover the installation part of the product)

The following steps are required:

  • Control UP COP Server Component Installation (Optionally use an external SQL instance or SQL EXPRESS present in the Server component installation)
  • Installing the Control UP Console (Can also be installed on the same server)
  • Installing Agent Control Up on the GoldImage
  • Horizon Infrastructure Inventory
  • VirtualMachine Inventory (For this step we can also implement an automatism)

Requirements for the server part:


COP Server
COP Server Console Machine
Machine Windows Server Windows Server orWindows
Operating System Windows Server supported versions:2022,2019,2016 Windows Server supported versions:2022,2019,2016
OR Windows 11, 10
CPU* 2 CPUs 2 CPUs
Memory* 8 GB RAM 8 GB RAM
Disk Space* 10 GB 10 GB
Required Software & Permissions
  • .NET Framework 4.8 or later
  • PowerShell 5.x or later
.NET Framework 4.5 or later

Requirements for Part DB:

MSSQL Versions (Standard, Enterprise, or Express) Maximum Database Size Collation
2022,2019,2017,2016,2014 10 GB SQL_Latin1_General_CP1_CI_AS

Requirements for the VDI part:


ControlUp Agent
ControlUp Agent
Machine No server installation necessary. Deployed onto Windows machines that are monitored by ControlUp(Linux monitored via API).
Operating system Windows Server supported versions:
202220192016 (Core or Full)ORWindows 11, 10
Required installed software .NET 4.5 or later
TCP PORT 40705

A Service Account to access the Horizon infrastructure:

The Read-Only role is sufficient for all monitoring purposes. If you want to perform built-in Horizon actions, then the service account needs the following permissions:

  • Enable Farm and Desktop Pools
  • Manage Machine
  • Manage Sessions
  • Manage Global Sessions (Cloud Pod architecture only)

So what is needed is:

Download the version of ControlUP COP from the VMware site

Log in to the customer portal and in the product area under Desktop & End-User Computing

A screenshot of a computer

Description automatically generated

Log in to OEM Addons

A screenshot of a computer

Description automatically generated

Download the on-premise version

Perform the basic installation

Once the COP version is installed and the console is installed, log in to our ControlUP installation

A screenshot of a computer

Description automatically generated

How to install the agent on the GoldImage:

  1. The agent MSI file is on the downloaded file zip from VMware Portal
  2. Open the Real-Time Console and go to Agent Settings and copy your Agents Authentication Key. The key is used to connect the Agent to your ControlUp environment.

A screenshot of a computer

Description automatically generated

  1. Run the installation of the MSI package on the machine where you want to install the Agent.
  2. During the installation, paste the authentication key that you copied from the Real-Time Console.

A screenshot of a computer

Description automatically generated

  1. Complete the installation. The Agent is installed on the machine and the machine can be monitored from the Real-Time Console.
  2. Take the snapshot
  3. Deploy the new master image on Desktop Pool

Now from the ControlUp Management console, we are able to:

  • Connect our Vmware Horizon infrastructure
  • Connect the instant clone machine

Add Horizon infrastructure:

A screenshot of a computer

Description automatically generated

Add the infrastructure info

A screenshot of a computer

Description automatically generated

Click on OK

A screen shot of a computer

Description automatically generated

Add the pod to the console

A screenshot of a computer

Description automatically generated

Now on the left panel, we have our Horizon infrastructure added.

A screenshot of a computer

Description automatically generated

To monitor correctly our instant clone (after adding the agent) we need to discover the VM like a Machine

A screenshot of a computer

Description automatically generated

Search with the partial name of the VDI machines

A screenshot of a computer

Description automatically generated

Select cancel

A screenshot of a computer error message

Description automatically generated

We are VM on the left control panel in black status

A screenshot of a computer

Description automatically generated

After a few seconds the VDI VM Goes to Green

A screenshot of a computer

Description automatically generated

Auto connect state must be enabled (this function is important when the instant clone VDI is removed and recreated).

A screenshot of a computer

Description automatically generated

Now we can monitoring the Instant-Clone VDI

Check the VDI logon duration

Now we can manage and control the infrastructure, for example, to check the logon duration

A screenshot of a computer

Description automatically generated

A screenshot of a computer

Description automatically generated

What happens when VDI instant clones are regenerated?

If a user disconnects from his VDI of the instant clone type, it is destroyed and recreated, on the ControlUp side this is put in the Red -> Yellow state until it returns to Green

When recreating

A screenshot of a computer

Description automatically generated

After recess

A screenshot of a computer

Description automatically generated

Dynamic inventory

For a dynamic inventory of VDI, we can use Synchronization with Universal Sync Script (I’ll talk about this in a future post)

EUC Synchronization with Universal Sync Script (controlup.com)

After installation, we can schedule or start manually the script to sync my ControlUP with my EUC infrastructure.

References:

How to Deploy the Agent on Your Master Image for PVS/MCS/Linked/Instant Clones (controlup.com)

EUC Synchronization with Universal Sync Script (controlup.com)

ControlUp On-Premises

Configure ControlUp for VMware Horizon Instant Clone VDI monitoring

vSphere Distributed Switch health check

For us VMware systems engineers who every day find ourselves “dialoguing” with those who manage the network ecosystem, we can only find the vSphere Distributed Switch health check function useful.

  1. What these checks allow us to highlight:

These are some of the common configuration errors that health check identifies:

  • Mismatched VLAN trunks between a vSphere distributed switch and a physical switch.
  • Mismatched MTU settings between physical network adapters, distributed switches, and physical switch ports.
  • Mismatched virtual switch teaming policies for the physical switch port-channel settings.

The network health check in vSphere monitors the following three network parameters at regular intervals:

  • VLAN: Checks whether vSphere distributed switch VLAN settings match trunk port configuration on the adjacent physical switch ports.
  • MTU: Checks whether the physical access switch port MTU setting based on per VLAN matches the vSphere distributed switch MTU setting.
  • Network adapter teaming: Checks whether the physical access switch ports EtherChannel setting matches the distributed switch distributed port group IP Hash teaming policy settings.
  1. How to activate:

Access the network section of our vCenter

Select the vDS on which we want to activate health checks

A screenshot of a computer

Description automatically generated

And enable the check that interests us:

A screenshot of a computer

Description automatically generated

  1. Where to check the outcome of the checks?

Wait a few minutes and already first feedback we can have it on ESXi hosts using the vDS in question, where if there are problems the classic red dot will be displayed

A screenshot of a computer

Description automatically generated

For more details, access the network section of our vCenter and select the vDS in question

A screenshot of a computer

Description automatically generated

And we can see that on the vmnic0 and vmnic3 of the first host, there are vLANs of which we have a Portgroup but which are not proposed correctly on all the ports of the switches to which we have attested our hosts. Then we have to have the configuration verified by our colleagues in the network.

  1. How to turn it off:

Repeat the enabling steps but this time select disable.

  1. Risks in activating it (we always consider activating it for a short time)

Depending on the options that you select, the vSphere Distributed Switch Health Check can generate a significant number of MAC addresses for testing teaming policy, MTU size, vLAN configuration, resulting in extra network traffic.
Ensure the number of MAC addresses to be generated by the health check will be less than the size of the physical switch(es) MAC table. Otherwise, there is a risk that the switches will run out of memory, with subsequent network connectivity failures. After you disable vSphere Distributed Switch Health Check, the generated MAC addresses age out of your physical network environment according to your network policy.

More info:

vDS Health Check reports unsupported VLANs for MTU and VLAN (2140503) (vmware.com)

Enabling vSphere Distributed Switch health check in the vSphere Web Client (2032878) (vmware.com)

vSphere Distributed Switch health check

Enable copy and paste between Guest Operating System and Remote Console

Copy and paste operations between the guest operating system and remote console are deactivated by default. 

To enable it:

  • Browse to the virtual machine in the vSphere Client inventory
  • Right-click the virtual machine and click Edit Settings.
  • Select Advanced Parameters.
  • Add or edit the following parameters.

    isolation.tools.copy.disable False
    isolation.tools.paste.disable False
    isolation.tools.setGUIOptions.enable True
    These options override any settings made in the guest operating system’s VMware Tools control panel.
  • Click OK.
  • (Optional) If you made changes to the configuration parameters, restart the virtual machine.

Enable copy and paste between Guest Operating System and Remote Console

Remove an instant clone desktop pool in a deleting state

If we are removing a desktop pool from a Horizon infrastructure and we find ourselves in a situation that remains in a deleting state:

Graphical user interface, text, application

Description automatically generated

We can force the removal as follows:

  • Remove any VDI VMs still in your vSphere infrastructure.
  • Remove VM template, replication and parent.

In my example, we have the following situation

Graphical user interface, text, application

Description automatically generated

To remove them we use the tool iccleanup.cmd, we find te command on the connection servers by launching the following command to access:

iccleanup.cmd -vc <ome of vcenter> -uid < admin user of vcenter>

We enter the account password

Run with the list command the list of service VMs to be deleted

Text

Description automatically generated

In my case, they are the VMs indicated with ID 2 and 3

We start from 2 and first launch the unprotect indicating with -I the number 2 (unprotect -I 2) and confirm by writing unprotect

Text

Description automatically generated

Then we delete with the question delete -I 2 and confirm by writing  delete

Graphical user interface, text

Description automatically generated

Let’s go back by writing Back

Relaunch the List command and verify that Index has taken the other chain of system VMs to be deleted

Text

Description automatically generated with medium confidence

Ha was taken as index 2

We review the operations of unprotect and delete once again for index 2

At the end of the vCenter  (they are service VM from other pools that should not be deleted)

Text

Description automatically generated

Already in this case, we may have deleted the DesktopPool that was in a deleting state.

If the Pool in question is still present and in a deleting state, we proceed to access the ADSI Edit console and modify the ADAM DB by deleting the references left to the pool in deleting state (in my case ICTPM)

  • Remove Desktop Pool from ADAM Database

To connect follow this KB:

Connecting to the Horizon View Local ADAM Database (vmware.com)

remove the pool from Adam as follows.

Graphical user interface, text, application

Description automatically generated

Graphical user interface, text

Description automatically generated

Remove an instant clone desktop pool in a deleting state