ESXi

Remove an instant clone desktop pool in a deleting state

fabio1975, , , , Leave a comment

If we are removing a desktop pool from a Horizon infrastructure and we find ourselves in a situation that remains in a deleting state:

Graphical user interface, text, application Description automatically generated

We can force the removal as follows:

  • Remove any VDI VMs still in your vSphere infrastructure.
  • Remove VM template, replication and parent.

In my example, we have the following situation

Graphical user interface, text, application Description automatically generated

To remove them we use the tool iccleanup.cmd, we find te command on the connection servers by launching the following command to access:

iccleanup.cmd -vc <ome of vcenter> -uid < admin user of vcenter>

We enter the account password

Run with the list command the list of service VMs to be deleted

Text Description automatically generated

In my case, they are the VMs indicated with ID 2 and 3

We start from 2 and first launch the unprotect indicating with -I the number 2 (unprotect -I 2) and confirm by writing unprotect

Text Description automatically generated

Then we delete with the question delete -I 2 and confirm by writing  delete

Graphical user interface, text Description automatically generated

Let’s go back by writing Back

Relaunch the List command and verify that Index has taken the other chain of system VMs to be deleted

Text Description automatically generated with medium confidence

Ha was taken as index 2

We review the operations of unprotect and delete once again for index 2

At the end of the vCenter  (they are service VM from other pools that should not be deleted)

Text Description automatically generated

Already in this case, we may have deleted the DesktopPool that was in a deleting state.

If the Pool in question is still present and in a deleting state, we proceed to access the ADSI Edit console and modify the ADAM DB by deleting the references left to the pool in deleting state (in my case ICTPM)

  • Remove Desktop Pool from ADAM Database

To connect follow this KB:

Connecting to the Horizon View Local ADAM Database (vmware.com)

remove the pool from Adam as follows.

Graphical user interface, text, application Description automatically generated

Graphical user interface, text Description automatically generated

Remove an instant clone desktop pool in a deleting state

vSphere DRS functionality was impacted due to an unhealthy state vSphere Cluster Service

fabio1975, , , , , , , Leave a comment

If you see such an error on the Cluster object of a vSAN (in my case it appeared on two vSAN clusters managed by the same vCenter)

vSphere DRS functionality was impacted due to an unhealthy state vSphere Cluster Service …….

an unhealthy state of the Service cluster

Graphical user interface, text, application, email Description automatically generated

Errors such as the following in the EAM log. vCenter LOG

EAM.log:

2023-01-26T13:16:39.996Z |  INFO | vim-monitor | VcListener.java | 131 | Retrying in 10 sec.
2023-01-26T13:16:41.432Z | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch
com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.
        at com.vmware.eam.vmomi.EAMInitRequestFilter.handleBody(EAMInitRequestFilter.java:57) ~[eam-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl$SingleRequestDispatcher.handleBody(DispatcherImpl.java:373) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl$SingleRequestDispatcher.dispatch(DispatcherImpl.java:290) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl.dispatch(DispatcherImpl.java:246) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.http.impl.CorrelationDispatcherTask.run(CorrelationDispatcherTask.java:58) [vlsi-server.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_345]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_345]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_345]
2023-01-26T13:16:50.007Z |  INFO | vim-monitor | ExtensionSessionRenewer.java | 190 | [Retry:Login:com.vmware.vim.eam:b55a7f93b59f0f7e] Re-login to vCenter because method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null
2023-01-26T13:16:50.007Z |  INFO | vim-monitor | OpId.java | 37 | [vim:loginExtensionByCertificate:913aec585658e328] created from [Retry:Login:com.vmware.vim.eam:b55a7f93b59f0f7e]
2023-01-26T13:16:51.440Z | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch
com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.


And you see the lack of vCLS VMs in the two vSANs

To resolve the anomaly you must proceed as follows:

  • vCenter Snapshots and Backup
  • Log in to the vCenter Server Appliance using SSH.
  • Run this command to enable access the Bash shell:

shell.set --enabled true

  • Type shell and press Enter.
  • Run this command to retrieve the vpxd-extension solution user certificate and key:

mkdir /certificate

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

  • Run this command to update the extension’s certificate with vCenter Server.

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s localhost -u "Administrator@domain.local"

Note: If this produces the error “Hostname mismatch, certificate is not valid for ‘localhost'”, change ‘localhost’ to the FQDN or IP of the vCenter. The process is checking this value against the SAN entries of the certificate.

Note: The default user and domain is Administrator@vsphere.local. If this was changed during configuration, change the domain to match your environment. When prompted, type in the Administrator@domain.local password.

  • Restart EAM and start the rest of the services with these commands:

service-control --stop vmware-eam

service-control --start --all

vSphere DRS functionality was impacted due to an unhealthy state vSphere Cluster Service

Horizon Instant Clone -VM Replica and Template in inaccessible state

fabio1975, , , Leave a comment

In the various maintenance activities of a Horizon infrastructure, it can happen to find VMs of the instant clone chain in an inaccessible state. (Caused by issues on hosts or vCenter such as sudden shutdowns without properly maintaining Horizon Desktop Pools.)

Image that contains text Auto-generated description

In the VMware Horizon solution there is a tool, from the command line, that allows the cleaning of these objects.

The tool is present in the directory

C:\Program Files\VMware\VMware View\Server\tools\bin>

of one of the connection servers of the Horizon infrastructure

The command is iccleanup.cmd

The first step is to connect to the vCenter in question by launching the following command

iccleanup.cmd -vc <fqdn of vCenter> -uid <administrative user>

Once you have entered the password you will have the possibility to list the VMs of the instant clone infrastructures implemented on that vCenter with the LIST command

Image that contains text Auto-generated description

Or delete objects in an inaccessible state, for example:

With the delete –index 2 command

Image that contains text Auto-generated description

After the completion of the cleaning, the situation will be as follows:

Image that contains text Auto-generated description

Horizon Instant Clone -VM Replica and Template in inaccessible state

Script to see Datastore Permission

fabio1975, , , , , Leave a comment

Last day in the VMware Community I saw a request for:

“I have AD group like mydomain\mygroup.

This group have access for many datastores.

How i can use powercli to get full list of datastores which the group can manage?”

I made this PowerCLI script:

$cred = Get-Credential
Connect-ViServer <vcenter-FQDN>; -Credential $cred
$datastores = Get-Datastore | Select Name
$groupAD = "domain\group"
$report = @()
foreach ($datastore in $datastores) {
  $report +=  Get-VIPermission
| Where-Object {($_.Entity.Name -Like $datastore.Name) -and ($_.Principal -eq $groupAD)} |Select Principal,Role,@{n='Datastore';E={$datastore.Name}},@{n='Entity';E={$_.Entity.Name}},@{N='Entity Type';E={$_.EntityId.Split('-')[0]}},@{N='vCenter';E={$_.Uid.Split('@:')[1]}}
}
$report | Export-Csv <path\csvfile> -NoTypeInformation

Script to see Datastore Permission

Copy-VMGuestFile

fabio1975, , , Leave a comment

I needed to copy a folder to virtual machine on the DMZ network segment.
The firewall rule blocks any access to VM, well I used the Copy-VMGuestFile Powercli Command.

Connect-VIServer <vcenter server FQDN or IP>

$vm = Get-VM -Name <VM target>

Get-Item “<Source Path>” | Copy-VMGuestFile -Destination “<Destination Path on VM>” -VM $vm -LocalToGuest -GuestUser <User VM Guest> -GuestPassword <Password User VM Guest>

After the first tentative I receive this error:

Copy-VMGuestFile : 09/12/2021 11:08:40 Copy-VMGuestFile The request was aborted: The request was cancelled.
At line:1 char:27

Probably it is a time out error and I try to change the WebOperation Timeout Seconds.

PS C:\Windows\system32> Set-PowerCLIConfiguration -WebOperationTimeoutSeconds -1

Scope ProxyPolicy DefaultVIServerMode InvalidCertificateAction DisplayDeprecationWarnings WebOperationTimeout
Seconds
—– ———– ——————- ———————— ————————– ——————-
Session UseSystemProxy Multiple Unset True -1
User Multiple
AllUsers -1

After the change the error it is resolved

Copy-VMGuestFile

LDAP Identity source and vCenter

fabio1975, , , , , Leave a comment

Whenever we installed a new vCenter the activity always included integration with Active Directory and normally IWA (Integrated Windows Authentication) was used.
Since vSphere 7.0 version this possibility has been deprecated
so it is good to start with the integration of the vCenter with Active Directory via LDAP.
In our case, we will use LDAPS which uses a certificate

For first the step we need to create the certificate:

  • Use SSH to vCenter connection

On shell use this command

openssl s_client -connect <DC FQDN>:636 -showcerts

Copy the certificate output with  —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–

Past on Notepad and save with .crt extension

Now we will go to configure the Identity Sources on vCenter:

  • Login as Single Sign-On Administrator to vCenter
  • Navigate to Menu > Administration > Single Sign-On > Configuration
  • In the Identity Provider tab, open Identity Sources
  • Click ADD
  • Select Active Directory over LDAP or OpenLDAP, depending on your directory type.

Fill out the remaining fields as follows:
Identity Source Name: Label
Base DN for users: The Distinguished Name (DN) of the starting point for directory server searches. Example: “DC=pollaio,DC=lan”.
Base DN for groups: The Distinguished Name (DN) of the starting point for directory server searches.
Domain name: Your domain name. Example: “pollaio.lan”
Domain alias: Your NetBIOS name. Example: “pollaio.lan”
Username: Domain user with at least browse privileges. Example: “pollaio\administrator”.
Connect to:  “ldaps://<DC FQDN>”.

  • Click Browse next to SSL Certificate
  • Select the .cer file created in before step
Now we are ready to login to the vCenter with domain user (remember to assign the correct permission to domain group or user group)

If you want check the correct use of SSL certificate on the authentication to Active Directory with LDAP connection check the websso.log:

LDAP Identity source and vCenter

Capture Code – vSphere Web Client

fabio1975, , , , , , , Leave a comment

One of the conveniences of administering VMware solutions is being able to use code to create scripts to perform repetitive tasks or automate processes

One of the vSphere Web Client features that can help those new to the PowerCli is the Capture Code, it basically allows you to list and save the Powercli commands of the actions you are doing with the vSphere Web Client.

To activate it just access the vSphere Web Client, from the Menu select Developer Center

Select Code Capture and enable it by placing the “Enable Code Capture” flag on the right (which turns green)

At this point, a space will appear in our frame where the commands will be listed with some operations, such as Clear and start another, Copy and Download

Where the Download option generates you the ps1 file with the Powercli commands of the recorded operations

To start and stop a recording session you can use the buttons:

Or the red button that appears at the top of the WebClient once “Enable Code Capture” is enabled

Bye

Capture Code – vSphere Web Client

Ingest your VMware VCSA Appliance logs into Azure Sentinel

fabio1975, , , , Leave a comment

In an old post, I described how to send ESXi logs to Azure Log Analytics to ingest at Azure Sentinel, now I describe Step to Step how to send vCenter logs.

The first step is to do step by step this configuration:

Ingest your VMware ESXi logs into Azure Sentinel
After this we need to:
  • Change to the settings of VCSA Appliance to send the logs to Syslog Gateway Server
  • Configure the Log Analytics Agent, installed on Syslog Gateway Server to process the Facility Local0
  • Change la function VMwareESXi (It was created for ESXi Log check my old post) or create a query custom to parse the log on Azure Log Analytics

Change to the settings of VCSA Appliance to send the logs to Syslog Gateway Server

For configuring the VCSA you can use this VMware KB

Forward vCenter Server Log Files to Remote Syslog Server (vmware.com)

Protocol and port depend on your infrastructure configuration (you need to enable communication from VCSA to Syslog Gateway Server on the select TCP/UDP port)

and enable send events (it is enabled by default, but a check is a good idea)

Configure Streaming of Events to a Remote Syslog Server (vmware.com)

Now you can connect to the Syslog Gateway Server and check if the Syslog server received the logs from the VCSA Appliance

Use SSH to connect at the Syslog Gateway Server and use this command

cat /var/log/syslog | grep <fqdn vCenter> | more

in my situation

cat /var/log/syslog | grep vcenter | more

Configure the Log Analytics Agent, installed on Syslog Gateway Server to process the Facility Local0

Connect to Azure Portal and on Azure Log Analytics Service enable the correct facility (local0)

After 10/15 minutes the new configuration will be applied on Syslog Gateway Server (you can check the file /etc/rsyslog.d/95-omsagent.conf on Syslog Gateway)

Change the VMwareESXi function (It was created for ESXi Log check my old post) or create a query custom to parse the log on Azure Log Analytics

Finally, you can query the data on Azure Log Analytics

Syslog | where HostName contains “<FQDN vCenter>”

or optionally you can edit the function create for Ingest ESXi log (check my old POST) and insert the vCenter FQDN Name in the same position where there is the ESXi FQDN Name.

Currently, on Azure Sentinel there are no specific Workbooks for VMware, all queries are to be created

Ingest your VMware VCSA Appliance logs into Azure Sentinel