vSphere DRS functionality was impacted due to an unhealthy state vSphere Cluster Service

If you see such an error on the Cluster object of a vSAN (in my case it appeared on two vSAN clusters managed by the same vCenter)

vSphere DRS functionality was impacted due to an unhealthy state vSphere Cluster Service …….

an unhealthy state of the Service cluster

Graphical user interface, text, application, email

Description automatically generated

Errors such as the following in the EAM log. vCenter LOG

EAM.log:

2023-01-26T13:16:39.996Z |  INFO | vim-monitor | VcListener.java | 131 | Retrying in 10 sec.
2023-01-26T13:16:41.432Z | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch
com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.
        at com.vmware.eam.vmomi.EAMInitRequestFilter.handleBody(EAMInitRequestFilter.java:57) ~[eam-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl$SingleRequestDispatcher.handleBody(DispatcherImpl.java:373) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl$SingleRequestDispatcher.dispatch(DispatcherImpl.java:290) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.impl.DispatcherImpl.dispatch(DispatcherImpl.java:246) [vlsi-server.jar:?]
        at com.vmware.vim.vmomi.server.http.impl.CorrelationDispatcherTask.run(CorrelationDispatcherTask.java:58) [vlsi-server.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_345]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_345]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_345]
2023-01-26T13:16:50.007Z |  INFO | vim-monitor | ExtensionSessionRenewer.java | 190 | [Retry:Login:com.vmware.vim.eam:b55a7f93b59f0f7e] Re-login to vCenter because method: currentTime of managed object: null::ServiceInstance:ServiceInstance failed due to expired client session: null
2023-01-26T13:16:50.007Z |  INFO | vim-monitor | OpId.java | 37 | [vim:loginExtensionByCertificate:913aec585658e328] created from [Retry:Login:com.vmware.vim.eam:b55a7f93b59f0f7e]
2023-01-26T13:16:51.440Z | ERROR | vlsi | DispatcherImpl.java | 468 | Internal server error during dispatch
com.vmware.vim.binding.eam.fault.EamServiceNotInitialized: EAM is still loading from database. Please try again later.


And you see the lack of vCLS VMs in the two vSANs

To resolve the anomaly you must proceed as follows:

  • vCenter Snapshots and Backup
  • Log in to the vCenter Server Appliance using SSH.
  • Run this command to enable access the Bash shell:

shell.set --enabled true

  • Type shell and press Enter.
  • Run this command to retrieve the vpxd-extension solution user certificate and key:

mkdir /certificate

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

  • Run this command to update the extension’s certificate with vCenter Server.

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s localhost -u "Administrator@domain.local"

Note: If this produces the error “Hostname mismatch, certificate is not valid for ‘localhost'”, change ‘localhost’ to the FQDN or IP of the vCenter. The process is checking this value against the SAN entries of the certificate.

Note: The default user and domain is Administrator@vsphere.local. If this was changed during configuration, change the domain to match your environment. When prompted, type in the Administrator@domain.local password.

  • Restart EAM and start the rest of the services with these commands:

service-control --stop vmware-eam

service-control --start --all

vSphere DRS functionality was impacted due to an unhealthy state vSphere Cluster Service

Copy file to VCSA with SCP

Well, in recent weeks we have often talked about how to heal vCenters from the log4j vulnerability.
I guess the first thing we all thought was “What a show VMware support released scripts to run to solve the problem …” and then every one to use WinSCP or similar tools/commands to copy the file …. but many will have found it impossible to copy files using the Root user …. but how SSH works but the SCP command does not work!
Well, the problem comes from the shell associated with the Root user. It is not the classic BASH but the APPLIANCESH.
Then we proceed as follows:

  • Let’s connect in SSH to the vCenter Virtual Appliance
  • We access the Bash SHELL with the command SHELL
  • We enable BASH as the default shell for the root user
  • We run our SCP
  • We re-enable APPLIANCESH for the root user

Copy file to VCSA with SCP

LDAP Identity source and vCenter

Whenever we installed a new vCenter the activity always included integration with Active Directory and normally IWA (Integrated Windows Authentication) was used.
Since vSphere 7.0 version this possibility has been deprecated
so it is good to start with the integration of the vCenter with Active Directory via LDAP.
In our case, we will use LDAPS which uses a certificate

For first the step we need to create the certificate:

  • Use SSH to vCenter connection

On shell use this command

openssl s_client -connect <DC FQDN>:636 -showcerts

Copy the certificate output with  —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–

Past on Notepad and save with .crt extension

Now we will go to configure the Identity Sources on vCenter:

  • Login as Single Sign-On Administrator to vCenter
  • Navigate to Menu > Administration > Single Sign-On Configuration
  • In the Identity Provider tab, open Identity Sources
  • Click ADD
  • Select Active Directory over LDAP or OpenLDAP, depending on your directory type.

Fill out the remaining fields as follows:
Identity Source Name: Label
Base DN for users: The Distinguished Name (DN) of the starting point for directory server searches. Example: “DC=pollaio,DC=lan”.
Base DN for groups: The Distinguished Name (DN) of the starting point for directory server searches.
Domain name: Your domain name. Example: “pollaio.lan”
Domain alias: Your NetBIOS name. Example: “pollaio.lan”
Username: Domain user with at least browse privileges. Example: “pollaio\administrator”.
Connect to:  “ldaps://<DC FQDN>”.

  • Click Browse next to SSL Certificate
  • Select the .cer file created in before step
Now we are ready to login to the vCenter with domain user (remember to assign the correct permission to domain group or user group)

If you want check the correct use of SSL certificate on the authentication to Active Directory with LDAP connection check the websso.log:

LDAP Identity source and vCenter

VMware Skyline Advisor

 

VMware has had a product for a while now called VMware Skyline that provides proactive monitoring, analysis, and support for your VMware environment. It monitors your VMware installation and will notify you when issues arise.

Skyline Advisor will be available to customers and partners with active Production and Premier Support, VMware Success 360 and vRealize Cloud Universal subscriptions at no additional cost.

Create a Cloud Services Organization

Login with My VMware account associate to Production and Premier Support on the site:

https://console.cloud.vmware.com/
  •  After clicking Get Started, a new web browser page, or tab, will open. You will be asked to sign-in

with your VMware account. If you have an existing My VMware account, you can use those same

account details (email address/password) to sign in to Cloud Services.

  •  If you are existing VMware Cloud Services customer, you can choose an existing Cloud Services

Organization for Skyline. If you have never used VMware Cloud Services, click Create New

Organization.

  •  Enter a Organization Name.

Name your Organization something meaningful, that can be easily

referenced by both you, and VMware. For example, name your Organization after you Company, or

Business name. You can also append a line-of-business, division, or team, to the end of your

Company or Business name.

The following are example Organization Names:

The company, LOB, Company LOB, Company-vSphere, Company-Desktop

  • Enter an Address for your Organization.

Click Add Address. You can also choose an existing

address if one was found for your account. If you choose an existing address, skip to substep f.

During the creation of your Cloud Services Organization, your country currency, and Tax ID, may be

displayed. The displaying of this information is a construct of Cloud Services. Skyline is available at

no additional cost, and you will not be required to enter any payment details while adopting Skyline.

  • Select a Country from the drop-down menu.
  •  Enter your street address on Address Line 1, and Address Line 2 (optional).
  • Enter your City.
  • Enter your State/Province.
  • Enter your Zip/Postal Code.
  • Review the Cloud Services Terms of Service. Click the checkbox to agree to the Terms of Service.
  • Click Continue.

Now on service, we have Skyline Advisor, click on this service

Link the Entitlement

Now, after clicking on LINK, we have the correct status LINKED

 
Deploy Skyline Collector and configure the connection to Cloud Services
 
Now we download the Skyline collector
https://vmware.com/go/skyline-collector-download
 
Download VMware Skyline Collector 
Select Version: 
•roduct Downloads 
Product 
VMware Skyline is an innovative proactive support offering that brings high-performing technology and tools to the workbench to 
radically transform customer support 
Automatically and securely collect product usage data with VMware Skyline Collector. Environment-specific analytics based on 
configuration, feature, and performance data are executed against this data. The resulting information may radically improve 
Read More 
Product Resources 
View My Download History 
Product Info 
Documentation 
Community 
Drivers & Tools 
Open Source 
Custom ISOS 
VMware Skyline Collector 28.0 
OEM Addons 
Release Date 
2021-07-15 
GO TO DOWNLOADS
 
To deploying the Skyline Collector Virtual appliance on our vSphere infrastructure we have need:
 
Resource Requirements 
vCPU 
RAM 
Disk Space 
87Ga (1.1 thin- 
provisioned)
 
And these are the Network Requirements
 
External Network Requirements 
Connection From 
Skyline Collector 
Skyline Collector 
Connection To 
vcsa_vmwarecom 
vapp-updates_vmwarecom 
Type 
H TTPS 
H TTPS 
Protocol 
T CP/1p 
T CP/1p 
Protocol 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
443 
443 
Internal Network Requirements 
Connection From 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Web Browser 
Connection To 
vCenter Server 
ESXi Hypervisor Hosts 
5.5 psc/sso Provider 
6.0 and above psc/sso 
Provider 
NSX-V Manager 
NSX-T Manager nodes cluster 
virtual P address (VIP) 
Horizon Connection Server 
vRedlize Operations Manager 
SDDC Manager 
Skyline Collector 
Type 
443 
443 
443 
443 
443 
443 
443 
443 
443
 
 
 
 
Account permission for vCenter, we need to create a Domain Account to use to permit access to vCenter from skyline.
 
• vCenter Server Read-Only role 
• GlobalDiagnostics 
• Globalkealth 
• GlobalLicenses 
• GlobalSettings 
• Host profile.View
Let’go we are starting with the OVF Deploy
Deploy OVF Template 
1 Select an OVF template 
2 Select a name and folder 
3 Select a compute resource 
4 Review details 
5 Select storage 
6 Ready to complete 
Select an OVF template 
Select an OVF template trom remote URL or local file system 
Enter a URL to download and install the OVF package trom the Inn 
location accessible trom your computer, such as a local hard drive. 
CD/DVD drive. 
O URL 
@ Local file 
Scegli i file
 
 
 
 
Deploy OVF Template 
1 Select an OVF template 
2 Select a name and folder 
3 Select a compute resource 
4 Review details 
5 Select storage 
6 Ready to complete 
Select a compute resource 
Select the destination compute resource for 
v VDICEDB 
v VDlRPLcluster 
Phesx07.
 
 
 
 
 
 
Deploy OVF Template 
1 Select an OVF template 
2 Select a name and folder 
3 Select a compute resource 
4 Review details 
5 License agreements 
6 Select storage 
7 Select networks 
8 Customize template 
9 Ready to complete 
Template name 
Download size 
Size on disk 
Folder 
Resource 
Storage mapping 
All disks 
Network mapping 
Network I 
IP allocation settings 
IP protocol 
IP allocation 
Properties 
557.6 MB 
87.0 Ga 
austerVSAN 
Policy: Stretched Mirror - No space reservation No Local Protection; Datastore: vsanDatastore; Format: A 
s defined in the VM storage policy 
DPG-IT 
1 pv4 
Static - Manual 
Default Gateway = 
Domain Name = VISKYOI 
Domain Search Path 
Domain Name servers = 172 16 
Network 1 P Address = 172 16 20.33 
Network 1 Netmask = 255 
CANCEL 
BACK 
FINISH
 
 
 
 
You must change your password on first login. 
Old Password 
New Password 
Repeat New Password 
CHANGE 
@ Your password was changed successfully! 
LOGIN AGAIN 
Password Requirements 
• Minimum length: 8 characters 
• One uppercase letter. 
• One special case letter 
• One digit. 
• One lowercase letter.
 
Skyline Collector 
admin 
For Active Directory login username format must be "user@domain" 
LOG IN
 
 
 
After deployed the Connector virtual appliance we need to configure the Skyline Collector for communicating with Cloud Services
 
 
After you have finished Linking Entitlement Accounts to your Cloud Services Organization, click Skyline Collector Setup. 
O COLLECTOR SETUP
 
 
Initial Configuration 
1 Network Configuration 
CELP Terms and Conditions 
Collector Registration 
Continue Configuration 
Collector Name 
6 Auto-upgrade 
Configure vCenter Server 
8 Configure NSX-V (optional) 
Configure NSX-T (optional) 
10 Configure Harizor View (op-.iona , 
Configure *'Realize Operations 
Network Configuration 
Hostname Verification 
By enabling Hostname Verification you add an additional check 
This will ensure the client is connecting to the appropriate server 
"subjectAItName" fields against the host in the URL of the conne 
NOTE: Enabling this option will likely break any traffic intercept 
generating certificates correctly. 
Hostname Verification 
Proxying 
Proxy 
TEST & SAVE 
C) Enabled 
Disabled
 
Test the network configuration
@ Successfully set Network Configuration!
 
Initial Configuration 
1 Network Configuration 
2 CEIP Terms and Conditions 
Collector Registration 
Cortinue Configuration 
Collector Name 
6 Auto-upgrade 
Configure vCenter Server 
8 Configure NSX-V (optional) 
Configure NSX-T (optional) 
10 Configure Harizor View (optional 
Customer Experience Improvement Program (CEIP) 
As part of the Enhanced Customer Experience Improvement Program ("CEIP"), VMware Skyline collects 
certain technical data and product logs about your organization's use of VMware products and services 
on a regular basis. The data collected may include device identifiers and information that identifies your 
users. This data is collected to enable VMware to diagnose and improve its products and services, fix 
product issues, provide proactive technical support and to advise you on how best to deploy and use 
our products 
For additional information regarding the CEIP, please see the Trust & Assurance Center at 
https://wwwvmware.com/solutions/trustvmware/ceip_ 
By configuring your VMware products to participate in the Skyline service, each product will be enabled 
to send product usage data to the Skyline services as part of the Enhanced Customer Experience 
Improvement Program. You may add, remove or modify your product configurations at any time. For 
detailed instructions see our Skyline Collector User Guide 
To disable participation in the Customer Experience Improvement Program, you must Deregister this 
Skyline Collector. For instructions how to Deregister a Skyline Collector, see VMware Knowledge Base 
Article 74677.
 
And now we have to insert the token create on Cloud Service
 
Initial Configuration 
2 
3 
6 
8 
Network Configuration 
CEP Terms and Conditions 
Collector Registration 
Continue Configuration 
Collector Name 
Auto-upgrade 
Configure vCenter Server 
Configure NSX-V (optional) 
Configure NSX-T (optional) 
Collector Registration 
VMware Skyline is now part of VMware Cloud Services. VMware Cloud Services uses the concept of 
"Organizations" to provide controlled access to one or more services. 
You can create an "0rganization" on the VMware Cloud Services Platform by following the steps 
described in the link below: 
You can generate a token on VMware Cloud Services Platform and enter it below in order to associate 
your Skyline Collector to the Organization. 
Collector Registration Token 
Paste Collector Registration Token Here
 
Get Token from Vmware Cloud Services Platform

VMware Cloud Services

I have already configured my organization

Copy and paste the token on Skyline Connect and register it.

Add source Data to Skyline Advisor

After complete the step 5 and 6 (I suggest to enable the auto-upgrade), we can access to skyline collector to configure the connection to vCenter (or multiple vCenter)

Go to https://<mySkylineCollectorFQDN/

Select Add a vCenter  Server

Use the Account AD to whom assign the correct permission on vCenter

We are able to see the vSphere infrastructures if we access on Skyline Advisor Service on Cloud Service Link

https://console.cloud.vmware.com/

Now we will be waiting….. for data populate (72 hours for Findings)

After 72 hours we are able to see all info   (wow I see six Critical alerts :-))

VMware Skyline Frequently Asked Questions (55928)

Skyline Collector User Accounts and Permissions (vmware.com)

VMware Skyline Advisor

Capture Code – vSphere Web Client

One of the conveniences of administering VMware solutions is being able to use code to create scripts to perform repetitive tasks or automate processes

One of the vSphere Web Client features that can help those new to the PowerCli is the Capture Code, it basically allows you to list and save the Powercli commands of the actions you are doing with the vSphere Web Client.

To activate it just access the vSphere Web Client, from the Menu select Developer Center

Select Code Capture and enable it by placing the “Enable Code Capture” flag on the right (which turns green)

At this point, a space will appear in our frame where the commands will be listed with some operations, such as Clear and start another, Copy and Download

Where the Download option generates you the ps1 file with the Powercli commands of the recorded operations

To start and stop a recording session you can use the buttons:

Or the red button that appears at the top of the WebClient once “Enable Code Capture” is enabled

Bye

Capture Code – vSphere Web Client

VMware Skyline Health Diagnostics

Last Year VMware release a Diagnostic Tool for Analyze vCenter, vSAN and ESXi infrastructure.

VMware Skyline Health Diagnostic is a Virtual Appliance (based on Photon OS) for analyzing our vSphere infrastructure.

In this link major info for install:

Introducing VMware Skyline Health Diagnostic Tool

but what can it be useful for?

Imagine you have a vSAN infrastructure and at some point, a diskgroup fails but errors are reported on the disks. With this tool, we can analyze the logs exported from the ESXi host on which the diskgroup resides and identify on which physical disks I / O errors have been highlighted

VMware Skyline Health Diagnostics

vSphere and Certificates

vSphere use TLS Certificates for protect and security communication from vCenter to ESXi host and when the user access to vCenter WEB GUI.

There are many possible configurations:

  • Full Managed Mode -> All certificates are managed from VMCA
  • Hybrid Mode -> The communication certificates for traffic from vCenter to ESXi are managed from VMCA. The Admin user import from Private PKI only the SSL certificate for Access to WEB GUI
  • Subordinate CA Mode -> Configure the VMCA as a Subordinate CA of Private PKI
  • Full Custom Mode –> All Certificates are generated and managed from the local Private PKI

The best solution is Hybrid Mode for correct balance of Security and effort for implementation.

vSphere and Certificates

Horizon Cloud on Azure 1 di 5

Attualmente è disponibile la possibilità di utilizzare una  trial della soluziona Horizon Cloud di VMware su cloud AZURE. 

La trial è disponibile per 90 giorni (prima del covid-19 erano meno estesa). 
Per procedere nell’utilizzo della trial è necessario eseguire una serie di semplici step:
  1. Acquire Horizon Cloud Service on Microsoft Azure
  2. Prepare Microsoft Networking and AD
  3. Azure Deploy Horizon Cloud Service On Microsoft Azure 
  4. Deploy Microsoft Azure-Based RDS Host and VDI Desktop 
  5. Install Applications and Assign Users 
Procediamo con ordine e partiamo da Acquire Horizon Cloud Service on Microsft Azure  dove:
– Attiviamo le licenze Trial e gli step che consiglio sono:
  • Crearsi un account GMAIL 
  • Registrarsi al sito  MyVMware con il nuovo account GMAIL 
  • Compilare l’apposito FORM (Request Form-Horizon Universal License Free Trial*)

       Le credenziali create per accedere MyVMware saranno anche quelle che ci permetteranno l’accesso al portale https://cloud.horizon.vmware.com

Nell’apposito FORM verrà richiesto:
– Tipologia di trial (Azure, On-Premises, AWS, Azure VMware Solutions, Google Cloud VMware Engine)
– Numero e tipologia (Concurrent o user) delle licenze Horizon Universal 
– Location della VMware Horizon Panel (America, Europa ecc..)
– Azure Region del deploy dell’ambiente 
 Utilizziamo una Microsoft Azure Subscription 

 Dal portale azure attiviamo la nostra Subscription dove andremo a crare i servizi necessarei per il deploy dei componenti (Horizon, VM ecc)
*Horizon Universal License sono le nuove licenze Horizon che posso essere utilizzate sia per ambienti on-premises o su cloud
Horizon Cloud on Azure 1 di 5

vSphere Skyline Health

Uno degli aspetti che mi spingono a proporre di aggiornare le infrastrutture VMware vSphere alle versione 6.7u3 è anche la presenza dello skyline health. Strumento indispensabile per monitorare lo stato della infrastruttura

 

 

 

Le informazioni che possiamo velocemente individuare analizzare sono relative a oggetti come i singoli host ESXi, il vCenter e lo stato della vSAN.

Che informazioni possiamo trovarci? Dalle vulnerabilità di sicurezza  presenti nella nostra infrastruttura a versioni di firmware da aggiornare. Ovviamente correlate da collegamenti a pagine della KB VMware con informazioni specifiche su come  gestirle.  

 

 

Per poter avere queste informazioni è necessario abilitare il CEIP (Customer Experience Improvement Program).

 

 

VMware investe da sempre nella soddisfazione dei clienti e per cui tutto il flusso di gestione e assistenza con l’avvento di skyline ha guadagnato   ulteriormente in efficienza.

 

 

 

vSphere Skyline Health