vSAN Skyline Health History

VMware with vSAN 7.0u2 introduced an interesting functionality on Skyline for reviewing the state of vSAN in the past time

The functionality is disabled by default, if I want to see the Health History I need to only check the correct option.

After enable it I can see all history

and I can select the red icon for to see what happened

On Jan 8, 12:08 we encountered a problem on a vSAN cluster HOST

It is possible to select the day to check the vSAN status, just select the correct date and time

vSAN Skyline Health History

Log4j, Horizon Connection Server Workaround

We continue to look at how to mitigate the log4j vulnerability, in this post we look at horizon connection servers in detail.
As indicated by the VMware KB


only the connection servers where the HTML Access Portal is active are vulnerable. But all versions are subject to vulnerability.
I recommend applying the workaround even if the HTML Access Portal is not active.
Again as indicated in the previously cited KB we have two possibilities:

  • Change the following registry key

1. Edit this registry value:
HKLM\Software\VMware, Inc.\VMware VDM\plugins\wsnm\TomcatService\Params\JVMOptions
2. Append a single space character followed by this text: -Dlog4j2.formatMsgNoLookups=true
3. Exit the registry editor and restart the Connection Server service or reboot the machine

  • Run the following script as administrator.
@echo off
setlocal
goto start
__________________________________________________

CVE-2021-44228 - Prevent log4j parameter expansion
Horizon Connection Server 7.x, 8.x
VMware, Inc. 2021
__________________________________________________

:start
set sigpath=HKLM\Software\VMware, Inc.\VMware VDM\plugins\wsnm\TomcatService
for /f "delims=" %%g in ('reg.exe query "%sigpath%" /v Filename') do set sigval=%%g
if "%sigval%"=="" goto notneeded
set killflag=-Dlog4j2.formatMsgNoLookups=true
set svcpath=HKLM\Software\VMware, Inc.\VMware VDM\plugins\wsnm\TomcatService\Params
for /f "tokens=2*" %%v in ('reg.exe query "%svcpath%" /v JVMOptions') do set svcval=%%w
echo %svcval%|find " %killflag%" >nul
if not errorlevel 1 goto notneeded
reg add "%svcpath%" /v JVMOptions /d "%svcval% %killflag%" /f
net stop wsbroker /y && net start wsbroker
echo Completed.
goto :EOF

:notneeded
echo Not required.
goto :EOF

I will proceed with the script.
I create a fix-log4j.bat file in the c: \ temp folder of my connection server and copy the script text to it.


I launch the command from a PowerShell with administrator rights:

I reboot the server

I verify that the workaround is applied by relaunching the bat file.

Obviously, I have to do this on all the Horizon Connection Servers present
in the Horizon infrastructure

Log4j, Horizon Connection Server Workaround

VMware Workstation Player and Port Forwarding

To configure a PortForwarding on Windows 10 to a VM hosted on VMware Player it is necessary to proceed as follows:

  • Configure a static IP address (not essential but recommended) use the DHCP reservation function present in the virtualization application
  • Configure port forwarding.

Configure DHCP Reservation

  • Retrieve the MAC Address assigned to the virtual machine to which you want a static IP
  • Modify (with Notepad running in Administrator mode) the vmnetdhcp.conf file present in C:\ProgramData\VMware by inserting the following lines:

Host <Name of virtual network> {

hardware ethernet <Mac Address in this format xx:xx:xx:xx:xx:xx;

fixed-address <ip address>*;

}

Example:

#Static IP WIN11 –> Comment to identify the VM

VMnet8 host {
Hardware Ethernet 00: 0C: 29: 41: E8: 0C;
fixed address 192.168.233.10;
}

Where in our case the VMnet8 is the one assigned by default to the “NAT” configuration of the VM network card

  • Restart the VMNETDHCP service
    net stop vmnetdhcp
    net start vmnetdhcp

Port Forwarding Configuration

  • Modify (with Notepad running in Administrator mode) the vmnetnat.conf file present in C:\ProgramData\VMware by inserting the following lines:
    <tcpPortSource> = <IPaddress VM>:<tcpPortDestination>

Example:
8889 = 192.168.233.10:3389

In this case, we follow an RDP session to the OS system hosting my VM using: 8889 I will access through RDS to my VM with IP 192.168.233.10

*To check the IP range to always use the vmnetdhcp.conf file and identify the correct network segment; In the case of my example the segment is 8 (VMnet8)

# Virtual ethernet segment 8

# Added at 11/10/21 23:49:40

subnet 192.168.233.0 netmask 255.255.255.0 {

range 192.168.233.128 192.168.233.254;            # default allows up to 125 VM’s

option broadcast-address 192.168.233.255;

option domain-name-servers 192.168.233.2;

option domain-name “localdomain”;

option netbios-name-servers 192.168.233.2;

option routers 192.168.233.2;

default-lease-time 1800;

max-lease-time 7200;

}

VMware Workstation Player and Port Forwarding

LDAP Identity source and vCenter

Whenever we installed a new vCenter the activity always included integration with Active Directory and normally IWA (Integrated Windows Authentication) was used.
Since vSphere 7.0 version this possibility has been deprecated
so it is good to start with the integration of the vCenter with Active Directory via LDAP.
In our case, we will use LDAPS which uses a certificate

For first the step we need to create the certificate:

  • Use SSH to vCenter connection

On shell use this command

openssl s_client -connect <DC FQDN>:636 -showcerts

Copy the certificate output with  —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–

Past on Notepad and save with .crt extension

Now we will go to configure the Identity Sources on vCenter:

  • Login as Single Sign-On Administrator to vCenter
  • Navigate to Menu > Administration > Single Sign-On Configuration
  • In the Identity Provider tab, open Identity Sources
  • Click ADD
  • Select Active Directory over LDAP or OpenLDAP, depending on your directory type.

Fill out the remaining fields as follows:
Identity Source Name: Label
Base DN for users: The Distinguished Name (DN) of the starting point for directory server searches. Example: “DC=pollaio,DC=lan”.
Base DN for groups: The Distinguished Name (DN) of the starting point for directory server searches.
Domain name: Your domain name. Example: “pollaio.lan”
Domain alias: Your NetBIOS name. Example: “pollaio.lan”
Username: Domain user with at least browse privileges. Example: “pollaio\administrator”.
Connect to:  “ldaps://<DC FQDN>”.

  • Click Browse next to SSL Certificate
  • Select the .cer file created in before step
Now we are ready to login to the vCenter with domain user (remember to assign the correct permission to domain group or user group)

If you want check the correct use of SSL certificate on the authentication to Active Directory with LDAP connection check the websso.log:

LDAP Identity source and vCenter

VMware Skyline Advisor

 

VMware has had a product for a while now called VMware Skyline that provides proactive monitoring, analysis, and support for your VMware environment. It monitors your VMware installation and will notify you when issues arise.

Skyline Advisor will be available to customers and partners with active Production and Premier Support, VMware Success 360 and vRealize Cloud Universal subscriptions at no additional cost.

Create a Cloud Services Organization

Login with My VMware account associate to Production and Premier Support on the site:

https://console.cloud.vmware.com/
  •  After clicking Get Started, a new web browser page, or tab, will open. You will be asked to sign-in

with your VMware account. If you have an existing My VMware account, you can use those same

account details (email address/password) to sign in to Cloud Services.

  •  If you are existing VMware Cloud Services customer, you can choose an existing Cloud Services

Organization for Skyline. If you have never used VMware Cloud Services, click Create New

Organization.

  •  Enter a Organization Name.

Name your Organization something meaningful, that can be easily

referenced by both you, and VMware. For example, name your Organization after you Company, or

Business name. You can also append a line-of-business, division, or team, to the end of your

Company or Business name.

The following are example Organization Names:

The company, LOB, Company LOB, Company-vSphere, Company-Desktop

  • Enter an Address for your Organization.

Click Add Address. You can also choose an existing

address if one was found for your account. If you choose an existing address, skip to substep f.

During the creation of your Cloud Services Organization, your country currency, and Tax ID, may be

displayed. The displaying of this information is a construct of Cloud Services. Skyline is available at

no additional cost, and you will not be required to enter any payment details while adopting Skyline.

  • Select a Country from the drop-down menu.
  •  Enter your street address on Address Line 1, and Address Line 2 (optional).
  • Enter your City.
  • Enter your State/Province.
  • Enter your Zip/Postal Code.
  • Review the Cloud Services Terms of Service. Click the checkbox to agree to the Terms of Service.
  • Click Continue.

Now on service, we have Skyline Advisor, click on this service

Link the Entitlement

Now, after clicking on LINK, we have the correct status LINKED

 
Deploy Skyline Collector and configure the connection to Cloud Services
 
Now we download the Skyline collector
https://vmware.com/go/skyline-collector-download
 
Download VMware Skyline Collector 
Select Version: 
•roduct Downloads 
Product 
VMware Skyline is an innovative proactive support offering that brings high-performing technology and tools to the workbench to 
radically transform customer support 
Automatically and securely collect product usage data with VMware Skyline Collector. Environment-specific analytics based on 
configuration, feature, and performance data are executed against this data. The resulting information may radically improve 
Read More 
Product Resources 
View My Download History 
Product Info 
Documentation 
Community 
Drivers & Tools 
Open Source 
Custom ISOS 
VMware Skyline Collector 28.0 
OEM Addons 
Release Date 
2021-07-15 
GO TO DOWNLOADS
 
To deploying the Skyline Collector Virtual appliance on our vSphere infrastructure we have need:
 
Resource Requirements 
vCPU 
RAM 
Disk Space 
87Ga (1.1 thin- 
provisioned)
 
And these are the Network Requirements
 
External Network Requirements 
Connection From 
Skyline Collector 
Skyline Collector 
Connection To 
vcsa_vmwarecom 
vapp-updates_vmwarecom 
Type 
H TTPS 
H TTPS 
Protocol 
T CP/1p 
T CP/1p 
Protocol 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
TCP/Ip 
443 
443 
Internal Network Requirements 
Connection From 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Skyline Collector 
Web Browser 
Connection To 
vCenter Server 
ESXi Hypervisor Hosts 
5.5 psc/sso Provider 
6.0 and above psc/sso 
Provider 
NSX-V Manager 
NSX-T Manager nodes cluster 
virtual P address (VIP) 
Horizon Connection Server 
vRedlize Operations Manager 
SDDC Manager 
Skyline Collector 
Type 
443 
443 
443 
443 
443 
443 
443 
443 
443
 
 
 
 
Account permission for vCenter, we need to create a Domain Account to use to permit access to vCenter from skyline.
 
• vCenter Server Read-Only role 
• GlobalDiagnostics 
• Globalkealth 
• GlobalLicenses 
• GlobalSettings 
• Host profile.View
Let’go we are starting with the OVF Deploy
Deploy OVF Template 
1 Select an OVF template 
2 Select a name and folder 
3 Select a compute resource 
4 Review details 
5 Select storage 
6 Ready to complete 
Select an OVF template 
Select an OVF template trom remote URL or local file system 
Enter a URL to download and install the OVF package trom the Inn 
location accessible trom your computer, such as a local hard drive. 
CD/DVD drive. 
O URL 
@ Local file 
Scegli i file
 
 
 
 
Deploy OVF Template 
1 Select an OVF template 
2 Select a name and folder 
3 Select a compute resource 
4 Review details 
5 Select storage 
6 Ready to complete 
Select a compute resource 
Select the destination compute resource for 
v VDICEDB 
v VDlRPLcluster 
Phesx07.
 
 
 
 
 
 
Deploy OVF Template 
1 Select an OVF template 
2 Select a name and folder 
3 Select a compute resource 
4 Review details 
5 License agreements 
6 Select storage 
7 Select networks 
8 Customize template 
9 Ready to complete 
Template name 
Download size 
Size on disk 
Folder 
Resource 
Storage mapping 
All disks 
Network mapping 
Network I 
IP allocation settings 
IP protocol 
IP allocation 
Properties 
557.6 MB 
87.0 Ga 
austerVSAN 
Policy: Stretched Mirror - No space reservation No Local Protection; Datastore: vsanDatastore; Format: A 
s defined in the VM storage policy 
DPG-IT 
1 pv4 
Static - Manual 
Default Gateway = 
Domain Name = VISKYOI 
Domain Search Path 
Domain Name servers = 172 16 
Network 1 P Address = 172 16 20.33 
Network 1 Netmask = 255 
CANCEL 
BACK 
FINISH
 
 
 
 
You must change your password on first login. 
Old Password 
New Password 
Repeat New Password 
CHANGE 
@ Your password was changed successfully! 
LOGIN AGAIN 
Password Requirements 
• Minimum length: 8 characters 
• One uppercase letter. 
• One special case letter 
• One digit. 
• One lowercase letter.
 
Skyline Collector 
admin 
For Active Directory login username format must be "user@domain" 
LOG IN
 
 
 
After deployed the Connector virtual appliance we need to configure the Skyline Collector for communicating with Cloud Services
 
 
After you have finished Linking Entitlement Accounts to your Cloud Services Organization, click Skyline Collector Setup. 
O COLLECTOR SETUP
 
 
Initial Configuration 
1 Network Configuration 
CELP Terms and Conditions 
Collector Registration 
Continue Configuration 
Collector Name 
6 Auto-upgrade 
Configure vCenter Server 
8 Configure NSX-V (optional) 
Configure NSX-T (optional) 
10 Configure Harizor View (op-.iona , 
Configure *'Realize Operations 
Network Configuration 
Hostname Verification 
By enabling Hostname Verification you add an additional check 
This will ensure the client is connecting to the appropriate server 
"subjectAItName" fields against the host in the URL of the conne 
NOTE: Enabling this option will likely break any traffic intercept 
generating certificates correctly. 
Hostname Verification 
Proxying 
Proxy 
TEST & SAVE 
C) Enabled 
Disabled
 
Test the network configuration
@ Successfully set Network Configuration!
 
Initial Configuration 
1 Network Configuration 
2 CEIP Terms and Conditions 
Collector Registration 
Cortinue Configuration 
Collector Name 
6 Auto-upgrade 
Configure vCenter Server 
8 Configure NSX-V (optional) 
Configure NSX-T (optional) 
10 Configure Harizor View (optional 
Customer Experience Improvement Program (CEIP) 
As part of the Enhanced Customer Experience Improvement Program ("CEIP"), VMware Skyline collects 
certain technical data and product logs about your organization's use of VMware products and services 
on a regular basis. The data collected may include device identifiers and information that identifies your 
users. This data is collected to enable VMware to diagnose and improve its products and services, fix 
product issues, provide proactive technical support and to advise you on how best to deploy and use 
our products 
For additional information regarding the CEIP, please see the Trust & Assurance Center at 
https://wwwvmware.com/solutions/trustvmware/ceip_ 
By configuring your VMware products to participate in the Skyline service, each product will be enabled 
to send product usage data to the Skyline services as part of the Enhanced Customer Experience 
Improvement Program. You may add, remove or modify your product configurations at any time. For 
detailed instructions see our Skyline Collector User Guide 
To disable participation in the Customer Experience Improvement Program, you must Deregister this 
Skyline Collector. For instructions how to Deregister a Skyline Collector, see VMware Knowledge Base 
Article 74677.
 
And now we have to insert the token create on Cloud Service
 
Initial Configuration 
2 
3 
6 
8 
Network Configuration 
CEP Terms and Conditions 
Collector Registration 
Continue Configuration 
Collector Name 
Auto-upgrade 
Configure vCenter Server 
Configure NSX-V (optional) 
Configure NSX-T (optional) 
Collector Registration 
VMware Skyline is now part of VMware Cloud Services. VMware Cloud Services uses the concept of 
"Organizations" to provide controlled access to one or more services. 
You can create an "0rganization" on the VMware Cloud Services Platform by following the steps 
described in the link below: 
You can generate a token on VMware Cloud Services Platform and enter it below in order to associate 
your Skyline Collector to the Organization. 
Collector Registration Token 
Paste Collector Registration Token Here
 
Get Token from Vmware Cloud Services Platform

VMware Cloud Services

I have already configured my organization

Copy and paste the token on Skyline Connect and register it.

Add source Data to Skyline Advisor

After complete the step 5 and 6 (I suggest to enable the auto-upgrade), we can access to skyline collector to configure the connection to vCenter (or multiple vCenter)

Go to https://<mySkylineCollectorFQDN/

Select Add a vCenter  Server

Use the Account AD to whom assign the correct permission on vCenter

We are able to see the vSphere infrastructures if we access on Skyline Advisor Service on Cloud Service Link

https://console.cloud.vmware.com/

Now we will be waiting….. for data populate (72 hours for Findings)

After 72 hours we are able to see all info   (wow I see six Critical alerts :-))

VMware Skyline Frequently Asked Questions (55928)

Skyline Collector User Accounts and Permissions (vmware.com)

VMware Skyline Advisor

VMware vSAN 7 Update 3 New Features

VMware vSAN 7 Update 3 New Features

With the release of VMware vSphere 7 Update 3, this also means there is a new version of vSAN as well. VMware vSAN has been trailblazing in the world of HCI for several years now and with over 30,000 customers and many releases behind it, the solution has certainly grown, matured, and become […]


VMware Social Media Advocacy

VMware vSAN 7 Update 3 New Features

Capture Code – vSphere Web Client

One of the conveniences of administering VMware solutions is being able to use code to create scripts to perform repetitive tasks or automate processes

One of the vSphere Web Client features that can help those new to the PowerCli is the Capture Code, it basically allows you to list and save the Powercli commands of the actions you are doing with the vSphere Web Client.

To activate it just access the vSphere Web Client, from the Menu select Developer Center

Select Code Capture and enable it by placing the “Enable Code Capture” flag on the right (which turns green)

At this point, a space will appear in our frame where the commands will be listed with some operations, such as Clear and start another, Copy and Download

Where the Download option generates you the ps1 file with the Powercli commands of the recorded operations

To start and stop a recording session you can use the buttons:

Or the red button that appears at the top of the WebClient once “Enable Code Capture” is enabled

Bye

Capture Code – vSphere Web Client

Ingest your VMware VCSA Appliance logs into Azure Sentinel

In an old post, I described how to send ESXi logs to Azure Log Analytics to ingest at Azure Sentinel, now I describe Step to Step how to send vCenter logs.

The first step is to do step by step this configuration:

After this we need to:
  • Change to the settings of VCSA Appliance to send the logs to Syslog Gateway Server
  • Configure the Log Analytics Agent, installed on Syslog Gateway Server to process the Facility Local0
  • Change la function VMwareESXi (It was created for ESXi Log check my old post) or create a query custom to parse the log on Azure Log Analytics

Change to the settings of VCSA Appliance to send the logs to Syslog Gateway Server

For configuring the VCSA you can use this VMware KB

Forward vCenter Server Log Files to Remote Syslog Server (vmware.com)

Protocol and port depend on your infrastructure configuration (you need to enable communication from VCSA to Syslog Gateway Server on the select TCP/UDP port)

and enable send events (it is enabled by default, but a check is a good idea)

Configure Streaming of Events to a Remote Syslog Server (vmware.com)

Now you can connect to the Syslog Gateway Server and check if the Syslog server received the logs from the VCSA Appliance

Use SSH to connect at the Syslog Gateway Server and use this command

cat /var/log/syslog | grep <fqdn vCenter> | more

in my situation

cat /var/log/syslog | grep vcenter | more

Configure the Log Analytics Agent, installed on Syslog Gateway Server to process the Facility Local0

Connect to Azure Portal and on Azure Log Analytics Service enable the correct facility (local0)

After 10/15 minutes the new configuration will be applied on Syslog Gateway Server (you can check the file /etc/rsyslog.d/95-omsagent.conf on Syslog Gateway)

Change the VMwareESXi function (It was created for ESXi Log check my old post) or create a query custom to parse the log on Azure Log Analytics

Finally, you can query the data on Azure Log Analytics

Syslog | where HostName contains “<FQDN vCenter>”

or optionally you can edit the function create for Ingest ESXi log (check my old POST) and insert the vCenter FQDN Name in the same position where there is the ESXi FQDN Name.

Currently, on Azure Sentinel there are no specific Workbooks for VMware, all queries are to be created

Ingest your VMware VCSA Appliance logs into Azure Sentinel

Check HW VMware Compatibility Matrix

I need to check the Compatibility Matrix for the network IO device of ESXi HOST

Connect to ESXi with SSH and start this command

vmkchdev -l | grep vmnic

and the value are:

If you want check storage IO device change vmnic to vmhba

Determining Network/Storage firmware and driver version in ESXi (1027206) (vmware.com)

Check HW VMware Compatibility Matrix