Log4j, Horizon Connection Server Workaround

We continue to look at how to mitigate the log4j vulnerability, in this post we look at horizon connection servers in detail.
As indicated by the VMware KB


only the connection servers where the HTML Access Portal is active are vulnerable. But all versions are subject to vulnerability.
I recommend applying the workaround even if the HTML Access Portal is not active.
Again as indicated in the previously cited KB we have two possibilities:

  • Change the following registry key

1. Edit this registry value:
HKLM\Software\VMware, Inc.\VMware VDM\plugins\wsnm\TomcatService\Params\JVMOptions
2. Append a single space character followed by this text: -Dlog4j2.formatMsgNoLookups=true
3. Exit the registry editor and restart the Connection Server service or reboot the machine

  • Run the following script as administrator.
@echo off
setlocal
goto start
__________________________________________________

CVE-2021-44228 - Prevent log4j parameter expansion
Horizon Connection Server 7.x, 8.x
VMware, Inc. 2021
__________________________________________________

:start
set sigpath=HKLM\Software\VMware, Inc.\VMware VDM\plugins\wsnm\TomcatService
for /f "delims=" %%g in ('reg.exe query "%sigpath%" /v Filename') do set sigval=%%g
if "%sigval%"=="" goto notneeded
set killflag=-Dlog4j2.formatMsgNoLookups=true
set svcpath=HKLM\Software\VMware, Inc.\VMware VDM\plugins\wsnm\TomcatService\Params
for /f "tokens=2*" %%v in ('reg.exe query "%svcpath%" /v JVMOptions') do set svcval=%%w
echo %svcval%|find " %killflag%" >nul
if not errorlevel 1 goto notneeded
reg add "%svcpath%" /v JVMOptions /d "%svcval% %killflag%" /f
net stop wsbroker /y && net start wsbroker
echo Completed.
goto :EOF

:notneeded
echo Not required.
goto :EOF

I will proceed with the script.
I create a fix-log4j.bat file in the c: \ temp folder of my connection server and copy the script text to it.


I launch the command from a PowerShell with administrator rights:

I reboot the server

I verify that the workaround is applied by relaunching the bat file.

Obviously, I have to do this on all the Horizon Connection Servers present
in the Horizon infrastructure

Log4j, Horizon Connection Server Workaround

Automate workaround for mitigating Log4j exploit on VCSA (vCenter Virtual Appliance)

After the post where I apply the VMware workaround for mitigating the Log4j exploit on the UAG appliance, now I suggest using this VMware KB to apply the workaround on vCenter.

Python script to automate the workaround steps of VMSA-2021-0028 vulnerability on vCenter Server Appliance (87088) (vmware.com)

The Python script attached at the KB check the vCenter version (6.5, 6.7 and 7) and apply the correct workaround indicate from VMware [see the Workaround instructions to address CVE-2021-44228 in vCenter Server and vCenter Cloud Gateway (87081) (vmware.com)]

Automate workaround for mitigating Log4j exploit on VCSA (vCenter Virtual Appliance)

Exploit Log4j mitigate on VMware Unified Access Gateway

## UPDATE 20/12/2021 ##

On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believed the previous instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers, we must assume the earlier workaround may not adequately address all attack vectors. 

We need to run a script:

#!/bin/bash

# Log contents to file by prefixing timestamp. Maximum file size is 50MB
function log_to_console() {
    echo "$(date +'%Y-%m-%d %T')" "$HOSTNAME" "$@"
}

log_to_console "Running script to remove JndiLookup.class from jars in Unified Access Gateway"

log_to_console "UAG Version: " $(tail -1 /opt/vmware/gateway/logs/version.info 2>/dev/null)

mkdir /tmp/test
mkdir /tmp/bkp

log_to_console "Unpacking archive and removing JndiLookup.class"
cp /opt/vmware/gateway/lib/ab-frontend-0.2.jar /tmp/bkp

unzip -q -o /opt/vmware/gateway/lib/ab-frontend-0.2.jar -d /tmp/test

unzip -q -o /tmp/test/hc.war -d /tmp/test/hc

zip -dq /tmp/test/hc/WEB-INF/lib/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

rm /tmp/test/hc.war
cd /tmp/test/hc

zip -r -q ../hc.war .

cd ..
rm -rf hc

log_to_console "Repackaging archive"

zip -r -q ab-frontend-0.2.jar .

chown gateway:users ab-frontend-0.2.jar
mv ab-frontend-0.2.jar /opt/vmware/gateway/lib

log_to_console "Replaced updated ab-frontend-0.2.jar, now looking for jndi in other places"

find / -type f \( -name "*.jar" -o -name *.war \) -exec sh -c "zipinfo -1 {} 2>/dev/null | grep 'JndiLookup.class' && echo {}" \; | grep .jar | while read -r line ; do
    jar_path=$line
    log_to_console "Updating $jar_path"
    zip -dq $jar_path org/apache/logging/log4j/core/lookup/JndiLookup.class
    chown gateway:users $jar_path
done


log_to_console "Restarting authbroker"
supervisorctl restart authbroker

log_to_console "Cleaning up."
cd /tmp
rm -rf /tmp/test

log_to_console "Verification: We are good if no jars are listed below"
find / -type f \( -name "*.jar" -o -name *.war \) -exec sh -c "zipinfo -1 {} 2>/dev/null | grep 'JndiLookup.class' && echo {}" \;

log_to_console "Verification: Grep authbroker-std-out.log for log4j errors, we are good if no exception is displayed below"
cat /opt/vmware/gateway/logs/authbroker-std-out.log | grep log4j

log_to_console "Done!"

Well we need to connect to UAG and create a uag_rm_log4j_jndilookup.sh file

vi uag_rm_log4j_jndilookup.sh

copy into the file the code, and enable it for execution

chmod +x uag_rm_log4j_jndilookup.sh

running the script

./uag_rm_log4j_jndilookup.sh

now if the UAG version is between 2009 and 2111 it is also necessary to set the -Dlog4j2.formatMsgNoLookups=true option on the authbroker service with the following commands. Note the space between “s/java /java”  and a space after “true /” in the command, these are important to ensure the command works correctly and doesn’t attempt to modify the wrong lines in the configuration file.

sed -i ‘s/java /java -Dlog4j2.formatMsgNoLookups=true /’ /opt/vmware/gateway/supervisor/conf/authbroker.ini

and restart the supervisorctl

supervisorctl update

more info in this VMware KB

Mitigation instructions to address CVE-2021-44228 and CVE-2021-45046 in VMware Unified Access Gateway (UAG) (87092)

#########################

In the middle of December month, we found a “little exploit”……

Ok it is not a joke, for mitigate on UAG (Unified Access Gateway that is a Security Server exposed on the Internet for remote access at Horizon infrastructure) it is necessary (To apply the workaround for CVE-2021-44228 to Unified Access Gateway version 2009 through to 2111):

  • Connect to UAG server with SSH Session

Check if SSH is enabled on UAG server to accept root connection.

Connect from WEB console or VMware Remote Console to UAG virtual appliance and modify in /etc/ssh/sshd_config the following line (for modify use vi commands):

PermitRootLogin no

to

PermitRootLogin yes

Save the file

Restart SSHD service with this command:

service sshd restart

now you are able to create an SSH connection to UAG server, REMEMBER TO DISABLE SSH CONNECTION FOR ROOT USER WITH ROLLBACK THE SETTING INTO SSHD_CONFIG FILE

  • Append the fix -Dlog4j2.formatMsgNoLookups=true

Type this command:

sed -i ‘s/java /java -Dlog4j2.formatMsgNoLookups=true /’ /opt/vmware/gateway/supervisor/conf/authbroker.ini

Reload the service

supervisorctl update

Check if the fix is applied with this command:

ps -ef | grep ab-frontend

the output of the command should need:

root@viuag03 [ ~ ]# ps -ef | grep ab-frontend
gateway 2799 849 99 09:51 ? 00:00:12 /usr/lib/jvm/zre-8/bin/java -Dlog4j2.formatMsgNoLookups=true -Dfile.encoding=UTF8 -Dport=8877 -Dlog4j.configuration=file:/opt/vmware/gateway/conf/log4j-authbroker.properties -Dspring.profiles.active=accesspoint -jar /opt/vmware/gateway/lib/ab-frontend-0.2.jar
root 2817 2622 0 09:51 pts/0 00:00:00 grep –color=auto ab-frontend

Ref: Workaround instructions to address CVE-2021-44228 in VMware Unified Access Gateway (87092)

Exploit Log4j mitigate on VMware Unified Access Gateway

VMware, CVE-2021-44228 and log4j (version 2)

Since the end of last week, a new critical vulnerability has spread, present in many programs (Many VMware applications use this Java Logging including vCenter, Horizon, etc.)

“an exploit in the popular Java logging library log4j (version 2) was discovered that results in Remote Code Execution (RCE) by logging a certain string.” as indicated in this link:

https://www.lunasec.io/docs/blog/log4j-zero-day/

After a few hours, the CVE also released its bulletin (CVE-2021-44228).

https://cve.mitre.org/cgi-bin/cvename.cgi?name=2021-44228

On the day of 10/12/2021, VMware released its security advisor (VMSA-2021-0028) with the workarounds to limit the vulnerability pending the release of patched versions to fix it:

https://www.vmware.com/security/advisories/VMSA-2021-0028.html

So good application everyone!

By the way, for the uninitiated:

What is CVE? CVE, short for Common Vulnerabilities and Exposures, is a list of publicly disclosed computer security flaws. When someone refers to a CVE, they mean a security flaw that’s been assigned a CVE ID number.
Security advisories issued by vendors and researchers almost always mention at least one CVE ID. CVEs help IT professionals coordinate their efforts to prioritize and address these vulnerabilities to make computer systems more secure.

VMware, CVE-2021-44228 and log4j (version 2)

Copy-VMGuestFile

I needed to copy a folder to virtual machine on the DMZ network segment.
The firewall rule blocks any access to VM, well I used the Copy-VMGuestFile Powercli Command.

Connect-VIServer <vcenter server FQDN or IP>

$vm = Get-VM -Name <VM target>

Get-Item “<Source Path>” | Copy-VMGuestFile -Destination “<Destination Path on VM>” -VM $vm -LocalToGuest -GuestUser <User VM Guest> -GuestPassword <Password User VM Guest>

After the first tentative I receive this error:

Copy-VMGuestFile : 09/12/2021 11:08:40 Copy-VMGuestFile The request was aborted: The request was cancelled.
At line:1 char:27

Probably it is a time out error and I try to change the WebOperation Timeout Seconds.

PS C:\Windows\system32> Set-PowerCLIConfiguration -WebOperationTimeoutSeconds -1

Scope ProxyPolicy DefaultVIServerMode InvalidCertificateAction DisplayDeprecationWarnings WebOperationTimeout
Seconds
—– ———– ——————- ———————— ————————– ——————-
Session UseSystemProxy Multiple Unset True -1
User Multiple
AllUsers -1

After the change the error it is resolved

Copy-VMGuestFile

VMware Workstation Player and Port Forwarding

To configure a PortForwarding on Windows 10 to a VM hosted on VMware Player it is necessary to proceed as follows:

  • Configure a static IP address (not essential but recommended) use the DHCP reservation function present in the virtualization application
  • Configure port forwarding.

Configure DHCP Reservation

  • Retrieve the MAC Address assigned to the virtual machine to which you want a static IP
  • Modify (with Notepad running in Administrator mode) the vmnetdhcp.conf file present in C:\ProgramData\VMware by inserting the following lines:

Host <Name of virtual network> {

hardware ethernet <Mac Address in this format xx:xx:xx:xx:xx:xx;

fixed-address <ip address>*;

}

Example:

#Static IP WIN11 –> Comment to identify the VM

VMnet8 host {
Hardware Ethernet 00: 0C: 29: 41: E8: 0C;
fixed address 192.168.233.10;
}

Where in our case the VMnet8 is the one assigned by default to the “NAT” configuration of the VM network card

  • Restart the VMNETDHCP service
    net stop vmnetdhcp
    net start vmnetdhcp

Port Forwarding Configuration

  • Modify (with Notepad running in Administrator mode) the vmnetnat.conf file present in C:\ProgramData\VMware by inserting the following lines:
    <tcpPortSource> = <IPaddress VM>:<tcpPortDestination>

Example:
8889 = 192.168.233.10:3389

In this case, we follow an RDP session to the OS system hosting my VM using: 8889 I will access through RDS to my VM with IP 192.168.233.10

*To check the IP range to always use the vmnetdhcp.conf file and identify the correct network segment; In the case of my example the segment is 8 (VMnet8)

# Virtual ethernet segment 8

# Added at 11/10/21 23:49:40

subnet 192.168.233.0 netmask 255.255.255.0 {

range 192.168.233.128 192.168.233.254;            # default allows up to 125 VM’s

option broadcast-address 192.168.233.255;

option domain-name-servers 192.168.233.2;

option domain-name “localdomain”;

option netbios-name-servers 192.168.233.2;

option routers 192.168.233.2;

default-lease-time 1800;

max-lease-time 7200;

}

VMware Workstation Player and Port Forwarding

LDAP Identity source and vCenter

Whenever we installed a new vCenter the activity always included integration with Active Directory and normally IWA (Integrated Windows Authentication) was used.
Since vSphere 7.0 version this possibility has been deprecated
so it is good to start with the integration of the vCenter with Active Directory via LDAP.
In our case, we will use LDAPS which uses a certificate

For first the step we need to create the certificate:

  • Use SSH to vCenter connection

On shell use this command

openssl s_client -connect <DC FQDN>:636 -showcerts

Copy the certificate output with  —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–

Past on Notepad and save with .crt extension

Now we will go to configure the Identity Sources on vCenter:

  • Login as Single Sign-On Administrator to vCenter
  • Navigate to Menu > Administration > Single Sign-On Configuration
  • In the Identity Provider tab, open Identity Sources
  • Click ADD
  • Select Active Directory over LDAP or OpenLDAP, depending on your directory type.

Fill out the remaining fields as follows:
Identity Source Name: Label
Base DN for users: The Distinguished Name (DN) of the starting point for directory server searches. Example: “DC=pollaio,DC=lan”.
Base DN for groups: The Distinguished Name (DN) of the starting point for directory server searches.
Domain name: Your domain name. Example: “pollaio.lan”
Domain alias: Your NetBIOS name. Example: “pollaio.lan”
Username: Domain user with at least browse privileges. Example: “pollaio\administrator”.
Connect to:  “ldaps://<DC FQDN>”.

  • Click Browse next to SSL Certificate
  • Select the .cer file created in before step
Now we are ready to login to the vCenter with domain user (remember to assign the correct permission to domain group or user group)

If you want check the correct use of SSL certificate on the authentication to Active Directory with LDAP connection check the websso.log:

LDAP Identity source and vCenter