Ingest your VMware VCSA Appliance logs into Azure Sentinel

In an old post, I described how to send ESXi logs to Azure Log Analytics to ingest at Azure Sentinel, now I describe Step to Step how to send vCenter logs.

The first step is to do step by step this configuration:

After this we need to:
  • Change to the settings of VCSA Appliance to send the logs to Syslog Gateway Server
  • Configure the Log Analytics Agent, installed on Syslog Gateway Server to process the Facility Local0
  • Change la function VMwareESXi (It was created for ESXi Log check my old post) or create a query custom to parse the log on Azure Log Analytics

Change to the settings of VCSA Appliance to send the logs to Syslog Gateway Server

For configuring the VCSA you can use this VMware KB

Forward vCenter Server Log Files to Remote Syslog Server (vmware.com)

Protocol and port depend on your infrastructure configuration (you need to enable communication from VCSA to Syslog Gateway Server on the select TCP/UDP port)

and enable send events (it is enabled by default, but a check is a good idea)

Configure Streaming of Events to a Remote Syslog Server (vmware.com)

Now you can connect to the Syslog Gateway Server and check if the Syslog server received the logs from the VCSA Appliance

Use SSH to connect at the Syslog Gateway Server and use this command

cat /var/log/syslog | grep <fqdn vCenter> | more

in my situation

cat /var/log/syslog | grep vcenter | more

Configure the Log Analytics Agent, installed on Syslog Gateway Server to process the Facility Local0

Connect to Azure Portal and on Azure Log Analytics Service enable the correct facility (local0)

After 10/15 minutes the new configuration will be applied on Syslog Gateway Server (you can check the file /etc/rsyslog.d/95-omsagent.conf on Syslog Gateway)

Change the VMwareESXi function (It was created for ESXi Log check my old post) or create a query custom to parse the log on Azure Log Analytics

Finally, you can query the data on Azure Log Analytics

Syslog | where HostName contains “<FQDN vCenter>”

or optionally you can edit the function create for Ingest ESXi log (check my old POST) and insert the vCenter FQDN Name in the same position where there is the ESXi FQDN Name.

Currently, on Azure Sentinel there are no specific Workbooks for VMware, all queries are to be created

Ingest your VMware VCSA Appliance logs into Azure Sentinel

Ingest your VMware ESXi logs into Azure Sentinel

The VMware ESXi connector is currently in PREVIEW

What is Azure Sentinel?

Microsoft Azure Sentinel is a scalable, cloud-native, security information event management (SIEM) and security orchestration automated response (SOAR) solution. Azure Sentinel delivers intelligent security analytics and threat intelligence across the enterprise, providing a single solution for alert detection, threat visibility, proactive hunting, and threat response.

Azure Sentinel ingests data from services and apps by connecting to the service and forwarding the events and logs to Azure Sentinel. For physical and virtual machines, you can install the Log Analytics agent that collects the logs and forwards them to Azure Sentinel. For Firewalls and proxies, Azure Sentinel installs the Log Analytics agent on a Linux Syslog server, from which the agent collects the log files and forwards them to Azure Sentinel.

How connect  VMware ESXi to Azure Sentinel?

Integration between VMware ESXi and Azure Sentinel makes use of a Syslog server with the Log Analytics agent installed. It also uses a custom-built log parser based on a Kusto function.

For the onboarding of ESXi on Azure Sentinel, these are the step:

  • Have up and running a  Azure Sentinel service.
  • Prepare a Linux Syslog Server
  • Install Log Analytics Agent
  • Create the VMwareESXi Kusto function
  • Configure your ESXi Hosts to forward log to Syslog server

Create a Azure Sentinel Service 

This example is related to a basic configuration of the Azure Sentinel infrastructure, for more information and details for sizing and costs check in the respective guides from Microsoft.

Login to Azure Portal (How to get an Azure subscription?)

Prepare Linux Syslog

I have installed a virtual machine with Ubuntu Guest OS

I have checked if rsyslog is installed and running

if rsyslog is not installed run the following installation command

 apt-get install rsyslog

Configure rsyslog

Verify the tcp port used from syslog server

Cat  /etc/rsyslog.conf

Configure Kusto function alias

On log analytics workspace

 create this function:

/ Title:           VMWare ESXi
// Author:          Microsoft
// Version:         1.0
// Last Updated:    11/13/2020
// Comment:         Inital Release
//  
// DESCRIPTION:
// This parser takes raw VMWare ESXi logs from a Syslog stream and parses the logs into a normalized schema.
//
// USAGE:
// 1. Open Log Analytics/Azure Sentinel Logs blade. Copy the query below and paste into the Logs query window. 
// 2. In the query window, on the second line of the query, enter the hostname(s) of your VMWare ESXi device(s) and any other unique identifiers for the logstream. 
//    For example: | where Computer in ("server1", "server2")
// 3. Click the Save button above the query. A pane will appear on the right, select "as Function" from the drop down. Enter a Function Name.
//    It is recommended to name the Function Alias, as VMwareESXi
// 4. Kusto Functions can typically take up to 15 minutes to activate. You can then use Function Alias for other queries.
//
// REFERENCES: 
// Using functions in Azure monitor log queries: https://docs.microsoft.com/azure/azure-monitor/log-query/functions
// 
// LOG SAMPLES:
// This parser assumes the raw log are formatted as follows:
//
// info vpxa[D089B70] [Originator@6876 sub=vpxLro opID=HB-host-89929@3678594-5d55f348-40] [VpxLRO] -- BEGIN session[52908bc7-673e-dc2f-8726-70d13fe8ef72]521881cd-707e-cf9b-01c4-f0fd16d7444d -- vpxa -- vpxapi.VpxaService.retrieveChanges -- 52908bc7-673e-dc2f-8726-70d13fe8ef72
// warning hostd[191C2B70] [Originator@6876 sub=VigorStatsProvider(409264032)] AddVirtualMachine: VM '67' already registered
// cpu25:1040586)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1101: Could not select path for device "Unregistered Device".
// 
let LogHeader = Syslog
| where Computer in ("ESXiserver1", "ESXiserver2") // ESXiserver1 and ESXiserver2 are examples, replace this list with your ESXi devices
| extend Parser = extract_all(@"^(\w+)?\s?(\w+)\[(\w+)\]\s([\s\S]+)", dynamic([1,2,3,4]), SyslogMessage)
| mv-expand Parser
| extend Substring = tostring(Parser[3])
| project-away Parser;
LogHeader
| extend Sub = extract(@"sub=([\w\d\(\)\-\.]+)\]?",1, Substring),
	 OpId = extract(@"opID=([\w\d\(\)\-@]+)\s?\]?",1, Substring),
         UserName = extract(@"\suser=([\w\d\(\)\-]+)\]",1, Substring)
| extend Message = extract(@"\[([\S\s]+)\]\s([\S\s]+)",2, Substring)
| extend Message = iif(isempty(Message),SyslogMessage,Message)
| extend Message = trim(@"^-- ", Message)
| project-away Substring

Install Log Analytics Agent

Go to Vmware ESXi Connector on Azure Sentinel

Go to linux syslog server and paste it the code for onboard agent to sentinel

For troubleshooting

/opt/microsoft/omsagent/bin/troubleshooter

In my installation was missing :

And i have installed it

apt-get install gdb

If the installation is ok

now we set which logs the linux agent must send to our workspace

And add local4 e auth

automatically this information will be sent to our agent

Configure ESXi to send data to Linux Syslog Gateway (Where is installed the Log Analytics Agent)

We configure our esxi hosts to send logs to our linux syslog with this powercli script:

Connect-ViServer 
$vmHosts = Get-VMHost
$remoteSyslog = 'tcp://<linuxlogserver>'
$syslogport = '514'
# Show current config
$vmHosts | ForEach-Object {
    Write-Host $_.Name
    Get-VMHostSysLogServer -VMHost $_
}
# Set syslog config in hypervisors
$vmHosts | ForEach-Object {
    Write-Host $_.Name
    Set-VMHostSysLogServer -SysLogServer $remoteSyslog":"$syslogPort -VMHost $_
}
# Restart syslog and set the allow rules in the ESXi
$vmHosts | ForEach-Object {
    Write-Host $_.Name
    (Get-Esxcli -v2 -VMHost $_).system.syslog.reload.Invoke()
    (Get-Esxcli -v2 -VMHost $_).network.firewall.ruleset.set.Invoke(@{rulesetid='syslog'; enabled=$true})
    (Get-Esxcli -v2 -VMHost $_).network.firewall.refresh.Invoke()
}
# Show current config
$vmHosts | ForEach-Object {
    Write-Host $_.Name
    Get-VMHostSysLogServer -VMHost $_
}

Check if ESXi Sentinel Connector is UP

Query to view log

Ingest your VMware ESXi logs into Azure Sentinel