Troubleshooting: SecureAuth Audit Logs Do Not Reach Cortex XDR Through the XDR Collector

Version Affected: All
 
 

Overview

The Palo Alto Networks Cortex Extended Detection and Response (XDR) Collector is running on the SecureAuth Identity Provider (IdP) server, but SecureAuth audit log records do not appear in Cortex XDR. The collector reports that it is healthy, and no error is logged.

There are four possible causes. Identify which one applies to you before reading further:

• See Cause 1 - The log path in the collector profile contains a wildcard, and the capitalization of the folder or file names on disk does not match the capitalization in the pattern
• See Cause 2 - A change was made, but it never reached the collector, or it was made directly in the Filebeat configuration file and was overwritten
• See Cause 3 - The input skips files because an age limit (ignore_older) is set, and the files have not been written to recently enough
• See Cause 4 - Records reach Cortex XDR, but the text is unreadable or padded with null characters

These causes are not related, so a fix for one cause will not resolve the others.

 

In this article

• Cause 1: Wildcard in the log path does not match because of capitalization
• Cause 2: Configuration change never reached the collector or was overwritten
• Cause 3: The age limit excludes every file
• Cause 4: Log file encoding does not match the input

Cause 1: Wildcard in the log path does not match because of capitalization

The XDR Collector runs Filebeat, which reads the log files named in the paths list of its input. Windows resolves a folder or file name written out in full without regard to capitalization, but Filebeat matches every part of the path from the first wildcard onward with a case-sensitive comparison. A pattern such as *.log therefore never matches a file named Audit.Log, and a pattern such as Company* never matches a folder named CompanyRealm1.

A pattern that matches no files is not an error. Filebeat logs no warning, and the input stays idle while the collector reports itself as healthy. Because Filebeat runs as a child process of the xdrcollectorsvc service and not as a Windows service, Get-Service *filebeat* returns nothing. This is normal. Check that the process is running with this command:

Get-Process filebeat

To confirm this cause, open the newest Filebeat log in C:\ProgramData\XDR Collector\Logs, which is a file named filebeat-<date>-<number>.ndjson. It contains periodic monitoring metric snapshots. If the field harvester.started is missing from the snapshot and pipeline.clients is 0, Filebeat has never opened a file. The field harvester.started is left out entirely when its value is zero. The field registrar.states.current staying at zero is normal for the filestream input and is not a sign of a fault.

To check how a pattern behaves, compare the real names against the pattern with a case-sensitive test. The -clike operator is case-sensitive, so it reproduces what Filebeat does:

(Get-ChildItem '<log-folder>' -File).Name | Where-Object { $_ -clike '*.log' }

If the command returns nothing but the folder contains files with a different capitalization such as Audit.Log, this is the cause.

Resolution 1:

To resolve this:

1. In the Cortex XDR console, open the XDR Collector profile that is assigned to the IdP server, and find the Filebeat input that lists the SecureAuth audit log path.
2. Compare every segment of the path that contains a wildcard with the real folder and file names, character for character, including capitalization. Segments written out in full do not need to match in case.
3. Replace each wildcard segment whose capitalization you cannot rely on with a bare *. For example, use <log-folder>\* in place of <log-folder>\*.log. Make sure the folder contains only the files that you want to collect before you do this.
4. Save the profile and distribute it to the collector.
5. Open C:\ProgramData\XDR Collector\Logs\scouter.log and confirm that a line beginning Policy filebeat has changed, new policy: appears with a timestamp later than your change.
6. Wait a minute, then open the newest Filebeat log again. Confirm that harvester.started is now present in the metrics snapshot and that pipeline.clients is greater than 0.

Cause 2: Configuration change never reached the collector or was overwritten

The Filebeat configuration file, C:\ProgramData\XDR Collector\Data\content\filebeat-windows-x86_64\filebeat.yml, is not the source of truth. The XDR Collector agent writes it from the XDR Collector profile it holds in a local cache. It rewrites the whole file every time the service starts and every time a policy change arrives, so any edit made directly to the file is lost without a warning in either log. Restarting the service re-applies the cached policy and does not fetch a newer one, so a restart does not apply a change made in the console. A change in the console reaches the collector only after the profile is distributed and the agent checks in.

To confirm this cause, open C:\ProgramData\XDR Collector\Logs\scouter.log and search for Policy filebeat has changed, new policy:. If there is no such line with a timestamp later than your change, the change did not arrive.

Resolution 2:

To resolve this:

1. Make every configuration change in the XDR Collector profile in the Cortex XDR console. Do not edit filebeat.yml on the host.
2. Confirm that the profile is assigned to this collector.
3. Distribute the profile.
4. Open scouter.log and confirm that a Policy filebeat has changed, new policy: line appears with a timestamp later than your change. The line contains the configuration that the agent is about to write, so you can compare it against the file.
5. If the line does not appear, check the profile assignment and distribution again. Do not restart the service in the expectation that it will fetch the change.

Cause 3: The age limit excludes every file

The ignore_older setting on a Filebeat input tells Filebeat to skip any file that has not been modified within the stated period. Filebeat skips those files silently. If the SecureAuth audit log files have not been written to within that period, none of them are collected. The symptoms are the same as in Cause 1: no error, a healthy collector, and no harvesters started. When ignore_older is not set, the limit is disabled.

Resolution 3:

To resolve this:

1. In the Cortex XDR console, open the XDR Collector profile assigned to the IdP server and find the Filebeat input for the SecureAuth audit logs.
2. Check whether ignore_older is set. If it is, either remove it or increase it so that it is longer than the longest expected period between writes to the audit log.
3. Save the profile and distribute it to the collector.
4. Open the newest Filebeat log in C:\ProgramData\XDR Collector\Logs and confirm that harvester.started is now present in the metrics snapshot.

Cause 4: Log file encoding does not match the input

By default, a Filebeat input reads each line as plain text in UTF-8 encoding. If a log file is written in UTF-16 little-endian (UTF-16LE) encoding, Filebeat still reads it and the metrics look healthy, but the records contain null characters between letters and cannot be read or parsed. Plain-text SecureAuth audit logs decode correctly with the default encoding, so this cause applies only if the file is genuinely UTF-16LE.

To confirm this cause, display the first bytes of the log file:

Format-Hex -Path '<log-file>' -Count 32

If the first two bytes are FF FE, the file is UTF-16LE. If they are not, leave the encoding at its default and look at the other causes.

Resolution 4:

To resolve this:

1. In the Cortex XDR console, open the XDR Collector profile assigned to the IdP server and find the Filebeat input for the SecureAuth audit logs.
2. Add the setting encoding: utf-16le to the input.
3. Save the profile and distribute it to the collector.
4. Check a new record in Cortex XDR and confirm that the text is readable and contains no null characters.

SecureAuth Knowledge Base Articles provide information based on specific use cases and may not apply to all appliances or configurations. Be advised that these instructions could cause harm to the environment if not followed correctly or if they do not apply to the current use case.

Customers are responsible for their own due diligence prior to utilizing this information and agree that SecureAuth is not liable for any issues caused by misconfiguration directly or indirectly related to SecureAuth products.