Troubleshooting: Connector Log Delivery Fails With KafkaProducerError (Local_Transport or Local_AllBrokersDown)

Version Affected: All Cloud IdPs
 
 

Overview

The SecureAuth Connector's automatic log delivery to SecureAuth fails, and the Connector's logs in C:\ProgramData\SecureAuth Corporation\SecureAuth Connector\logs repeatedly show errors such as:

KafkaProducerError, Code: Local_Transport, Reason: sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstrap: Connect to ipv4#<ip-address>:9092 failed: Unknown error
KafkaProducerError, Code: Local_AllBrokersDown, Reason: 1/1 brokers are down

The Connector stores its logs locally, but also forwards them automatically to SecureAuth over the network for troubleshooting purposes. These errors mean a firewall is blocking that outbound traffic, so the logs never reach SecureAuth.

Cause

The Connector forwards its logs to a Kafka service. It first contacts the bootstrap address pkc-4nym6.us-east-1.aws.confluent.cloud on port 9092. The bootstrap address returns the list of Kafka brokers, and the Connector then connects to each broker directly, by the broker's own hostname, on the same port: b0-pkc-4nym6.us-east-1.aws.confluent.cloud through b23-pkc-4nym6.us-east-1.aws.confluent.cloud (24 hostnames). All of these addresses can resolve to any of the AWS US-East-1 public IP addresses, and the specific IPs they resolve to can change over time.

The errors appear when any of these addresses is blocked. Allowing only the bootstrap address is not enough: the Connector still cannot reach the brokers, and the errors continue.

Resolution:

Allow outbound TCP traffic on port 9092 from every Connector server, using whichever method your firewall supports.

If your firewall can whitelist by DNS name, allow the bootstrap address pkc-4nym6.us-east-1.aws.confluent.cloud and all 24 broker hostnames, b0-pkc-4nym6.us-east-1.aws.confluent.cloud through b23-pkc-4nym6.us-east-1.aws.confluent.cloud.

If your firewall can only whitelist by IP address, retrieve the current list of AWS public IP ranges from https://ip-ranges.amazonaws.com/ip-ranges.json and whitelist the ranges for the us-east-1 region. Because the brokers are also hosted in AWS US-East-1, these ranges cover the broker addresses as well. To filter the list down to the us-east-1 region using PowerShell:

(Invoke-RestMethod -Uri "https://ip-ranges.amazonaws.com/ip-ranges.json").prefixes |
    Where-Object { $_.region -eq "us-east-1" } |
    Select-Object -ExpandProperty ip_prefix |
    Sort-Object -Unique

Once the firewall allows these addresses, the firewall logs show no more denied traffic to them, and the KafkaProducerError entries stop appearing in the Connector's logs.

SecureAuth Knowledge Base Articles provide information based on specific use cases and may not apply to all appliances or configurations. Be advised that these instructions could cause harm to the environment if not followed correctly or if they do not apply to the current use case.

Customers are responsible for their own due diligence prior to utilizing this information and agree that SecureAuth is not liable for any issues caused by misconfiguration directly or indirectly related to SecureAuth products.