Azure EventHub - PySpark Failed to configure SaslClientAuthenticator works when using Confluent cloud

[ozkary]

Hi,

A Python and CSharp producer/consumers are able to connect to Azure EventHub and send messages. The problem is when adding an application with Spark streaming (pyspark). All the configurations and dependencies look to be correct.

The Spark app runs successfully when using Confluent Cloud.

Confluent:

sasl.mechanisms=PLAIN
sasl.username=*****

Azure EventHub:

security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.username=$ConnectionString
sasl.password=Endpoint=*********

SASL:
sasl_config = f'org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="kafka" username="{settings["sasl.username"]}" password="{settings["sasl.password"]}";'

Bug Report:

The Spark (pyspark) application fails to authenticate with the Azure EventHub due to a Sasl process.

• Actual behavior
  The app is not able to authenticate. These errors are shown:

  • Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator

  • Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator

• Expected behavior

The application should be able to communicate with the eventhub.

• Spark version

3.3.2

• spark-eventhubs artifactId and version

com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.22

Regards