AAD Authentication is terminated after running for a couple of minutes

[vaiyarUp]

Thanks for filing an issue with us! Below are some guidelines when filing an issue. In general, the more detail the better!
Bug Report:
Making use of AAD authentication as mentioned in the doc https://github.com/Azure/azure-event-hubs-spark/blob/master/docs/PySpark/use-aad-authentication-pyspark.md.

The structured streaming job works with the initial connection and reads records but after a couple of minutes fails with "com.microsoft.aad.msal4j.MsalClientException: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake"

Stack Trace
ERROR ConfidentialClientApplication: [Correlation ID: 8ac24ed7-93be-4b43-9d19-14b98b32268b] Execution of class com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier failed.
com.microsoft.aad.msal4j.MsalClientException: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:53)
at com.microsoft.aad.msal4j.OAuthHttpRequest.send(OAuthHttpRequest.java:33)
at com.microsoft.aad.msal4j.TokenRequestExecutor.executeTokenRequest(TokenRequestExecutor.java:33)
at com.microsoft.aad.msal4j.AbstractClientApplicationBase.acquireTokenCommon(AbstractClientApplicationBase.java:128)
at com.microsoft.aad.msal4j.AcquireTokenByAuthorizationGrantSupplier.execute(AcquireTokenByAuthorizationGrantSupplier.java:63)
at com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.acquireTokenByClientCredential(AcquireTokenByClientCredentialSupplier.java:86)
at com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.execute(AcquireTokenByClientCredentialSupplier.java:49)
at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)
at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1601)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1431)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1329)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:444)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:415)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1347)
at sun.net.www.protocol.http.HttpURLConnection.access$100(HttpURLConnection.java:97)
at sun.net.www.protocol.http.HttpURLConnection$8.run(HttpURLConnection.java:1314)
at sun.net.www.protocol.http.HttpURLConnection$8.run(HttpURLConnection.java:1312)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1311)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264)
at com.microsoft.aad.msal4j.DefaultHttpClient.executeHttpPost(DefaultHttpClient.java:59)
at com.microsoft.aad.msal4j.DefaultHttpClient.send(DefaultHttpClient.java:37)
at com.microsoft.aad.msal4j.HttpHelper.executeHttpRequestWithRetries(HttpHelper.java:96)
at com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:49)
... 14 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:481)
at sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:470)
at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:110)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1423)
... 31 more

I have attached the spark logs and the callback class we have written for getting the token

• Databricks 10.4 LTS. Spark 3.2.1, Scala 2.12
• azure-eventhubs-spark_2.12, 2.3.22 version
  AadIssue.zip

[vaiyarUp]

Hi,

Any update on the issue