EH - Trigger once

[rbushri]

I want to create a cron job that read from EH (with checkpoint) processes the new msgs and terminates, so I tried to run this code:

 spark.readStream.format("eventhubs").options(...).load()
.writeStream
.option("checkpointLocation",checkpointLocationPath)
.trigger(Trigger.Once())
.foreachBatch((ds : Dataset[String],id : Long) => {..}
.start()
.awaitTermination()

And I expected the job to run once and terminates but the job keeps running and ignores the trigger,
I also tried to remove the .awaitTermination() but the job didn't read anything from EH at all.
what did I do wrong?

[ronban]

I am facing the same issue on GCP Dataproc Batches. The job doesnt end. I have to use cancel after 30 seconds to manually end the job