[Bug]: Databricks does not always show failure logs

[sdreyer]

Expected Behavior

When a databricks job fails, the jobs should be printed to the dashboard

Actual Behavior

No error message

Error Message: transformation failed to complete: spark submit job for transformation {transactions_transformation thirsty_kilby Transformation} failed to run: the '599492515411191' job failed:

Steps To Reproduce

Run a transformation on an s3 file. Provide incorrect credentials for S3.

What mode are you running Featureform in?

Hosted

What version of Python are you running?

3.8

Featureform Python Package Version

1.10.5

Featureform Helm Chart Version

Docker from main

Kubernetes Version

No response

Relevant log output

An error occurred while calling o407.head.
: java.nio.file.AccessDeniedException: s3a://featureform-spark-testing/behave/featureform/scripts/spark/offline_store_spark_runner.py: getFileStatus on s3a://featureform-spark-testing/behave/featureform/scripts/spark/offline_store_spark_runner.py: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://featureform-spark-testing.s3.us-east-2.amazonaws.com behave/featureform/scripts/spark/offline_store_spark_runner.py {} Hadoop 3.3.6, aws-sdk-java/1.12.390 Linux/5.15.0-1042-azure OpenJDK_64-Bit_Server_VM/25.372-b07 java/1.8.0_372 scala/2.12.15 kotlin/1.6.0 vendor/Azul_Systems,_Inc. cfg/retry-mode/legacy com.amazonaws.services.s3.model.GetObjectMetadataRequest; Request ID: G9RPJAFCVADEQC1H, Extended Request ID: