[Bug]: Databricks Cluster version 14.0 with Spark version 3.50 with S3

[ahmadnazeri]

Expected Behavior

When running with Databricks (v14.0) on Azure and reading data from S3, it fails with the following error message:

transactions_transformation: FAILED - transformation failed to complete: spark submit job for transformation failed to run: (name: transactions_transformation variant:awesome_ride) the '532227170932654' job failed: java.lang.ClassCastException: shaded.databricks.org.apache.hadoop.fs.s3a.S3AStorageStatistics cannot be cast to org.apache.hadoop.fs.s3a.S3AStorageStatistics

But it works fine when I downgrade the cluster to v11.3.

Actual Behavior

The expected behavior is that we are able to run the job successfully end to end.

Steps To Reproduce

1. Create a Databricks Cluster with version 14.0 on Azure
2. Register Databricks with S3 provider
3. Run a transformation

What mode are you running Featureform in?

Hosted

What version of Python are you running?

3.9

Featureform Python Package Version

1.10.0

Featureform Helm Chart Version

No response

Kubernetes Version

No response

Relevant log output

transactions_transformation: FAILED - transformation failed to complete: spark submit job for transformation failed to run: (name: transactions_transformation variant:awesome_ride) the '532227170932654' job failed: java.lang.ClassCastException: shaded.databricks.org.apache.hadoop.fs.s3a.S3AStorageStatistics cannot be cast to org.apache.hadoop.fs.s3a.S3AStorageStatistics