[Feature Request]: Surface Discrepancy b/w Client Python version and PySpark version

[epps]

Description

Due to known issues around pickling Python code with dill, it's important to use the same version of Python on the client as is used by the user's Spark cluster.

Current Behavior

No response

Desired Behavior

When registering Spark as a provider:

• compare the version of Python on the Client with PySpark's Python version
• communicate any discrepancy between the versions so the user is aware of the possibility of pickling errors

Benefits

"Failing" early to avoid registering transformations that are likely to fail due to unpickling issues.

Possible Implementation

No response

Additional Context

No response