Dev container config update

[algattik]

Type of PR

• Documentation changes
• Code changes

Purpose

Upgrade dev container configuration to current standards, and misc improvements.

• Removed JAVA_HOME Docker env var, as the path is incorrect on non AMD64 arch (e.g. Mac M1), and variable is not needed by Pyspark
• Aligned python versions with Databricks Runtime
• Updated devcontainers.json format to current standard (previous format gave warnings in viscose)
• Include .git folder in container to enable git integration
• Also lint tests with flake8, not only runtime code
• Updated README for new behavior of test extension
• Updated obsolete collect_ignore pytest parameter
• Misc improvements

Does this introduce a breaking change? If yes, details on what can break

No

Author pre-publish checklist

Validation steps

Run unit tests in dev container on VSCode

Issues Closed or Referenced