Support for EKS operator

[VolkerSchiewe]

Description

We are using MWAA in combination with EKS so that all our dags in airflow are running in our EKS. We would like to use the same setup with cosmos.

What changes?

• New EKSOperator classes (inheriting from KubernetesOperators) - Based on the original EksOperator
• Tests
• Adjusted documentation

Related Issue(s)

Breaking Change?

No - only an additional feature

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	6faf960
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/664739102707d100088384f3

[tatiana]

Thanks for the contribution, @VolkerSchiewe !
Please, could you address the tests currently failing?

[VolkerSchiewe]

Thanks for the contribution, @VolkerSchiewe ! Please, could you address the tests currently failing?

Hi @tatiana thanks for getting back to me! I was already working on it, but I ran into this issue: apache/airflow#39103

Seems to be affecting the latest version of the amazon airflow provider. I already tried with pinning an older version of xmlsec1 with no success :/

[tatiana]

Hi @VolkerSchiewe , just a head's up: we're wrapping up the Cosmos 1.4 release and either myself or @pankajkoti will support you on this next week!

[VolkerSchiewe]

Looking in to the failing tests I'm not sure where the problems are coming from 🤔

Unit tests:
Seem to be mostly fine. The 1-2 tests that are failing might just be a version incompatibility. Maybe we can exclude this somehow in the compatibility matrix

Integration tests:
I think the tests that I added should not be executed in integration, but I think the imports are still executed and failing because I'm missing the amazon packages in the integration environment. Should I add a try ... except around the import to not import the package if it is not installed? How do you handle those cases normally?

Also let me know if the test coverage is enough. I tried to cover the parts that are special for the eks operator, but didn't want to copy everything from the KubernetesOperator since it's tested there already.

[pankajkoti]

It appears that earlier versions of Flask are not compatible with newer versions of Jinja2; and hence we have the failure in our tests

These are the constraints for Airflow 2.3 which suggests Flask 1.1.2 and Jinja2 3.0.3, however dbt-core is upgrading it to jinja2-3.1.4 (Jinja2<4,>=3.1.3 (from dbt-core)) which is not supported on Flask 1.1.2 that comes with Airflow.

[VolkerSchiewe]

Hey there, the only failing parts are the tests with Airflow 2.3 and python 3.9. For some reason the install step fails with

pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

when installing the test requirements . I don't really understand why and how to solve this. Is this problem you ran into earlier already?

[VolkerSchiewe]

Fixing boto3 in the runs with airflow 2.3 solved the problem 🚀