Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run unit tests with airflow installed from packages #39513

Merged
merged 1 commit into from May 15, 2024
Merged

Run unit tests with airflow installed from packages #39513

merged 1 commit into from May 15, 2024

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented May 9, 2024 •
edited

This PR adds the option of running unit tests for providers against
a specific airflow version (for example released version in PyPI)
and enables it for back-compatibility testing for 2.9.1. In the
future it could be used to run forward-compatibility testing with
Airflow 3 as well.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk added the full tests needed We need to run full set of tests for this PR to merge label May 9, 2024
@potiuk potiuk closed this May 9, 2024
@potiuk potiuk reopened this May 9, 2024
@potiuk potiuk force-pushed the run-unit-tests-with-airflow-installed-from-packages branch 9 times, most recently from 1194fde to f115589 Compare May 9, 2024 20:57
@potiuk potiuk marked this pull request as ready for review May 9, 2024 20:59
@potiuk
Copy link
Member Author

potiuk commented May 9, 2024

As explained in the devlist - This PR - specifically this job https://github.com/apache/airflow/actions/runs/9023444361/job/24795410794?pr=39513 runs Provider unit tests against Airflow 2.9.1 installed from PyPI. I will add a few comments and tag relevant people in the places where I need help to understand if my changes are ok or not as we need to add compatibility layer for our tests to make it works for 2.9.1 (and later we should do the same for 2.8 and 2.7).

potiuk
potiuk commented May 9, 2024
View reviewed changes
tests/conftest.py Outdated Show resolved Hide resolved
potiuk added a commit to potiuk/airflow that referenced this pull request May 15, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
283a0b2 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 15, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
f39f773 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 15, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
7effe29 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 15, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
e1b9f59 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
vincbeck
vincbeck reviewed May 15, 2024
View reviewed changes
Copy link
Contributor

@vincbeck vincbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nits

contributing-docs/testing/unit_tests.rst Outdated Show resolved Hide resolved
contributing-docs/testing/unit_tests.rst Outdated Show resolved Hide resolved
contributing-docs/testing/unit_tests.rst Outdated Show resolved Hide resolved
contributing-docs/testing/unit_tests.rst Outdated Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
@potiuk
Run unit tests for Providers with airflow installed as package.
512155d 
This PR adds the option of running unit tests for providers against
a specific airflow version (for example released version in PyPI)
and enables it for back-compatibility testing for 2.9.1. In the
future it could be used to run forward-compatibility testing with
Airflow 3 as well.
@potiuk potiuk force-pushed the run-unit-tests-with-airflow-installed-from-packages branch from 1080122 to 512155d Compare May 15, 2024 19:19
@potiuk
Copy link
Member Author

potiuk commented May 15, 2024

Comments resolved / nits merged :)

@potiuk
Copy link
Member Author

potiuk commented May 15, 2024

All conversations resolved and answers found :). ... I think we are good to go.

@potiuk potiuk merged commit 81a82d8 into apache:main May 15, 2024
69 checks passed
@potiuk potiuk deleted the run-unit-tests-with-airflow-installed-from-packages branch May 15, 2024 20:16
@potiuk
Copy link
Member Author

potiuk commented May 15, 2024

Finaly :). RIGHT before a meetup where I will talk about CI/CD tests :)

potiuk added a commit to potiuk/airflow that referenced this pull request May 18, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
3fb63e3 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 19, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
4c5f6b0 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 19, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
6755739 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.

The changes include simplifying specification of providers to exclude
for each version and adds a warning for a problem detected in common.io
provider where common.io FileTransfer operator does not work for
Airflow 2.8.
potiuk added a commit to potiuk/airflow that referenced this pull request May 19, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
93722c1 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.

The changes include simplifying specification of providers to exclude
for each version and adds a warning for a problem detected in common.io
provider where common.io FileTransfer operator does not work for
Airflow 2.8.
potiuk added a commit to potiuk/airflow that referenced this pull request May 20, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
2a0db22 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.

The changes include simplifying specification of providers to exclude
for each version and adds a warning for a problem detected in common.io
provider where common.io FileTransfer operator does not work for
Airflow 2.8.
potiuk added a commit to potiuk/airflow that referenced this pull request May 22, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
7c7e00a 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.

The changes include simplifying specification of providers to exclude
for each version and adds a warning for a problem detected in common.io
provider where common.io FileTransfer operator does not work for
Airflow 2.8.
potiuk added a commit to potiuk/airflow that referenced this pull request May 22, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
9653535 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 22, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
dced766 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.

The changes include simplifying specification of providers to exclude
for each version and adds a warning for a problem detected in common.io
provider where common.io FileTransfer operator does not work for
Airflow 2.8.
potiuk added a commit to potiuk/airflow that referenced this pull request May 22, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
f778dd7 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 22, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
c149fbd 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 22, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
0abb5ec 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 22, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
81a7e35 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 22, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
d05e641 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.
potiuk added a commit to potiuk/airflow that referenced this pull request May 22, 2024
@potiuk
Run tests for Providers also for Airflow 2.8
76cd3c0 
This is a follow-up on apache#39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.

This change includes:

* simplifying the way how we specify provider exclusions in tests
* update to the unit test documentation describing testing
* updating to latest pytest tooling in case older airflow version
  is installed (pulling the correct versions and correct packages
  for pytest extensions)
* implementing 2.8 compatibility for the conftest/test common code
* implementing 2.8 compatibility for provider tests that relied
  on 2.9+ behaviours
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@uranusjr uranusjr uranusjr left review comments

@Taragolis Taragolis Taragolis left review comments

@Lee-W Lee-W Lee-W left review comments

@kacpermuda kacpermuda kacpermuda left review comments

@vincbeck vincbeck vincbeck approved these changes

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

@mobuchowski mobuchowski Awaiting requested review from mobuchowski mobuchowski is a code owner

@eladkal eladkal Awaiting requested review from eladkal eladkal is a code owner

@o-nikolas o-nikolas Awaiting requested review from o-nikolas o-nikolas is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy

@dstandish dstandish Awaiting requested review from dstandish

Assignees
No one assigned
Labels
area:dev-tools full tests needed We need to run full set of tests for this PR to merge kind:documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
10 participants
@potiuk @bolkedebruin @Lee-W @uranusjr @Taragolis @mobuchowski @dstandish @o-nikolas @vincbeck @kacpermuda