Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow users to add dag_integrity_exceptions.txt to ignore errors #1613

Merged
merged 2 commits into from May 10, 2024

Conversation

collinmcnulty
Copy link
Contributor

Description

Sometime a DAG doesn't parse in a CI pipeline for a reason that is ok. This provides a way to make that exception just for one dag file instead of having to use -f. What I'm not sure of right now is how to put this code such that astro dev parse is what's affected, as we were using an existing astro project.

🧪 Functional Testing

I wrote a dag that doesn't parse, made sure it failed astro dev parse. Then added it to tests/dag_integrity_exceptions.txt and it didn't fail to astro dev parse anymore.

📋 Checklist

  • Rebased from the main (or release if patching) branch (before testing)
  • Ran make test before taking out of draft
  • Ran make lint before taking out of draft
  • Added/updated applicable tests
  • Tested against Astro-API (if necessary).
  • Tested against Houston-API and Astronomer (if necessary).
  • Communicated to/tagged owners of respective clients potentially impacted by these changes.
  • Updated any related documentation

Copy link

codecov bot commented Apr 1, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.55%. Comparing base (976d2d4) to head (e728c5f).
Report is 33 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1613   +/-   ##
=======================================
  Coverage   86.55%   86.55%           
=======================================
  Files         114      114           
  Lines       15935    15936    +1     
=======================================
+ Hits        13792    13793    +1     
  Misses       1288     1288           
  Partials      855      855           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@collinmcnulty collinmcnulty marked this pull request as ready for review May 6, 2024 17:36
@collinmcnulty
Copy link
Contributor Author

@sunkickr I don't know how to make this pass the codecov test

@sunkickr
Copy link
Contributor

sunkickr commented May 6, 2024

@collinmcnulty we should also make astro dev init create the file .astro/dag_integrity_exceptions.txt for users. If it was just an empty text file would that be okay?

@@ -115,6 +115,7 @@ func Init(path, airflowImageName, airflowImageTag string) error {
"README.md": Readme,
"tests/dags/test_dag_example.py": DagExampleTest,
".astro/test_dag_integrity_default.py": DagIntegrityTestDefault,
".astro/dag_integrity_exceptions.txt": "# Add dag files to exempt from parse test below. ex: dags/<test-file>",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would users with existing projects fix this? Do we need something in our release notes for this ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah If they want this file they would need to delete .astro/test_dag_integrity_default.py and run astro dev init

@sunkickr sunkickr merged commit a8d64c9 into main May 10, 2024
4 of 5 checks passed
@sunkickr sunkickr deleted the allow-integrity-exceptions branch May 10, 2024 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants