Prototype of stopping after N rows loaded from taps

[dcowden]

This is a sketch for how i introduced ability to limit rows from taps for the purposes of testing.

Warning: This code has not been executed. I have working code based on an older revision that i tested. So, i know the approach works with code very similar to this. I cleaned up the code to fit into latest meltano main, so that it would be easier to see the essence of the changes.

If this overall path is of interest, i could do more work-- i really would like thoughts about the overall approach before I do that. I dont know the meltano code well at all, so this might be a totally wrong way to do it.

Other notes that might be helpful:

• i first tried a similar approach in the singer runner code, until i realized that was not the code used for meltano run
• This uses an environment var to set the row limit. A much better solution of course would be to allow a property on the project and/or taps, so the row limit can be different for each tap. But (a) I dont know enough to approach that, and (b) for us this is sufficient
• I am worried about whether this code would work in environments with concurrency more than just one. I did not do any testing along those lines.

[netlify]

✅ Deploy Preview for meltano canceled.

Name	Link
🔨 Latest commit	ad81936
🔍 Latest deploy log	https://app.netlify.com/sites/meltano/deploys/65c26ec521ac740008b105ef

[wesseljt]

@edgarrmondragon and team let us know if we can do anything to improve or test any specific use cases.

[codecov]

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (13963b0) 91.50% compared to head (ad81936) 91.48%.
Report is 5 commits behind head on main.

Files	Patch %	Lines
src/meltano/core/block/extract_load.py	45.83%	12 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8364      +/-   ##
==========================================
- Coverage   91.50%   91.48%   -0.03%     
==========================================
  Files         244      245       +1     
  Lines       19125    19168      +43     
  Branches     2139     2143       +4     
==========================================
+ Hits        17501    17536      +35     
- Misses       1344     1352       +8     
  Partials      280      280              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.