feat: Just-in-time plugin installation

[ReubenFrankel]

Implements just-in-time install for the following commands:

• invoke
• run
• el/elt
• test
• config <plugin> test
• select --list

[netlify]

👷 Deploy request for meltano pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	6fb26f6

[codecov]

Codecov Report

Attention: Patch coverage is 98.42520% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 91.73%. Comparing base (9b75e89) to head (90e0d67).
Report is 1 commits behind head on main.

❗ Current head 90e0d67 differs from pull request most recent head 6fb26f6. Consider uploading reports for the commit 6fb26f6 to get more accurate results

Files	Patch %	Lines
src/meltano/cli/select.py	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8482      +/-   ##
==========================================
- Coverage   91.78%   91.73%   -0.06%     
==========================================
  Files         245      245              
  Lines       19325    19353      +28     
  Branches     2149     2150       +1     
==========================================
+ Hits        17738    17753      +15     
- Misses       1313     1326      +13     
  Partials      274      274              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ReubenFrankel]

We probably want to suppress install progress to stdout from meltano.cli.utils.install_plugins here, or emit as logs to stderr... For invoke, this currently breaks the common tap/target debugging technique of

meltano invoke <tap> > tap.out
cat tap.out | meltano invoke <target>

[edgarrmondragon]

We probably want to suppress install progress to stdout from meltano.cli.utils.install_plugins here, or emit as logs to stderr... For invoke, this currently breaks the common tap/target debugging technique of

meltano invoke <tap> > tap.out
cat tap.out | meltano invoke <target>

Yeah, I'd be more than OK with starting to unify our output away from click.echo and into the logging framework.

[ReubenFrankel]

Think I have covered all relevant commands here. Wondering if we want a new --no-install option for these (same convention as for meltano add) to opt-out of auto-install behaviour (e.g. for CI), or stick to the original proposal in #2089:

Interactive (TTY) experience

If a tty is available, we can simply ask the user if they'd like to auto-install the missing plugins.

Updated error message with hints

If a TTY is not available and --install is not specified, then the error message would be changed to something like:

ERROR: Plugin not installed. Please install with meltano install ... or rerun with meltano invoke --install ....

Why require confirmation / opt-in

Install operations are not always safe to run, and they can hide other issues in the invocation or installation, such as using the wrong plugin name or the wrong base image in a scheduled job. They could also trigger unnecessarily for a container-backed spec that doesn't require installation.

I haven't yet addressed install progress/status as logs, but the output is suppressed for meltano invoke for now.

[ReubenFrankel]

Should this be added for select too?

[edgarrmondragon]

Should this be added for select too?

If it's opt-in, probably yes 👍

[ReubenFrankel]

Should we collect all these newly-added install reasons under an aggregate PREREQUISITE reason, or something like that?

[edgarrmondragon]

Yeah, PREREQUISITE or something like JIT or similar seems like a good option

[ReubenFrankel]

This is a very basic check to determine whether or not to install based on the presence of the plugin virtual environment directory. Is there anything more intelligent we can do here to satisfy the stale plugin requirement from #2089? #6416 (comment) and #2651 seem relevant.