Discussion on options to pin Python package versions and manage python version bump lifecycles

[aaronsteers]

Re:

• feat: Lock package versions in plugin lock files #6416

Currently we have lockfiles to lock plugin definitions, but those plugin definitions are most often using unpinned repo references and/or unpinned PyPi references.

Today, users are advised to manually add version constraints to the pip_url declaration in their own meltano.yml, which at that point would keep their project behaving consistently. We currently do not enforce or provide an automated way to pin those versions, nor do we offer a way to electively bump versions after they are pinned.

We should come up with a plan to let users more effectively manage their Python versions.

Some things to consider:

• do we need a separate constraints definition separate from the pinned version?
• should we lock in the lockfile or in meltano.yml?
• what is the CLI interface?
• do we only solve a one-time lock operation, or do we also assist in bumping the version?
• do we need the ability to lock or constrain dependencies as well as the named packages?

Related:

• Spec discussion: Updating or freezing pip URL when running `lock --update` #6445

[aaronsteers]

One option that does not require changes to Meltano Core would be to run a periodic job on the Hub (perhaps mimicking what HubCap already does for dbt Hub), we could scrape versions and bump them from GitHub and/or PyPi. Then, all plugins on the hub would be pinned to specific Python versions - if and when pinnable versions exist for each library.

Then, next time the user locks their plugin, or updates their lockfile from the hub, those definitions would be pinned to some version or version range. (And from there, they would either manually bump to get new features, or repeat the "update lockfile from hub" workflow.)

This might only be a partial solve. It handles providing stability to the main (named) packages. It provides users an option to bump version by getting the latest from the Hub. It doesn't give robust constraints controls, and depending on if we provide ranges or not, it doesn't solve for whether patch releases should be auto-accepted or ignored.

Another challenge is that attempting to bump multiple libraries on the Hub, which are all referenced on the same pip_url entry (delimited by spaces) does not guarantee a successful resolution, since the most recent versions of the multiple packages in some cases won't be compatible with each other. We'd likely need some way to specify which plugins are prioritized and which are dependent. If explicit control would be given to have plugin maintainers constrain the versions to something other than latest, then we'd need to decouple those constraints from the existing pip_url field, which would be serving an overloaded function of 'version lock' as well as 'constraint bounds' - leading to not being able to do both at the same time for a specific plugin.

[edgarrmondragon]

It's only a problem with a small minority of plugins in the Hub, but the pip_url may contain arbitrary args for pip install that would force us to skip those or have slightly more complex logic to detect the package URIs in the string.

[WillDaSilva]

do we need a separate constraints definition separate from the pinned version?

Yes. Dependency version constraints are conceptually separate from pinned versions. Also, we don't get to decide what the version constraints are: that's up to the plugin authors. We are only providing a service on top of that by resolving those constraints into pinned versions.

should we lock in the lockfile or in meltano.yml?

The lockfiles. Platform-independent pinned Python deps with hashes will be thousands of lines of barely readable text, as we see with poetry.lock. Putting them in meltano.yml would make for a poor user experience since that file is meant to be edited by humans, while lockfiles are explicitly not meant to be edited by humans, and it would also result in serious performance issues because Meltano cannot handle large yaml files.

what is the CLI interface?

meltano lock - this is something that I think everyone will want by default, so I see no reason to hide it behind a flag or anything like that.

do we only solve a one-time lock operation, or do we also assist in bumping the version?

Dependency version pinning/locking typically takes the latest version of each package that abides by the version constraints, and is compatible with all other pinned dependencies. As such, if the user wants to get the latest versions of transitive dependencies allowed by the constraints imposed by the plugin at the version they chose, they need only recreate the lock files by running meltano lock again.

do we need the ability to lock [...] dependencies as well as the named packages?

The version of the named package (i.e. the plugin itself) should be set by the user in meltano.yml, but whether it is or not it'll be pinned in the lockfile along with all the other dependencies. I don't see why we should give it special treatment when it comes to pinning dependencies in a lockfile.

---

We want to be able to commit lockfiles to a repository for reuse. That means that the lockfiles should be platform-independent, which is very difficult. I suggest we leverage Poetry's dependency resolver to tackle this problem, rather than trying to figure it out ourselves.

Python makes it especially difficult because each Python package can change its dependencies dynamically at install-time. While our solution won't be perfect in the face of this challenge, the main factors we'll have to consider to make this work in a platform-independent way is:

• OS
• Python version
• Python implementation (e.g. CPython, PyPy, IronPython, etc.)
• min glibc version (for Linux)
• CPU architecture

Poetry handles those challenges well enough to be very useful for a large number of Python developers. We can probably work with that.

[WillDaSilva]

@aaronsteers I agree that it's a large scope. If we only pin the versions of the top-level "named" packages (i.e. direct dependencies, as opposed to transitive dependencies), we haven't accomplished anything that couldn't already be done. We've made it more convenient, but we'd be misleading if we did that and then told the community "you can now lock your dependencies".

The expectation with locked dependencies is reproducibility, and unless we pin the versions of all transitive dependencies we won't have a way to make meltano install reproducible.

If the scope is just pinning the version of the direct dependencies, then I'd reframe this whole discussion away from the large, complex, but ultimately valuable thing proposed above.

Instead we could just have meltano add insert ==X.Y.Z after the PyPI dependency name, and @vX.Y.Z (or the commit hash if no suitable tag could be found) after the git URL in pip_url. Then if users want to stick with these version pins they can, and if not they're free to change it.

It's a very simple approach that doesn't commit us to maintaining much of anything, and it doesn't stop us from implement plugin dependency locking (including transitive dependencies) later on. I consider an approach like this to be mostly unrelated to plugin dependency locking. I see it as more of a quality of life improvement for meltano add.

[aaronsteers]

If the scope is just pinning the version of the direct dependencies, then I'd reframe this whole discussion away from the large, complex, but ultimately valuable thing proposed above.

Instead we could just have meltano add insert ==X.Y.Z after the PyPI dependency name, and @vX.Y.Z (or the commit hash if no suitable tag could be found) after the git URL in pip_url. Then if users want to stick with these version pins they can, and if not they're free to change it.

I think this smaller scope still has a lot of value. It doesn't solve for absolute freeze, but it does handle the first-layer and very important base requirement: moving users away from widespread unpinned plugin refs, and to provide a default experience that gives them at least some out-of-box stability. (This is a real problem affecting tap-gitlab, for instance.)

[aaronsteers]

Instead we could just have meltano add insert ==X.Y.Z after the PyPI dependency name, and @vX.Y.Z (or the commit hash if no suitable tag could be found) after the git URL in pip_url.

Yes, exactly this. Except, even with this smaller scope, it's not clear to me if we should override/ignore any existing version pins, or if we should bump them. And if we do bump them, do we need a way to define a limit? Again, this is still a very hard problem because of how pip_url is a space-delimited set of package names at best, and potentially also can contain random pip url args like -e.

Even to get to the basic scope, I wonder if we need to introduce something like a package_constraints array - where users (or the Hub) could optionally specify a list of version constraints that would serve as bound for an automated bump of pip_url entries.

Alternatively, if underlying libraries support it, the pip_url could serve as the wide bounds or unbounded ref (the default for new additions today), and a new constraints array could provide exact or ranged limits. This is not to say that constraints would try to constrain every package, but it could be used to pin dbt-duckdb to an exact version while either not pinning dbt-ext at all, or else giving a wide constraint, like dbt-ext<=2.0.

pip_url: dbt-duckdb dbt-ext
pip_constraints:
- dbt-duckdb==2.1.0  # keep stable until I ask to bump
- dbt-ext<2.0        # ok if this adjusts to fit compatibility with the above

Then, optionally, a lock or bump operation with the above already set, might insert exact version numbers into the pip_url:

pip_url: dbt-duckdb==2.1.0 dbt-ext=1.2.1
pip_constraints:
- dbt-duckdb==2.1.0  # keep stable until I ask to bump
- dbt-ext<2.0        # ok if this adjusts to fit compatibility with the above

And another bump or re-lock might move dbt-ext without moving dbt-duckdb:

pip_url: dbt-duckdb==2.1.0 dbt-ext=1.2.5
pip_constraints:
- dbt-duckdb==2.1.0  # keep stable until I ask to bump
- dbt-ext<2.0        # ok if this adjusts to fit compatibility with the above

Noting that for the examples above, having dbt-ext<2.0 is functionally equivalent to not mentioning it, until such time as dbt-ext reaches a 2.x release.

[WillDaSilva]

Instead we could just have meltano add insert ==X.Y.Z after the PyPI dependency name, and @vX.Y.Z (or the commit hash if no suitable tag could be found) after the git URL in pip_url.

Yes, exactly this. Except, even with this smaller scope, it's not clear to me if we should override/ignore any existing version pins, or if we should bump them.

It's during meltano add, so there would be no pre-existing version pins to contend with.

Even to get to the basic scope, I wonder if we need to introduce something like a package_constraints array - where users (or the Hub) could optionally specify a list of version constraints that would serve as bound for an automated bump of pip_url entries.

I like the idea of a constraints array for dependencies, but that's yet another separate topic - I'm having trouble keeping up.

Currently we defer to pip for our dependency syntax and semantics. A dependency constraints array would naturally leverage pip's constraint file feature.

Then, optionally, a lock or bump operation with the above already set, might insert exact version numbers into the pip_url

This is where we get into the difficulties you mentioned when you said "it's not clear to me if we should override/ignore any existing version pins, or if we should bump them". I'm not sure if this specifically is worth doing since it's so much more complicated than just making meltano add pin the main direct dependency if possible. At the very least we should treat it as a follow-up to making meltano add pin the main direct dependency, rather than part of the same issue.

If we do ultimately decide to implement that feature (updating existing pip_urls and constraints in meltano.yml, then I suggest we do not use the meltano lock command for it, because it is largely separate from locking dependencies and lockfiles in general. I'm not sure what the appropriate CLI interface would be, but I do not think it is meltano lock.

- dbt-duckdb==2.1.0 # keep stable until I ask to bump

As for bumping pinned dependencies, that's yet another separate topic (and a complex one at that!). It's the sort of thing that something like dependabot addresses, rather than lockfiles, version pinning, or anything else discussed so far here.

[aaronsteers]

Agreed with basically all of your comments above. This is a hard problem, with many viable solutions (all of them non-trivial in scope/complexity) and no obvious winner from the viable paths forward.

On this point:

It's during meltano add, so there would be no pre-existing version pins to contend with.

This is only true if we don't do the Hub-side work first, and if maintainers never pin or constrain their pip_urls. If we're successful in encouraging maintainers (or our own bots) to bump versions in the Hub, then we get back to this question. (I'd have to check if any versions are pinned on the hub, and how many if so.)

Hub-side convo here: #7297 (comment)

[aaronsteers]

Thread Re CLI inteface

what is the CLI interface?

meltano lock - this is something that I think everyone will want by default, so I see no reason to hide it behind a flag or anything like that.

Referencing https://docs.meltano.com/reference/command-line-interface#lock for context.

I think there are at least two things to tease out here:

1. Initial operation (either add or first-timelock)
   • I think it's safe to change the default behavior to lock the plugin refs by default, if the Hub has not already done so. That said, if the Hub does have pinned versions in pip_url, should we ignore those, should we use them as constraints, or should we respect them?
   • To me, this "bump or not" during initial add or lock operation might need to be configurable. And it specifically gets messy when the Hub has a pinned version (or version range) already, and Meltano then needs to decide whether to respect that or re-lock based on available versions at time of the lock operation.
2. Following update operations (always meltano lock or meltano lock --update)

• Today --update means get the latest lock file definition from the Hub.
• Without --update, lock and lock --all is a no-op (I think?) for already-locked plugins.
  • Would we change this to also pin versions even if the file is already locked? I think yes.
  • Would we change this to additionally bump already-pinned versions? I think probably not.
• When --update is provided, do we always reset and bump to the latest pinnable version - or do we get the latest from the Hub? What if the version has been pinned by the user? Do we override the user-pinned version?
  • Maybe --bump or --upgrade could be added for version bumps, in addition to --update on the metadata side? Or --update could take an argument: --update=metadata, --update=packages, --update=all.

I think minimal changes could probably accomodate this, but I just want to be mindful of the overloaded lock and update meanings when considering plugin metadata vs package versions, and "bump within constraint" vs "bump to whatever is latest available". It doesn't help that we don't have constraints separate from pip_url, but that's probably better for its own thread.

[WillDaSilva]

I don't understand what you're talking about with regard to the Hub on this topic. Plugins are currently unversioned on the Hub, so the Hub cannot meaningfully pin anything - not even the main direct dependency - without preventing people from being able to update, and forcing us to update the plugin on the Hub each time its main direct dependency is updated.

I think we should try to clearly separate these mostly distinct topics:

• Plugin versions on the Hub (which doesn't currently exist).
• Pinning the version of the primary dependency of a plugin when it is added (as mentioned here).
• Pinning the versions of a plugins dependencies, including all transitive dependencies (as mentioned here).
• Updating plugin lockfiles, the meaning of which changes dramatically based on decisions made around the three points above.

Conflating these topics will lead to confusion.

Does this breakdown seem reasonable to you @aaronsteers?

[aaronsteers]

Does this breakdown seem reasonable to you @aaronsteers?

Yes, definitely.

[aaronsteers]

Thread for the Hub bullet point in a new thread: #7297 (comment)

[WillDaSilva]

Related discussion: #6445

[aaronsteers]

Plugins are currently unversioned on the Hub, so the Hub cannot meaningfully pin anything

Agreed. But just to call out: this is due to lack of automation though and not at all by design. HubCap does this function for dbt Hub - bumping pinned referenced based on results of a daily scan. And if we did solve in the hub, then it is a valid alternative first iteration to the problem as a whole, since the Hub definitions fully encapsulate the first-use experience for plugins added to projects. It's not clear to me yet, if the code for finding a valid pinnable reference needs to be written anyway, whether this should first live on the Hub - which benefits all users of all Meltano versions, and also brings error handling out of the runtime experience - versus living in Meltano, where failures and exceptions will break users and where it can only benefit users on future versions of Meltano.

Certainly it is good to have this in both places, but which should come first I think it still an open question in my mind.

[WillDaSilva]

My last point in this message is related, since this is about version bumping, as opposed to version pinning.

Certainly it is good to have this in both places, but which should come first I think it still an open question in my mind.

I think it makes sense to have a way to pin dependencies first, and then figure out how to bump them. If you can't pin them, there's nothing to bump, right?

[aaronsteers]

My point was if we automate pinning+bumping them in the Hub, then Meltano doesn't need to change.

The bump vs pin distinction is important and I agree. On the Hub side, it's a continuous version bump problem. On the Meltano side, its (primarily) a one-time version pinning problem.

If we solve the bump process in Hub, the Meltano-side pin problem is then already solved - at least for all new additions and for everyone who runs meltano lock --update.

[WillDaSilva]

I see what you mean, because Meltano can get the pinned pip_url from the Hub, and then use that to overwrite the pip_url in meltano.yml. Whether it only overwrites it in-memory, or actually overwrites it in the file is another matter.

Do you think it would make sense to implement this in meltano, and then have a job that runs on a schedule/trigger that uses meltano to update the Hub?

[aaronsteers]

I see what you mean, because Meltano can get the pinned pip_url from the Hub, and then use that to overwrite the pip_url in meltano.yml. Whether it only overwrites it in-memory, or actually overwrites it in the file is another matter.

Yeah, exactly. 👍

Do you think it would make sense to implement this in meltano, and then have a job that runs on a schedule/trigger that uses meltano to update the Hub?

To me, at least, it just seems a lot harder to solve first in Meltano if we're going to still want Hub to have pinned versions. Meltano can't read or write natively to the yaml format on the hub so we'd need custom code in the Hub anyway. (Or maybe it can if the yaml files are mostly schema-equivalent to the json lock files and the meltano.yml spec (?) - but anyway it's not something we've tested.)