Refactoring settings/config/environment variable management -- when and how? #7809
Replies: 1 comment
-
@cjohnhanson You and I have talked a fair bit about this topic, so you're already aware that I'm keen for us to tackle a refactor of how Meltano handles configuration for myriad reasons. The most concrete suggestion I've made to date on this topic is: In a nutshell, the above suggests using the library The big (related) questions I have on my mind when considering these approaches towards refactoring Meltano's config management are:
In considering the scope of the refactor we have to perform a balancing act. On one hand the refactor must be large/significant enough that it can actually root out the underlying problems with the existing approaches and address them, rather than merely add on another layer of complexity. On the other hand we need to avoid making this such a large and daunting piece of work that it's impractical to ever accomplish. Countless refactors/rewrites have failed completely because they could not be completed in an iterative fashion, so while keeping in mind that the refactor ought to produce a significant and foundational change, we should aim to design and implement a minimum viable refactor that can be built upon sustainably. With all that said, as tempting as starting from scratch with the config format/design would be (with a bump to the In summary:
|
Beta Was this translation helpful? Give feedback.
-
Settings management and environment variable expansion has been a consistent pain point and source of bugs within meltano core. We've made some progress--particularly with regards to better testing and all the bugfixes done as part of the meltano manifest work--over the past year, but it's often one step forward and two steps back as bugfixes and changes inevitably cause new bugs. I've increasingly been leaning towards doing a full refactor, possibly leaning on a third party library/framework like hyedra or Dynaconf. It's not feasibly at this point to do such a significant refactor as part of our 3.0 launch, but taking this on as part of a major version bump would be definitely be preferable since there's almost definitely going to be changes which break existing users'
meltano.yml
or otherwise changes the existing behavior. But on the other hand, this is a pressing issue for our users and likely not one we want to wait until a 4.0 release to handle.I'm interested in gathering everyone's thoughts about how best to approach this and what a potential timeline for a significant refactor could look like, whether that's as part of a 4.0 release or if there's an approach we can take that would be guaranteed to only make changes that could be released in a minor version bump.
Related slack thread: https://meltano.slack.com/archives/C03GKHWS0HM/p1685131341359449
CC: @meltano/engineering
Beta Was this translation helpful? Give feedback.
All reactions