Change from CLI-first to YAML-first Workflows

[sbalnojan]

All documentation and most marketing material for Meltano so far advertise "CLI-first" workflow. Urging developers to run interactive configs, sequences of CLI commands and only default to editing the meltano.yml file if no other option is available.

I propose to exchange this order, advertising developers to use a YAML-first workflow, defaulting to CLI for convenience functionality. Typical convenience functionality that IMHO make would make sense inside the CLI are:

• Commands where the CLI still does additional magic like performing a look up ("meltano add extractor xyz")
• Commands that are shorter inside the CLI (honestly I don't know of any right now).

Reasoning:

• For me, coming from a more DevOps world, the YAML-first approach feels very natural. The parallels to all IaC tools are quite visible here.
• The YAML-syntax are the artifacts that have to be versioned (the declarative piece of code), not the CLI-call (which is an imperative piece of code).
• For me it does make sense to have as "unit of work" my YAML code, otherwise things like "looking at my diff" don't actively translate to my work as data developer (If I run "meltano add tap-csv" on a project that already has tap-csv installed in my active environment, I have done work. And I yet haven't. That doesn't happen in a YAML first approach.)

New proposed workflow:

• This new workflow would mean, we would ask people to copy & paste example code. It would mean, we would have working examples of a few of the standard configurations for all taps on the hub very much akin to https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance.

Why:

• I think this change makes meltano easier to use, especially on the first couple of projects.
• I think it strengthens meltanos positioning against other tools.
• I think it makes data people more productive by advocating a better workstyle. (I might be biased here!)

I'm very curious for your thoughts!
@aaronsteers @tayloramurphy @DouweM

[menzenski]

I've just finished configuring 49 taps that inherit from tap-google-analytics. I wrote a longer note in #7165 (comment) but I think if I had done all the configuration of those plugins in YAML I'd have been more efficient.

[jaceksan]

I definitely prefer the YAML way, I wouldn't say I like interactive CLIs and have never used the Meltano one.

It would be great if, after adding a tap/loader, meltano.yml contains all mandatory properties with either placeholder values describing the purpose or empty values. Several times I struggled to find the proper property names in various documentation sites or related GitHub repositories. If a particular version of tap/loader could bootstrap its section in meltano.yml properly, it would be awesome!

[tombriggsallego]

I found the CLI commands for configuration useful at the very beginning. They were a useful way to change a setting conceptually and then see what that looked like in the YAML. It didn't take me long to start doing everything right in the YAML though... it's so much faster once you know what you're doing.

[sbalnojan]

@tombriggsallego do you think that this way around is still better if you e.g. have examples on each tap/target hub page? Would love to know whether you can give some examples.

[edgarrmondragon]

The main problem I see with YAML-based workflow is that the YAML schema is not validated at runtime. Users can validate their meltano.yml against the JSON schema present in the repo using IDE extensions, but ideally any meltano command should fail with a clear message if meltano.yml is not valid.

[WillDaSilva]

This will likely be changing soon: #7148 (comment)

[radbrt]

I want to echo @tombriggsallego here. Initially I got a lot of mileage out of the UI and seeing how meltano.yml changed when I did the visual. The --interactive is a decent alternative, but today I almost always write YAML directly.

But of course, the issue isn't an undying love for the CLI, it is remembering to ask "what will help people onboard". If we are, somehow, able to get a YAML config experience that autosuggests some settings or provides tooltips, that might be much better than the CLI. In the meantime, the CLI might be the best default option.

[SBurwash]

I agree with the .yml first approach as well. We almost exclusively use the yaml files when adding/editing plugins, only using the CLI for invoke and run commands.

The moment we find the CLI command most useful is when trying to add a new plugin to which we don't know the yaml layout. This could be circumvented by simply adding code examples in the plugin definitions though.

• To expand on this, I think a code example with all plugin settings shown would allow for easier integration and comprehension of the plugin's different functionalities.

[acarter24]

Same here. On the intro tutorials I found myself scrolling over the CLI examples to find out 'what does the output yaml look like'. I think having parity between the two would be great. Like the Google API code examples, you can toggle tabs between JS and Python, you could toggle between CLI and YAML view of the code example.

[tayloramurphy]

@acarter24 we'll actually have examples like this on our new website coming soon. We'll work to get the Hub updated as well with this info.

[tayloramurphy]

I'm pretty sure this is the most interacted with discussion we've ever had from the community 👀

From the product side, yes I'm supportive of this. I don't think we would update our copy to say "your YAML for ELT" or anything like that, but we should not shy away from the YAML backed nature of Meltano. In fact, when we moved to Ruamel we made it a lot easier to make customizations to the YAML and not have them disappear.

Edgar brings up a good point about validation of the YAML file and we'll likely soon have a solve for that as well.

@sbalnojan it seems like you had a few primary suggestions:

• Show the example YAML on the Hub for a given connector - we used to have that and it got dropped in the Hub revamp. We definitely should bring it back and it wouldn't be too hard to do.
• Show more examples of complete and working YAML - yes 💯 we need more of this.

As to the workflow piece - I think for building your Meltano project we should still encourage and promote the CLI as there are validation benefits to configuring the YAML through the CLI - but we can also promote directly editing the YAML and then having a meltano compile or validate type command to very types and structure. The CLI will still clearly be a priority for running your workloads, fetching logs, interacting with the cloud - but on the build side we can be more friendly and spark more joy.

[sbalnojan]

"spark more joy." <= I like.

[tombriggsallego]

@tayloramurphy can you say more about using the CLI for "fetching logs"? That sounds like something I don't know about but could get a lot of value from... 🤔

[tayloramurphy]

@tombriggsallego as part of our cloud offering we'll have a CLI way to fetch logs (meltano cloud logs [list|get|etc]). We're still scoping and it's not yet 100% determined how this feature will come appear in Meltano Core. But you're not the only one who's been interested in this 😄

[aphethean1]

100% for yml first. For this to fly, I think every .yml needs a version so that the meltano runtime is able to understand whether the configuration at rest is usable. Tons of parallels with kubectl and kubernetes ymls here.

This is something we have for our artefacts, but not for meltano.yml. So we could have a situation where an older project cannot be understood: https://www.matatika.com/docs/dataml/

[menzenski]

Ran into something today that made me think of this discussion.

I'm adding tap-github to my project. The Hub page suggests running meltano config tap-github set --interactive, so I do this, but I can't set my organizations:

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Extractor 'tap-github' Setting 7 of 14                                                                                                                                                                            │
│ ┌─────────────────────────┬──────────────────────────┐                                                                                                                                                            │
│ │                    Name │ organizations            │                                                                                                                                                            │
│ │ Current Value (default) │ (empty string)           │                                                                                                                                                            │
│ │                    Kind │ array                    │                                                                                                                                                            │
│ │                  Env(s) │ TAP_GITHUB_ORGANIZATIONS │                                                                                                                                                            │
│ └─────────────────────────┴──────────────────────────┘                                                                                                                                                            │
│  Description:                                                                                                                                                                                                     │
│ ╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │
│ │ An array of strings containing the github organizations to be included                                                                                                                                        │ │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│  To learn more about extractor 'tap-github' and its settings, visit https://hub.meltano.com/extractors/tap-github--meltanolabs                                                                                    │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

New value: payitgov

Failed to set value: Expecting value: line 1 column 1 (char 0)

New value: ['payitgov']

Failed to set value: Expecting value: line 1 column 2 (char 1)

New value: [payitgov]

Failed to set value: Expecting value: line 1 column 2 (char 1)

New value: @payitgov

Failed to set value: Expecting value: line 1 column 1 (char 0)

New value: ''

Failed to set value: Expecting value: line 1 column 1 (char 0)

I then did some searching for examples, and found one here. I edited my meltano.yml directly to add the organizations array:

    - name: tap-github
      variant: meltanolabs
      pip_url: git+https://github.com/MeltanoLabs/tap-github.git@8c3a82464c392b9656bbe384cc1dfe0ca3362392
      config:
        organizations:
          - payitgov

and that worked just fine.

[tayloramurphy]

Looks like AJ opened a related issue to this here as well:

• Bug: Setting array values with --interactive doesn't support common array syntaxes, and doesn't provide a format hint. #7339