Consider a leaner Meltano Project scaffold

[aaronsteers]

aj@AJs-MacBook-Pro Source % meltano init test-project
Creating .meltano folder
created .meltano in /Users/aj/Source/test-project/.meltano
Creating project files...
  test-project/
   |-- meltano.yml
   |-- README.md
   |-- requirements.txt
   |-- output/.gitignore
   |-- .gitignore
   |-- extract/.gitkeep
   |-- load/.gitkeep
   |-- transform/.gitkeep
   |-- analyze/.gitkeep
   |-- notebook/.gitkeep
   |-- orchestrate/.gitkeep

Counting the .meltano system directory and the plugins directory (not shown below), this is a total of 9 directories, 7 of which are empty besides the .gitkeep files.

I think we can safely remove these:

• /requirements.txt
• /analyze/.gitkeep
• /notebook/.gitkeep

We could optionally drop these as well from the first-run experience, if the parent folder would get added as part of the scaffold for dbt and Airflow, respectively.

• /transform/.gitkeep
• /orchestrate/.gitkeep

Then, the new user experience would create a leaner scaffold with less wasted attention on empty dirs.

  test-project/
   |-- .meltano/
   |-- extract/
   |-- load/
   |-- output/
   |-- transform/
   |-- plugins/
   |-- ./ # 
      |-- .gitignore
      |-- meltano.yml
      |-- README.md

Additionally, I think even for directories we will keep, we can replace .gitkeep files with a README.md that explains how each subdirectory can be used.

Regarding the output/ folder, I think we added this because target-csv and/or target-jsonl would fail in tutorials if the folder did not exist. Now, I think one or both have since added "add directory if not exists" behavior, so we can consider removing that output folder as well. If any of our default targets does not auto-create the base directory, we should just add that behavior anyway.

My personal preference would be to remove output directory from the scaffold, but add output (and/or .output) to the default root .gitignore file. Many types of projects automatically hide their build results and temporary output files, so for Meltano to also default a folder called 'output' to ignored in Git, I think that would be a very reasonable default - which users could disable if they really wanted to.

[edgarrmondragon]

There's also the plugins/ directory for lock files

[aaronsteers]

Good catch. I'll update the above to include this...

[aaronsteers]

Updated! It occurs to me that another approach here would be to use plugins as the recommended place for extractors and `loaders. I could see two approaches working:

Lock files and -meltano.yml files as siblings.

In the squared project, if I move everthying from ./load into ./plugins/loaders and everything from ./extract into ./plugins/extractors, then we have something like this:

Users who want this could do so today by simply pointing their include_paths to something like /plugins/**/*-meltano.yml.

Personally, I really like this approach. It cleans up the root directory a lot while still giving users control if they want to spin things to top-level folders. It will clean itself up even further as dbt, airflow, etc. all become utilities type.

Out of box with Meltano plugin config, users can already state their preference for DBT_PROJECT_ROOT being any of these or anything completely different: $MELTANO_PROJECT_DIR/transform (today's default), $MELTANO_PROJECT_DIR/plugin/transformers, $MELTANO_PROJECT_DIR/plugin/transformers/dbt, or something else.

Lock files in a separate path /plugins/lock

This approach admittedly would take some refactoring and some thinking about backwards compatibility, and might not be worth the extra effort.

[aaronsteers]

Here's what the squared repo looks like if I combine everything under /plugins:

[tayloramurphy]

@aaronsteers I'm in favor of simplifying the starting experience. While it's possibly nice that we show users they can do more with Meltano, it can be distracting and it's not necessary early in the project lifecycle.

I don't like dropping the meltano.yml to a ./ folder though. That's the main part of the project and we shouldn't bury it.

Going even further - do we really need even the extract, load, and transform folders? Those can be added as needed, but even when you add an extractor or a loader nothing goes in there. Unless we're seeding it with extractors-meltano.yml then it's not used I think.

Would it be crazy if we did something like this? @sbalnojan @DouweM what do you think?

  test-project/
   |-- .meltano/
   |-- meltano.yml
   |-- README.md
   |-- .gitignore
   |-- output/      
   |-- plugins/

One other thing we'd need to check - I think some of the simpler targets might write to /output by default - we should just make sure they can create the folder if needed.