Spec discussion: new meltano log CLI

[aaronsteers]

We would love to have a logging feature accessible via the CLI.

meltano log list (raw list of executions)

List all the available logs (up to max count 'n'). Log list can be filtered by schedule name, job name, plugin name, or state ID.

meltano log list [PLUGIN_NAME|SCHEDULE_NAME|JOB_NAME|STATE_ID] [--order=ASC|DESC] --limit=n

meltano log summary (deduped/aggregated)

Would show a summary of all job and schedule names which have executed over the specified local period.

Summary should have columns for: latest execution start/end time, latest duration, latest log ID, latest job status (success/fail), and number of failures and success within the lookback period

meltano log summary [--lookback=4d]

meltano log print

Print a full log and exit.

# Print a specific already known log-id.
meltano log print --log-id=LOG_ID [--tail=n] [--head=n]

# Same as above but auto-lookup the latest known log-id from the provided job or schedule name
meltano log print [SCHEDULE_NAME|JOB_NAME|STATE_ID] [--tail=n] [--head=n]

meltano log tail

A more advanced version of log print. Start by printing 'n' log lines and continue to print any additional lines as they are emitted to the log.

# Tail a specific already known log-id.
meltano log tail --log-id=LOG_ID [--lines=n]

# Same as above but auto-lookup the latest known log-id from the provided job or schedule name
meltano log tail [SCHEDULE_NAME|JOB_NAME|STATE_ID] [--tail=n] [--head=n]

cc @tayloramurphy, @pandemicsyn

Log filtering

In addition to a GUID-like field for unique Log ID, we would ideally also want to be able to filter logs by the following fields:

1. Job Name
2. Schedule Name
3. State ID
4. Environment Name
5. Execution Context ID
6. Plugin Name

To aid in filtering, we should ensure that log tables record all of these fields if possible as separate columns and (of the backend supports it) perhaps also as indexed columns.

Scalability and Remote Backend Support

We likely could first build this using systemdb as the backend for log summary & list operations and the local filesystem as the backend for detailed logs.

In future, we'd likely also want to support log backends using S3 and/or Cloudwatch, and we would want to also support a logging backend hosted on the future Meltano Cloud offering,

GUI and "Point and Click" option

By adding a --format=json CLI flag or similar to most/all of the above CLI commands, we'd also then be able to call them and render a nice UI in the future VS Code Meltano Extension.

Other UIs could similarly be developed, but the VS Code option could be a very nice in-IDE log view experience, and could be run in the cloud using GitHub Codespaces.

Related to:

• Spec discussion: Interface to show running jobs, kill jobs, and print job logs #6595

[tayloramurphy]

@aaronsteers Apologies for the delayed response here.

meltano log summary

In thinking more on this one, I'm less a fan because it's trying to do too much.

Would show a summary of all job and schedule names which have executed over the specified local period.

I'd rather we add more to meltano log list to have the default be all logs with the option to do the lookback via time period and via total number.

# list last 10 available logs across all schedule / job / run. assume default limit=10
meltano log list 

# list last 10 available logs across schedule / job / run. still limit to 10
meltano log list --lookback=1d

# list last 50 or so available logs across schedule / job / run
meltano log list --lookback=1d --limit=50

# list last 10 logs available for specified optoin
meltano log list [PLUGIN_NAME|SCHEDULE_NAME|JOB_NAME|STATE_ID] [--order=ASC|DESC] --limit=n

Seems like here we'd also want to allow for filtering by either jobs or schedules - not sure how to do that either.

---

As for the summary command, I think we'd want a different command. Thinking more about what the user is trying to do, they're likely not trying to get a summary of all of the logs - they probably want a summary of their deployment and overall runs. What about something like this:

# show an overall summary of the Meltano deployment for the past day. default to "all"
meltano status

# show list of all jobs with their state (running, finished, pending, etc) with the relevant id (basically the columns you mentioned previously)
meltano status jobs [--job-name=x]

# similar to above
meltano status schedules [--schedule-name=x]

What I like about having status (or something else) is it can be a homebase for users when they come to the command line. Unsure what to do? Run meltano status and we can point them into a few different directions (login if they're not already, check running jobs, etc).

---

Otherwise, I like the overall direction for the other log commands. Thoughts?

[aaronsteers]

Lookback limits vs Last-N limits

I'd rather we add more to meltano log list to have the default be all logs with the option to do the lookback via time period and via total number.

My preference is to use a time threshold as the auto-limit default, here and in status/summary, because (1) job invocations are not evenly distributed. I think the time-based default is the least likely to miss the user's intent. We may show too many or too few versus what they wanted, but we default to probably including the record they were looking for. For example, you might have some jobs that run every 5 minutes, or you might have recently been debugging and the most recent 50 log IDs are from fail/retry iterations. I feel like at minimum, its a good idea to make sure we include all the jobs from the last 24 hours by default.

That said, as long as we have both options as CLI args, I feel we can iterate as we go on default filter logic. I don't feel a strong need to get those defaults fully ironed out at this stage. I think we're in agreement that both are valuable: a time-based lookback ("I know I ran this today or yesterday but I don't have the exact log ID") as well as a 'most recent n jobs' lookback ("I just ran this a few minutes ago but I don't remember the job ID").

Probably a good thing to specify though here, is that whatever the default filer is, it should not intersect with a manually specified filter. So, if I explicitly request all jobs from the last 24 hours, don't limit to 10 and make me also override the count. And if I explicitly request 50 logs, don't apply a 24 filter.

status vs summary

I like your status idea better than summary. What do you think of meltano job status and meltano schedule status?

We could perhaps also have a top-level meltano status that prints the output of meltano job status and also meltano schedule status, plus any kind of meltano debug we create in the future or something like meltano compile --warnings=yes to do sanity checks on certain project assumptions. A top level meltano status could also print out the latest test result of each plugin - like meltano test --all except using cached results where available.

Status of schedules vs status of jobs and other bespoke executions.

Of all the things we're able to view and summarize logs of, I think schedules are the primary thing to be checked up on; presumably job failures that are not also run from schedules would be debugged at the time - whereas there is a real need to understand and summarize unattended schedule workloads and to have a 'dashboard'-type interface to summarize which are failing and what the latest and aggregate status is of each. This also would encourage users to configure schedules for things that they are "scheduling". Although technically speaking, they can invoke jobs any way they want, the dashboard paradigm really only works if we're talking about recurring schedules.

There might be a job that's only been run once called 'rebuild-from-scratch' - but it wouldn't be a scheduled job and it wouldn't likely have any history in the past several months. When we want to know the status of the meltano project, we can safely ignore that job. But anything scheduled would naturally have relevance to the overall project health.

In thinking more on this one (meltano log summary), I'm less a fan because it's trying to do too much.

Do you mean the summaries are too much or because it tries to cover jobs, schedules, and everything else all in one command?

I would agree that summarizing all logs from all sources probably doesn't make sense. (Per the above, I think status of jobs/schedules does make sense. And as a high-ish priority for Q1/Q2 timeline, the schedule status is likely going to be necessary for a human readable project health status. Specifically, this could answer: "did anything break?" or "what broke?" - whereas meltano log list won't be able to provide that in a human-readable way.

That said, this would probably be built last - on top of meltano log list and after meltano log print. We might build this before tail - but only because tail could end up being technically challenging, depending on which backends we need to support.

[tayloramurphy]

What do you think of meltano job status and meltano schedule status?

That makes sense to me and fits better with a narrowed scope. We don't have to have a top-level meltano status in the first iteration, but knowing we want to head towards that can help th eoverall design.

Do you mean the summaries are too much or because it tries to cover jobs, schedules, and everything else all in one command?

Just that it tries to cover everything in one command and it's not really clear from the command what I'm actually going to get. I like have the more scoped versions first and then we can build the more generic ones after some testing.

That said, this would probably be built last - on top of meltano log list and after meltano log print

Yep, agreed - just wanted to make sure we were mostly headed in the same direction @aaronsteers 😄

[aaronsteers]

Awesome! 🚀 😄

Yeah, I agree on all the above.