Spec discussion: Give users better control of catalog caching for taps

[aaronsteers]

As raised here:

• Refresh catalog on every invoke (fresh_catalog: true) #2848
• And in slack

Requirement overview

Some taps generate volatile schema, where the upstream source cannot be considered a closed system or a governed/stable resource. For instance, as a non-admin user on a SQL Server database, I might reasonably assume that my catalog will not change from one minute to the next, but on any given day new tables could be added and existing tables could have their schemas modified - all without notification.

Options

Users need a way to do one or more of the following:

1. explicitly force a refresh of a catalog,
2. explicitly invalidate the catalog cache,
3. define an expiration period for a tap's catalog (for instance 12 or 24 hours), after which time the catalog will be considered stale, and/or
4. specify that a tap's catalog should simply not be cached.

These are all subtly different takes on the same core problem.

Best workaround as of today

As of today, the only real option is for the user to manually delete the contents of the catalog, for instance by running something similar to: rm .meltano/run/tap-name/*.

This is problematic because it is cumbersome but also because we don't want users to have to understand or navigate the internal .meltano directory by hand.

[aaronsteers]

Option 1: Explicitly forcing the refresh of a catalog

A new catalog command with the subcommand update (or refresh, or similar) could explicitly rerun discovery.

# refresh the specific extractor's catalog before running EL
meltano catalog update tap-name
meltano run tap-name target-name

Or:

# refresh all catalogs
meltano catalog update --all
meltano run tap-name target-name

[edgarrmondragon]

The select command already works as an interface to the catalog, so perhaps the new sub-command can be added there? My concern is that catalog functionality sprawls across multiple methods.

[aaronsteers]

This is a good point and I want to give it some more thought... I wonder if select would make sense refactored as a subcommand of meltano catalog.

Adapting slightly from https://docs.meltano.com/reference/command-line-interface#select:

meltano catalog select [--list] [--all] <tap_name> [ENTITIES_PATTERN] [ATTRIBUTE_PATTERN]

[aaronsteers]

@edgarrmondragon - While in the long run, I like the idea of everything catalog related living under a top-level meltano catalog command, this is not going to happen fast enough.

I logged an issue following on your proposal here:

• feature: Add a --refresh option for meltano select to refresh a new copy of the discovered catalog #6919

[aaronsteers]

Option 2: Explicitly forcing the invalidation of a catalog

A new catalog command with the subcommand delete (or clear, remove, rm, or similar) could explicitly delete the cached catalog.

# Delete the specific extractor's catalog before running EL
meltano catalog clear tap-name
# Invoking the tap will force a catalog refresh
meltano run tap-name target-name

Or:

# Delete all catalogs
meltano catalog update --all
# Invoking the tap will force a catalog refresh
meltano run tap-name target-name

[aaronsteers]

Option 3: Specify max time-to-live for a catalog cache

meltano config tap-name set catalog_max_age '2 hr'

plugins:
  extractors:
  - name: tap-name
    catalog_max_age: 2 hr
    config:
      // ...

[visch]

Option 3 and Option 3++ are my favorites :)

[aaronsteers]

Option 4: Specify a tap's catalog should not be cached

meltano config tap-name set _catalog_caching disabled

plugins:
  extractors:
  - name: tap-name
    catalog_caching: disabled
    config:
      // ...

[aaronsteers]

On thinking about this more, we technically do have to cache the file - but presumably we'd delete it / clean it up before and after each invocation? UPDATE: Or store it at a path specific to the particular execution, not global to the tap.

meltano config tap-name set _catalog_caching disabled

# This invokes `--discover` and cleans up before and after execution
meltano select tap-name --list

# This invokes `--discover` again, cleaning up the catalog before and after execution
meltano run tap-name target-name

# This invokes `--discover` again as well
meltano invoke tap-name --dump=catalog

We could run into a little bit of trouble if the tap is invoked twice at the same time, but this is already something we check for - at least for test runs.

Probably a better approach than deleting the file afterward is (when caching is disabled) to store the file with a subfolder path or file name specific to that execution. This approach is safer than simply deleting afterwards, since in theory multiple simultaneous invocations could each have their own run-results subfolder and they would not impact each other.

[visch]

@aaronsteers could copy what happens right now with config files where we add a UUID so we don't have an issue with parallelization. So it'd be something like properties-{{uuid}}.json

[aaronsteers]

@visch - Definitely! Yeah, I think that would work!

[aaronsteers]

Related issue came up today:

Having some means of disabling caching also seems like a necessary safety measure to unblock :

• feature: add support for env var interpolation in select blocks #6770

[pnadolny13]

This came up in slack again today https://meltano.slack.com/archives/C01TCRBBJD7/p1680249669517619