Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: Add a --refresh option for meltano select to refresh a new copy of the discovered catalog #6919

Open
aaronsteers opened this issue Oct 19, 2022 · 7 comments · May be fixed by #8518
Open

feature: Add a --refresh option for meltano select to refresh a new copy of the discovered catalog #6919

aaronsteers opened this issue Oct 19, 2022 · 7 comments · May be fixed by #8518
Labels
Accepting Pull Requests kind/Feature

Comments

@aaronsteers
Copy link
Contributor

aaronsteers commented Oct 19, 2022 •
edited

This feature would add a --refresh option, or similar CLI arg to the meltano select CLI command. https://docs.meltano.com/reference/command-line-interface#select

# Refresh the cached catalog by re-running discovery
meltano select tap-github --refresh

# Print the normal --list output *after* refreshing the catalog
meltano select tap-github --list --refresh

This would give users the ability to force a refresh of the cached catalog.json file.

Related
This was referenced Oct 19, 2022
Closed
Open
@pnadolny13 pnadolny13 mentioned this issue Jan 5, 2023
Closed
@pnadolny13 pnadolny13 mentioned this issue Apr 11, 2023
Merged
3 tasks
@stale
Copy link

stale bot commented May 2, 2023

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

@stale stale bot added the stale label May 2, 2023
@stale stale bot closed this as completed May 24, 2023
@TyShkan
Copy link

TyShkan commented Dec 13, 2023

Since the ‎SingerTap.catalog_cache_key explicitly‎ ignores selection filter rules update, this feature would be nice to have instead of manual unlinking of .meltano/run dir before running select --list

@stale stale bot removed the stale label Dec 14, 2023
@edgarrmondragon
Copy link
Collaborator

For anyone interested in taking a looking at this, the caching logic lives in

meltano/src/meltano/core/plugin/singer/tap.py

Lines 362 to 371 in ab6e2d9

catalog_cache_key_path = plugin_invoker.files["catalog_cache_key"]
if catalog_path.exists():
with suppress(FileNotFoundError):
cached_key = catalog_cache_key_path.read_text()
new_cache_key = self.catalog_cache_key(plugin_invoker)
if cached_key == new_cache_key:
logger.debug("Using cached catalog file")
return
logging.debug("Cached catalog is outdated, running discovery...")

@edgarrmondragon
Copy link
Collaborator

Another user ran into this, with tap-salesforce this time: https://www.linen.dev/s/meltano/t/16176363/tap-salesforce-not-adding-new-columns#242718da-e029-470b-a9bb-37e110669d4d.

@isoctcolo
Copy link

isoctcolo commented Dec 19, 2023 •
edited

Hello all,
It was me who ran into the issue. Is this --refresh command an acceptable solution to be taken into consideration ?
As described above I think the refresh would be enough. As I understood the list takes then the cached catalog anyway.

@edgarrmondragon
Copy link
Collaborator

Hello all, It was me who ran into the issue. Is this --refresh command an acceptable solution to be taken into consideration ? As described above I think the refresh would be enough. As I understood the list takes then the cached catalog anyway.

@isoctcolo I think it is acceptable. The idea is that the --refresh flag would force the catalog to be refreshed, disregarding whether a catalog exists or the cache key matches.

@isoctcolo
Copy link

Ok I dig into the code - could take me some time but for me it is good practice to get to know meltano better

@andyoneal andyoneal linked a pull request Apr 29, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Accepting Pull Requests kind/Feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Ignore Catalog Cache andyoneal/meltano
4 participants
@TyShkan @edgarrmondragon @aaronsteers @isoctcolo