Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vmalert: tolerate missing rules file when loading from object storage #6209

Open
hagen1778 opened this issue Apr 30, 2024 · 0 comments
Open
Assignees
Labels
bug Something isn't working enterprise Enterprise issue, not included in free open source version vmalert

Comments

@hagen1778
Copy link
Collaborator

hagen1778 commented Apr 30, 2024

Describe the bug

When loading rule files from object storage vmalert performs to subsequent actions:

  1. List all files in configured bucket. Having the list allows prefix matching by file name, to select only specific files to load.
  2. Read files that match filters from the p1.

Because p1 and p2 could happen in different moment of times (seconds usually), it could happen that a file was present at p1 stage and was deleted before p2 stage. In this case, vmalert will produce an error and set configuration_success flag to false.

vmalert should skip files that do not exist.

To Reproduce

Create a bucket with big number of rule files and trigger config reload in vmalert. Try deleting random rules file from bucket while vmalert is still in the process of reloading the config.

Version

All versions that support https://docs.victoriametrics.com/vmalert/#reading-rules-from-object-storage

@hagen1778 hagen1778 added bug Something isn't working enterprise Enterprise issue, not included in free open source version vmalert labels Apr 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enterprise Enterprise issue, not included in free open source version vmalert
Projects
None yet
Development

No branches or pull requests

2 participants