Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unsuccessful Network Delay experiment keeps running after being paused. Deletion is also problematic #4392

Open
sjafarli opened this issue Apr 8, 2024 · 2 comments

Comments

@sjafarli
Copy link

sjafarli commented Apr 8, 2024

Bug Report

What version of Kubernetes are you using?
Client version: v1.29.3
Server version: v1.27.10

What version of Chaos Mesh are you using?
GitVersion:"chart-2.6.3", GitCommit:"b159d1c5a361ca350ca7789c9eb7fc917479db39", GoVersion:"go1.19.7"}

What did you do? / Minimal Reproducible Example
When networkchaos experiment is not successful (due to large network delay value which was 5m in my experiment), even if the experiment is paused, it keeps running in the background and events being produced consistently (can be seen from the chaos mesh UI).
On an additional note, deletion is also problematic and takes forever. We remove the finalizer from the resource definition and then try to delete the experiment, it works.

What did you expect to see?
When the experiment is paused, it should not produce any more events.

What did you see instead?
Experiment kept running even if it was paused.

Output of chaosctl

@sjafarli sjafarli changed the title Unsuccessful Network Delay experiment with NetworkChaos resource keeps running after being paused. Deletion is also problematic Unsuccessful Network Delay experiment keeps running after being paused. Deletion is also problematic Apr 8, 2024
@STRRL
Copy link
Member

STRRL commented Apr 16, 2024

Hi @sjafarli , could you provide more information to help us address the problem? like:

  • how do you measure the network metrics
  • the log of chaos-daemon
  • chasoctl debug outputs

@sjafarli
Copy link
Author

Hi @STRRL , thanks for the reply.

You can quickly set up a test experiment with network delay and give the delay >= 5 m. This will fail due to the limit on max delay value and the experiment will keep producing failed events over and over again and wont be possible to delete the experiment until you delete the pods which the experiment applied on. Then the experiment can also be deleted. I cannot provide the logs since the chaos project i was doing is pretty much done and I figured out how i can basically workaround it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants