You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Please vote on this issue by adding a 馃憤 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
Tell us about your request
The new StopTimeout that allows us to set interval between SIGTERM and SIGKILL to be longer than 2 minutes is great.
We set it for 24 hours for our batch jobs and it works great.
However, we have run into issues where we know that that containers have received a SIGTERM and started draining (Desired State: Stopped), but the container has run into a busy wait loop (looking at their logs).
In these cases we want to "Force stop" the container/task.
It would be nice to have this features within the ECS gui under "ECS->->Tasks" as "Stop Selected Forced".
Right now we way we deal with this problem is with automation use SSM to kill the offending containers using the "docker kill" command.
Reference: https://aws.amazon.com/blogs/containers/improvements-to-amazon-ecs-task-launch-behavior-when-tasks-have-prolonged-shutdown/
Which service(s) is this request for?
ECS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Help us and other customers better manage rogue tasks which need to be force killed before "stopTimeout" has expired.
Are you currently working around this issue?
How are you currently solving this problem?
We built a helper program in python that finds tasks where desiredStatus=="STOPPED" and "lastStatus" == "RUNNING", find the underlying containers and ec2 instances and then use SSM to run "docker kill" on each server.
Community Note
Tell us about your request
The new StopTimeout that allows us to set interval between SIGTERM and SIGKILL to be longer than 2 minutes is great.
We set it for 24 hours for our batch jobs and it works great.
However, we have run into issues where we know that that containers have received a SIGTERM and started draining (Desired State: Stopped), but the container has run into a busy wait loop (looking at their logs).
In these cases we want to "Force stop" the container/task.
It would be nice to have this features within the ECS gui under "ECS->->Tasks" as "Stop Selected Forced".
Right now we way we deal with this problem is with automation use SSM to kill the offending containers using the "docker kill" command.
Reference: https://aws.amazon.com/blogs/containers/improvements-to-amazon-ecs-task-launch-behavior-when-tasks-have-prolonged-shutdown/
Which service(s) is this request for?
ECS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Help us and other customers better manage rogue tasks which need to be force killed before "stopTimeout" has expired.
Are you currently working around this issue?
How are you currently solving this problem?
We built a helper program in python that finds tasks where desiredStatus=="STOPPED" and "lastStatus" == "RUNNING", find the underlying containers and ec2 instances and then use SSM to run "docker kill" on each server.
Additional context
If you want to discuss this issue, you can reach out to me at ashish.desai@zillasecurity.com
The text was updated successfully, but these errors were encountered: