Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker restart policy "always" not working when pumba kill the container randomly #249

Open
EtherealRise opened this issue Jul 2, 2023 · 0 comments

Comments

@EtherealRise
Copy link

EtherealRise commented Jul 2, 2023

Below is my docker compose file, I added restart policy as always for every service. My expected behavior is that when the chaos engineer kill one of the containers, it would automatically restart. However, it now just exit and never restart. The compose file comes from docker official example voting app, you can try to reproduce it :https://github.com/dockersamples/example-voting-app

Maybe this related to #11, but I have tried both signal KILL and TERM and none of them worked.

The log of pumba looks correct:

./pumba --interval=1m --random -l info kill --signal=SIGTERM "re2:^example-voting-app_vote"
INFO[0000] killing container                             dryrun=false id=5fe93ff11e5a6fbb9dc584848159341bc1fb710c747599fa03276f234a3faa19 name=/example-voting-app_vote_1 
```signal=SIGTERM

and the container log:

vote_1 | WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
vote_1 | * Running on all addresses (0.0.0.0)
vote_1 | * Running on http://127.0.0.1:80

vote_1    |  * Running on http://172.20.0.5:80
vote_1    | Press CTRL+C to quit
vote_1    |  * Restarting with stat
worker_1  | Connected to db
worker_1  | Connecting to redis
worker_1  | Found redis at 172.20.0.2
result_1  | [nodemon] 2.0.22
result_1  | [nodemon] to restart at any time, enter `rs`
result_1  | [nodemon] watching path(s): *.*
result_1  | [nodemon] watching extensions: js,mjs,json
result_1  | [nodemon] starting `node server.js`
vote_1    |  * Debugger is active!
vote_1    |  * Debugger PIN: 861-258-488
result_1  | Sun, 02 Jul 2023 00:48:49 GMT body-parser deprecated bodyParser: use individual json/urlencoded middlewares at server.js:73:9
result_1  | Sun, 02 Jul 2023 00:48:49 GMT body-parser deprecated undefined extended: provide extended option at ../node_modules/body-parser/index.js:104:29
result_1  | App running on port 80
result_1  | Connected to db
example-voting-app_vote_1 exited with code 0

And my compose file

# version is now using "compose spec"
# v2 and v3 are now combined!
# docker-compose v1.27+ required

services:
  vote:
    build: ./vote
    # use python rather than gunicorn for local dev
    command: python app.py
    depends_on:
      redis:
        condition: service_healthy
    healthcheck: 
      test: ["CMD", "curl", "-f", "http://localhost"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 10s
    volumes:
     - ./vote:/app
    ports:
      - "5000:80"
    networks:
      - front-tier
      - back-tier
    restart: "always"

  result:
    build: ./result
    # use nodemon rather than node for local dev
    entrypoint: nodemon server.js
    depends_on:
      db:
        condition: service_healthy 
    volumes:
      - ./result:/app
    ports:
      - "5001:80"
      - "5858:5858"
    networks:
      - front-tier
      - back-tier
    restart: "always"

  worker:
    build:
      context: ./worker
    depends_on:
      redis:
        condition: service_healthy 
      db:
        condition: service_healthy 
    networks:
      - back-tier
    restart: "always"

  redis:
    image: redis:alpine
    volumes:
      - "./healthchecks:/healthchecks"
    healthcheck:
      test: /healthchecks/redis.sh
      interval: "5s"
    networks:
      - back-tier
    restart: "always"

  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_USER: "postgres"
      POSTGRES_PASSWORD: "postgres"
    volumes:
      - "db-data:/var/lib/postgresql/data"
      - "./healthchecks:/healthchecks"
    healthcheck:
      test: /healthchecks/postgres.sh
      interval: "5s"
    networks:
      - back-tier
    restart: "always"

  # this service runs once to seed the database with votes
  # it won't run unless you specify the "seed" profile
  # docker compose --profile seed up -d
  seed:
    build: ./seed-data
    profiles: ["seed"]
    depends_on:
      vote:
        condition: service_healthy 
    networks:
      - front-tier
    restart: "no"

volumes:
  db-data:

networks:
  front-tier:
  back-tier:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant