SIGTERM and container exitting at least once a day

[ghanjiboy]

Subject of the issue

Over the last 2-3 months, I've noticed my vaultwarden container getting a SIGTERM and exiting. I am not terminating it and I don't see any errors or access patterns that would cause it to exit. I have enabled tracing logs to see if it would show something more, but not seeing anything indicative of an issue. The logs are mostly similar to the following excerpt:

[2024-04-05 05:01:38.445][tracing::span][TRACE] parse_headers;
[2024-04-05 05:01:38.445][tracing::span::active][TRACE] -> parse_headers;
[2024-04-05 05:01:38.445][tracing::span::active][TRACE] <- parse_headers;
[2024-04-05 05:01:38.446][tracing::span][TRACE] -- parse_headers;
[2024-04-05 05:01:38.446][request][INFO] GET /alive
[2024-04-05 05:01:38.449][response][INFO] (alive) GET /alive => 200 OK
[2024-04-05 05:01:38.450][tracing::span][TRACE] encode_headers;
[2024-04-05 05:01:38.450][tracing::span::active][TRACE] -> encode_headers;
[2024-04-05 05:01:38.450][tracing::span::active][TRACE] <- encode_headers;
[2024-04-05 05:01:38.450][tracing::span][TRACE] -- encode_headers;
[2024-04-05 05:01:40.027][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-04-05 05:01:40.028][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-04-05 05:02:38.689][tracing::span][TRACE] parse_headers;
[2024-04-05 05:02:38.690][tracing::span::active][TRACE] -> parse_headers;
[2024-04-05 05:02:38.690][tracing::span::active][TRACE] <- parse_headers;
[2024-04-05 05:02:38.690][tracing::span][TRACE] -- parse_headers;
[2024-04-05 05:02:38.690][request][INFO] GET /alive
[2024-04-05 05:02:38.692][response][INFO] (alive) GET /alive => 200 OK
[2024-04-05 05:02:38.692][tracing::span][TRACE] encode_headers;
[2024-04-05 05:02:38.693][tracing::span::active][TRACE] -> encode_headers;
[2024-04-05 05:02:38.693][tracing::span::active][TRACE] <- encode_headers;
[2024-04-05 05:02:38.693][tracing::span][TRACE] -- encode_headers;
[2024-04-05 05:02:40.031][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-04-05 05:02:40.031][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-04-05 05:03:10.034][vaultwarden::api::core::emergency_access][DEBUG] Start emergency_notification_reminder_job
[2024-04-05 05:03:10.041][vaultwarden::api::core::emergency_access][DEBUG] No emergency request reminder notification to send
[2024-04-05 05:03:40.035][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-04-05 05:03:40.035][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-04-05 05:04:40.037][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-04-05 05:04:40.038][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-04-05 05:05:10.040][vaultwarden::api::core::sends][DEBUG] Purging sends
[2024-04-05 05:05:40.042][vaultwarden::api::core::accounts][DEBUG] Purging auth requests
[2024-04-05 05:05:40.043][vaultwarden::api::core::two_factor][DEBUG] Sending notifications for incomplete 2FA logins
[2024-04-05 05:06:11.431][rocket::server][WARN] Received SIGTERM. Requesting shutdown.
[2024-04-05 05:06:11.432][vaultwarden][INFO] Vaultwarden process exited!

Deployment environment

Your environment (Generated via diagnostics page)

• Vaultwarden version: v1.30.5
• Web-vault version: v2024.1.2b
• OS/Arch: linux/x86_64
• Running within a container: true (Base: Debian)
• Environment settings overridden: true
• Uses a reverse proxy: true
• IP Header check: false (X-Forwarded-For)
• Internet access: true
• Internet access via a proxy: false
• DNS Check: true
• Browser/Server Time Check: true
• Server/NTP Time Check: true
• Domain Configuration Check: true
• HTTPS Check: true
• Database type: PostgreSQL
• Database version: PostgreSQL 14.0 (Debian 14.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
• Clients used:
• Reverse proxy and version:
• Other relevant information:

Config (Generated via diagnostics page)

Show Running Config

Environment settings which are overridden: DOMAIN, ADMIN_TOKEN, SMTP_HOST, SMTP_PORT, SMTP_FROM

{
  "_duo_akey": null,
  "_enable_duo": false,
  "_enable_email_2fa": true,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_icon_service_csp": "",
  "_icon_service_url": "",
  "_ip_header_enabled": true,
  "_smtp_img_src": "cid:",
  "admin_ratelimit_max_burst": 3,
  "admin_ratelimit_seconds": 300,
  "admin_session_lifetime": 20,
  "admin_token": "***",
  "allowed_iframe_ancestors": "",
  "attachments_folder": "data/attachments",
  "auth_request_purge_schedule": "30 * * * * *",
  "authenticator_disable_time_drift": false,
  "data_folder": "data",
  "database_conn_init": "",
  "database_max_conns": 10,
  "database_timeout": 30,
  "database_url": "**********://*****************************************************",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": false,
  "disable_icon_download": false,
  "domain": "*****://****************",
  "domain_origin": "*****://****************",
  "domain_path": "",
  "domain_set": true,
  "duo_host": null,
  "duo_ikey": null,
  "duo_skey": null,
  "email_attempts_limit": 3,
  "email_change_allowed": true,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "emergency_access_allowed": true,
  "emergency_notification_reminder_schedule": "0 3 * * * *",
  "emergency_request_timeout_schedule": "0 7 * * * *",
  "enable_db_wal": false,
  "event_cleanup_schedule": "0 10 0 * * *",
  "events_days_retain": null,
  "experimental_client_feature_flags": "fido2-vault-credentials",
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": "***",
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "data/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "icon_redirect_code": 302,
  "icon_service": "internal",
  "incomplete_2fa_schedule": "30 * * * * *",
  "incomplete_2fa_time_limit": 3,
  "invitation_expiration_hours": 120,
  "invitation_org_name": "Bitwarden_RS",
  "invitations_allowed": false,
  "ip_header": "X-Real-IP",
  "job_poll_interval_ms": 30000,
  "log_file": null,
  "log_level": "trace",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "login_ratelimit_max_burst": 10,
  "login_ratelimit_seconds": 60,
  "org_attachment_limit": null,
  "org_creation_users": "",
  "org_events_enabled": false,
  "org_groups_enabled": false,
  "password_hints_allowed": true,
  "password_iterations": 100000,
  "push_enabled": false,
  "push_identity_uri": "https://identity.bitwarden.com",
  "push_installation_id": "***",
  "push_installation_key": "***",
  "push_relay_uri": "https://push.bitwarden.com",
  "reload_templates": false,
  "require_device_email": false,
  "rsa_key_filename": "data/rsa_key",
  "send_purge_schedule": "0 5 * * * *",
  "sendmail_command": null,
  "sends_allowed": true,
  "sends_folder": "data/sends",
  "show_password_hint": false,
  "signups_allowed": true,
  "signups_domains_whitelist": "",
  "signups_verify": false,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_embed_images": true,
  "smtp_explicit_tls": null,
  "smtp_from": "*********************",
  "smtp_from_name": "Bitwarden_RS",
  "smtp_host": "************",
  "smtp_password": null,
  "smtp_port": 25,
  "smtp_security": "off",
  "smtp_ssl": null,
  "smtp_timeout": 15,
  "smtp_username": null,
  "templates_folder": "data/templates",
  "tmp_folder": "data/tmp",
  "trash_auto_delete_days": null,
  "trash_purge_schedule": "0 5 0 * * *",
  "use_sendmail": false,
  "use_syslog": false,
  "user_attachment_limit": null,
  "user_send_limit": null,
  "web_vault_enabled": true,
  "web_vault_folder": "web-vault/",
  "websocket_address": "0.0.0.0",
  "websocket_enabled": false,
  "websocket_port": 3012,
  "yubico_client_id": null,
  "yubico_secret_key": null,
  "yubico_server": null
}

• vaultwarden version:

• Install method:

• I've been using vaultwarden in a docker swarm deployment for at least 3 years - I pull the latest probably shortly after the release is made.

• Clients used:
  I use web client, IOS BW app, Android BW app - all on latest as well.

• Reverse proxy and version:
  nginx

• MySQL/MariaDB or PostgreSQL version:
  PostgreSQL 14.0 (Debian 14.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit

• Other relevant details:

Steps to reproduce

1. Docker swarm takes care of ensuring there is 1 container of vaultwarden running
2. Every day, at different times, at least once, a SIGTERM will be logged and the docker container will exit
3. A new container will spin up shortly after that

Expected behaviour

The container should stay up and running

Actual behaviour

The container abruptly stops.

Troubleshooting data

I am able to look back at logs for up to last 30 days and this is the pattern of SIGTERM occurrences:

04/05/2024 2
04/04/2024 2
04/03/2024 1
04/02/2024 3
04/01/2024 1
03/31/2024 4
03/30/2024 3
03/29/2024 1
03/28/2024 3
03/27/2024 5
03/26/2024 3
03/25/2024 5
03/24/2024 5
03/23/2024 4
03/22/2024 3
03/21/2024 2
03/20/2024 3
03/19/2024 3
03/18/2024 5
03/17/2024 4
03/16/2024 3
03/15/2024 4
03/14/2024 3
03/13/2024 2
03/12/2024 3
03/11/2024 3
03/10/2024 1
03/09/2024 3
03/08/2024 2
03/07/2024 3

[stefan0xC]

SIGTERM is send from outside Vaultwarden so you should check why (i.e. if it's because of a resource limit or something else that might be triggering the signal).

[ghanjiboy]

Thanks @stefan0xC , so since I am running in docker swarm - is it something from docker swarm that would be initiating the SIGTERM?

[stefan0xC]

I don't know and I can't diagnose your system for you. Have you checked your resource usage at the time the SIGTERM has been received? You might just have run out of memory...

[BlackDex]

Check the swarm logs for oomkills or other items related to Vaultwarden.

[ghanjiboy]

Thank you both, I will investigate further - really appreciate the quick responses

[ghanjiboy]

I was not able to find RCA - I moved the docker container to another machine and its been working properly for the last 2 weeks. I think we can close this discussion - thanks again for the assistance.