Gunicorn worker getting exit in airflow web server with Received signal: 15. Closing gunicorn

[indrajeetsinghthakur750]

Gunicorn worker getting exit in airflow web server with Received signal: 15. Closing gunicorn. while installing airflow on kubernetes.

Scheduler and other DB migration and user sync pod are running fine only web server pod are restarting with below error.

Here some details :

--------------------
Airflow version       : 2.6.3
Airflow Chart version : 8.8.0
Executor              : KubernetesExecutor
Airflow Image         : apache/airflow:2.6.3-python3.10
Chart repo            : https://github.com/airflow-helm/charts.git

What could be the issue we have only changes some configuration like Executor to KubernetesExecutor and DB detail. in same environment airflow version 2.2.3 working with same configuration.

[2023-09-13 10:48:36 +0000] [46] [INFO] Listening at: https://0.0.0.0:8080 (46)
[2023-09-13 10:48:36 +0000] [46] [INFO] Using worker: sync
[2023-09-13 10:48:36 +0000] [154] [INFO] Booting worker with pid: 154
[2023-09-13 10:48:36 +0000] [155] [INFO] Booting worker with pid: 155
[2023-09-13 10:48:36 +0000] [156] [INFO] Booting worker with pid: 156
[2023-09-13 10:48:36 +0000] [157] [INFO] Booting worker with pid: 157
[2023-09-13 10:49:04 +0000] [157] [INFO] Worker exiting (pid: 157)
[2023-09-13 10:49:04 +0000] [155] [INFO] Worker exiting (pid: 155)
[2023-09-13 10:49:04 +0000] [156] [INFO] Worker exiting (pid: 156)
[2023-09-13 10:49:04 +0000] [154] [INFO] Worker exiting (pid: 154)
[2023-09-13 10:49:04 +0000] [46] [INFO] Handling signal: term
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat: 
=================================================================
[2023-09-13T10:49:04.697+0000] {webserver_command.py:432} INFO - Received signal: 15. Closing gunicorn.
[2023-09-13 10:49:07 +0000] [46] [INFO] Shutting down: Master

[thesuperzapper]

@indrajeetsinghthakur750 check the Kubernetes events on the Pods that are being killed.

It is almost certainly Kubernetes itself killing the Pod, which could be for a number of reasons:

1. The pod is failing its liveness check
2. The pod is running on a Node which is being shut down
3. The pod is exceeding its memory limits

[indrajeetsinghthakur750]

Checked in events : and showing this connection refuse after I try with increasing liveness still same

54m Normal Created pod/my-airflow-cluster-new-web-ff457bc6f-p6dnv Created container airflow-web
51m Normal Started pod/my-airflow-cluster-new-web-ff457bc6f-p6dnv Started container airflow-web
55m Warning Unhealthy pod/my-airflow-cluster-new-web-ff457bc6f-p6dnv Liveness probe failed: Get "https://[------]:8080/health": dial tcp [removing ip for security]:8080: connect: connection refused
54m Warning Unhealthy pod/my-airflow-cluster-new-web-ff457bc6f-p6dnv Readiness probe failed: Get "https://[removing ip for security]:8080/health": dial tcp [removing ip for security]:8080: connect: connection refused
55m Normal Killing pod/my-airflow-cluster-new-web-ff457bc6f-p6dnv Container airflow-web failed liveness probe, will be restarted
56m Normal SuccessfulCreate replicaset/my-airflow-cluster-new-web-ff457bc6f Created pod: my-airflow-cluster-new-web-ff457bc6f-p6dnv

[thesuperzapper]

@indrajeetsinghthakur750 you should try running a curl inside the pod to see what the actual error is when you try and access the liveness probe URL at https://127.0.0.1:8080/health.

I wonder if it's some kind of HTTPS error, ensure your AIRFLOW__WEBSERVER__WEB_SERVER_SSL_CERT and AIRFLOW__WEBSERVER__WEB_SERVER_SSL_KEY configs are both correctly set, and point to a valid certificate.

[indrajeetsinghthakur750]

@thesuperzapper tls.key and tls.cert has already configured.
AIRFLOW__WEBSERVER__WEB_SERVER_SSL_CERT: "/opt/airflow/secrets/ssl/tls.crt"
AIRFLOW__WEBSERVER__WEB_SERVER_SSL_KEY: "/opt/airflow/secrets/ssl/tls.key"

[indrajeetsinghthakur750]

We have install without trigger

CURL RESPONSE

`

GET /health HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: curl/7.74.0
Accept: /

• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• old SSL session ID is stale, removing
• Mark bundle as not supporting multiuse
  < HTTP/1.1 200 OK
  < Server: gunicorn
  < Date: Mon, 18 Sep 2023 10:16:20 GMT
  < Connection: close
  < Content-Type: application/json
  < Content-Length: 247
  < X-Robots-Tag: noindex, nofollow
  <
• Closing connection 0
• TLSv1.3 (OUT), TLS alert, close notify (256):
  {"metadatabase": {"status": "healthy"}, "scheduler": {"latest_scheduler_heartbeat": "2023-09-14T08:17:29.070525+00:00", "status": "unhealthy"}, "triggerer": {"latest_triggerer_heartbeat": "2023-09-13T08:26:18.464913+00:00", "status": "unhealthy"}}
  `

[npgretz]

I faced the same issue but with Apache's Airflow Helm Chart ( Airflow 2.8.2 and Chart 1.3.0 ). I had this issue randomly occurring in my development deployment but not my production deployment. I believe the issue is related to Kubernetes killing the Webserver pod because it fails its liveness check like was mentioned above.

Apache's Airflow Helm Chart has a Kubernetes Startup Probe on the Webserver pod so that Kubernetes does not start up the liveness checks while the pod is still starting up and then remove the pod because it is failing the check due to still being in the process of booting up. For my development deployment, I had to increase the startup probe's delay so the Webserver pod had more time to boot before the liveness checks started.

I do not believe the community chart includes a startup probe so in the templates for the webserver deployment under spec.template.spec.containers, you can add:

( NOT TESTED CODE!!! Simply my best guess at the template for the community helm chart. An easy setup for the startup probe would just use the same http endpoint as the liveness probe.)

       {{- if .Values.web.livenessProbe.enabled }}
          startupProbe:
            initialDelaySeconds: {{ .Values.web.livenessProbe.initialDelaySeconds }}
            periodSeconds: {{ .Values.web.livenessProbe.periodSeconds }}
            timeoutSeconds: {{ .Values.web.livenessProbe.timeoutSeconds }}
            failureThreshold: {{ .Values.web.livenessProbe.failureThreshold }}
            successThreshold: 1
            httpGet:
              scheme: {{ include "airflow.web.scheme" . }}
              {{- $airflowUrl := .Values.airflow.config.AIRFLOW__WEBSERVER__BASE_URL | default "" | printf "%s/health" | urlParse }}
              path: {{ get $airflowUrl "path" }}
              port: web
          {{- end }}

The periodSeconds will be multiplied by the failureThreshold to determine how many seconds the startup probe will delay the liveness checks by.