Cannot connect to redis://:**@airflow-redis:6379/0: invalid username-password pair or user is disabled..

[kajanajustin]

Official Helm Chart version

1.10.0 (latest released)

Apache Airflow version

2.6.2

Kubernetes Version

1.24

Helm Chart configuration

# Configuration for the redis provisioned by the chart
redis:
  enabled: true
  terminationGracePeriodSeconds: 600

  # Create ServiceAccount
  serviceAccount:
    # Specifies whether a ServiceAccount should be created
    create: true
    # The name of the ServiceAccount to use.
    # If not set and create is true, a name is generated using the release name
    name: ~

    # Annotations to add to worker kubernetes service account.
    annotations: {}

  persistence:
    # Enable persistent volumes
    enabled: true
    # Volume size for worker StatefulSet
    size: 1Gi
    # If using a custom storageClass, pass name ref to all statefulSets here
    storageClassName:
    # Annotations to add to redis volumes
    annotations: {}

  resources: {}
  #  limits:
  #   cpu: 100m
  #   memory: 128Mi
  #  requests:
  #   cpu: 100m
  #   memory: 128Mi

  # If set use as redis secret. Make sure to also set data.brokerUrlSecretName value.
  passwordSecretName: ~

  # Else, if password is set, create secret with it,
  # Otherwise a new password will be generated on install
  # Note: password can only be set during install, not upgrade.
  password: ~

  # This setting tells kubernetes that its ok to evict
  # when it wants to scale a node down.
  safeToEvict: true

  # Select certain nodes for redis pods.
  nodeSelector: {}
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
          - key: kubernetes.io/arch
            operator: In
            values:
            - arm64

  tolerations: []
  topologySpreadConstraints: []

  # Set to 0 for backwards-compatiblity
  uid: 0
  # If not set, `redis.uid` will be used
  securityContext: {}
  #  runAsUser: 999
  #  runAsGroup: 0

  # Detailed default security context for redis for container and pod level
  securityContexts:
    pod: {}
    container: {}

  podAnnotations: {}
# Auth secret for a private registry
# This is used if pulling airflow images from a private registry
registry:
  secretName: ~

  # Example:
  # connection:
  #   user: ~
  #   pass: ~
  #   host: ~
  #   email: ~
  connection: {}

values.yaml specific redis section

Docker Image customizations

No edits or customizations to Dockerfile using the official one from the 310865762107.dkr.ecr.us-east-1.amazonaws.com/apache/airflow and 310865762107.dkr.ecr.us-east-1.amazonaws.com/redis 7-bullseye

What happened

No response

What you think should happen instead

No response

How to reproduce

Set up a new cluster install helm chart version 1.10.0 for airflow, and just set enable Redis to true. It should work with no problem but instead, I get errors such as on the worker pod and the flower pod

ERROR/MainProcess] consumer: Cannot connect to redis://:**@airflow-redis:6379/0: invalid username-password pair or user is disabled..Trying again in 20.00 seconds... (10/100)[2023-09-22 15:33:43,436: 
ERROR/MainProcess] consumer: Cannot connect to redis://:**@airflow-redis:6379/0: invalid username-password pair or user is disabled..Trying again in 22.00 seconds... (11/100)[2023-09-22 15:34:05,480: 
ERROR/MainProcess] consumer: Cannot connect to redis://:**@airflow-redis:6379/0: invalid username-password pair or user is disabled..Trying again in 24.00 seconds... (12/100)

And

File "/home/airflow/.local/lib/python3.7/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors    raise ConnectionError(str(exc)) from exckombu.exceptions.OperationalError: invalid username-password pair or user is disabled.

Anything else

Occurs everytime, need to know what i am doing wrong with my values.yaml

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[kajanajustin]

@potiuk I know you have helped me with issues before, can you please take a look when you get the chance, so strange I am getting this error

[Taragolis]

This looks like troubleshooting rather than actual issue in the Airflow.

Please do not mention specific users without an important need. All contributors have their life, usually someone for community help if have a free time for that. OSS Maintainers/Contributors/PMC is not free tech support. Jarek have a status on slack "On vacations Until Sun October, 8th, 5:00 AM" so let's keep him relax

You could also try ask around in #airflow-aws in the slack channel maybe someone could help or have the same issue in the past.

[Vinaum8]

Can any lifeless maintainer help us with this problem? kkkkkkkkkkkk

[hasland]

I've got this issue sometimes. It happens that if you let the helm chart create a random password sometimes the worker can't get it and start throwing that error. If you just restart Redis and Workers pods it just works. Some workaround is to set some Redis password on helm chart values redis.password.

[Vinaum8]

I'll to test this tomorrow.

[Vinaum8]

I've got this issue sometimes. It happens that if you let the helm chart create a random password sometimes the worker can't get it and start throwing that error. If you just restart Redis and Workers pods it just works. Some workaround is to set some Redis password on helm chart values redis.password.

so, I just deleted the pod from redis and worker.
Shortly after restarting both pods, the connection log appeared:

2024-02-02 12:33:08,835: INFO/MainProcess] Connected to redis://:**@airflow-aks-dataops-redis:6379/0

I'll follow the probes.
Thanks @hasland

[acelinkio]

Following up on this. I had the same thing happen. I am not sure if probes would solve this.

Restarting the worker results in the same error. Redis itself needs to be restarted in order for the worker to connect. Really odd behavior in order to get them to connect.