ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler'

[dachosen1]

I'm getting the above error after running the below compose file in a Linux VM

version: '3.4'

x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.1}
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
    AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
  user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
  env_file: .env
  depends_on:
    - redis
    - postgres


services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: [ "CMD", "pg_isready", "-U", "airflow" ]
      interval: 5s
      retries: 5
    restart: always

  redis:
    image: redis:latest
    ports:
      - 6379:6379
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 30s
      retries: 50
    restart: always

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 8080:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    healthcheck:
      test:
        - "CMD-SHELL"
        - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-init:
    <<: *airflow-common
    command: version
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}

  flower:
    <<: *airflow-common
    command: celery flower
    ports:
      - 5555:5555
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

volumes:
  postgres-db-volume:

Full Error Log:

airflow-scheduler_1  |
airflow-worker_1     | ....................
airflow-worker_1     | ERROR! Maximum number of retries (20) reached.
airflow-worker_1     |
airflow-worker_1     | Last check result:
airflow-worker_1     | $ airflow db check
airflow-worker_1     | Unable to load the config, contains a configuration error.
airflow-worker_1     | Traceback (most recent call last):
airflow-worker_1     |   File "/usr/local/lib/python3.6/pathlib.py", line 1248, in mkdir
airflow-worker_1     |     self._accessor.mkdir(self, mode)
airflow-worker_1     |   File "/usr/local/lib/python3.6/pathlib.py", line 387, in wrapped
airflow-worker_1     |     return strfunc(str(pathobj), *args)
airflow-worker_1     | FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/logs/scheduler/2021-07-04'
airflow-worker_1     |
airflow-worker_1     | During handling of the above exception, another exception occurred:
airflow-worker_1     |
airflow-worker_1     | Traceback (most recent call last):
airflow-worker_1     |   File "/usr/local/lib/python3.6/logging/config.py", line 565, in configure
airflow-worker_1     |     handler = self.configure_handler(handlers[name])
airflow-worker_1     |   File "/usr/local/lib/python3.6/logging/config.py", line 738, in configure_handler
airflow-worker_1     |     result = factory(**kwargs)
airflow-worker_1     |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 46, in __init__
airflow-worker_1     |     Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True)
airflow-worker_1     |   File "/usr/local/lib/python3.6/pathlib.py", line 1252, in mkdir
airflow-worker_1     |     self.parent.mkdir(parents=True, exist_ok=True)
airflow-worker_1     |   File "/usr/local/lib/python3.6/pathlib.py", line 1248, in mkdir
airflow-worker_1     |     self._accessor.mkdir(self, mode)
airflow-worker_1     |   File "/usr/local/lib/python3.6/pathlib.py", line 387, in wrapped
airflow-worker_1     |     return strfunc(str(pathobj), *args)
airflow-worker_1     | PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler'
airflow-worker_1     |
airflow-worker_1     | During handling of the above exception, another exception occurred:
airflow-worker_1     |
airflow-worker_1     | Traceback (most recent call last):
airflow-worker_1     |   File "/home/airflow/.local/bin/airflow", line 5, in <module>
airflow-worker_1     |     from airflow.__main__ import main
airflow-worker_1     |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 46, in <module>
airflow-worker_1     |     settings.initialize()
airflow-worker_1     |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 446, in initialize
airflow-worker_1     |     LOGGING_CLASS_PATH = configure_logging()
airflow-worker_1     |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 73, in configure_logging
airflow-worker_1     |     raise e
airflow-worker_1     |   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/logging_config.py", line 68, in configure_logging
airflow-worker_1     |     dictConfig(logging_config)
airflow-worker_1     |   File "/usr/local/lib/python3.6/logging/config.py", line 802, in dictConfig
airflow-worker_1     |     dictConfigClass(config).configure()
airflow-worker_1     |   File "/usr/local/lib/python3.6/logging/config.py", line 573, in configure
airflow-worker_1     |     '%r: %s' % (name, e))
airflow-worker_1     | ValueError: Unable to configure handler 'processor': [Errno 13] Permission denied: '/opt/airflow/logs/scheduler'

[potiuk]

Did you follow the https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#initializing-environment instructions?

mkdir ./dags ./logs ./plugins
echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env

[dachosen1]

it worked thanks.

[Mahmoud-Elbahnasawy]

What if i am on windows ?, how to solve it

[Taragolis]

If you run under the WSL 2 then the solution should be the same.
If you directly want to mount directories from Windows to Docker it might require additional permissions (and also it might be slow), so better run in under WSL 2

[Mahmoud-Elbahnasawy]

@Taragolis Thanks

[fermat01]

@Mahmoud-Elbahnasawy , if you are using windows just install WSL2 with Ubuntu or Debian. And install docker on your VM .
Here some ressources

• Install WSL2 : https://learn.microsoft.com/en-us/windows/wsl/install
• Install Docker on WSL2 : https://dev.to/0xkoji/install-docker-on-wsl2-2ma5

Hope this helps.