Airflow 2.8.1 pods are not coming up.

[infa-kvaibhav]

We are upgrading airflow from 2.5.0 to 2.8.1
Following exception coming from schedule service and from the migration:-
Also our airflow home is set airflowHome: /opt/airflow in values.yaml

[centos@IICS-NP-MGMT-NODE ~]$ kubectl logs oi-opsinsights-airflow-run-airflow-migrations-dck-grk8q
/usr/local/lib/python3.9/site-packages/airflow/configuration.py:812 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
/usr/local/lib/python3.9/site-packages/airflow/configuration.py:738 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
/usr/local/lib/python3.9/site-packages/airflow/settings.py:194 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/lib64/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/airflow/logs/scheduler/2024-03-12'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/airflow/logs/scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/airflow/logs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.9/logging/config.py", line 564, in configure
handler = self.configure_handler(handlers[name])
File "/usr/lib64/python3.9/logging/config.py", line 745, in configure_handler
result = factory(**kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/utils/log/file_processor_handler.py", line 50, in init
Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True)
File "/usr/lib64/python3.9/pathlib.py", line 1327, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/usr/lib64/python3.9/pathlib.py", line 1327, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/usr/lib64/python3.9/pathlib.py", line 1327, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/usr/lib64/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/airflow'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/bin/airflow", line 5, in
from airflow.main import main
File "/usr/local/lib/python3.9/site-packages/airflow/init.py", line 68, in
settings.initialize()
File "/usr/local/lib/python3.9/site-packages/airflow/settings.py", line 554, in initialize
LOGGING_CLASS_PATH = configure_logging()
File "/usr/local/lib/python3.9/site-packages/airflow/logging_config.py", line 74, in configure_logging
raise e
File "/usr/local/lib/python3.9/site-packages/airflow/logging_config.py", line 69, in configure_logging
dictConfig(logging_config)
File "/usr/lib64/python3.9/logging/config.py", line 809, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib64/python3.9/logging/config.py", line 571, in configure
raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'processor'
/usr/local/lib/python3.9/site-packages/airflow/configuration.py:812 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
/usr/local/lib/python3.9/site-packages/airflow/configuration.py:738 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
/usr/local/lib/python3.9/site-packages/airflow/settings.py:194 DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/lib64/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/airflow/logs/scheduler/2024-03-12'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/airflow/logs/scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/airflow/logs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.9/logging/config.py", line 564, in configure
handler = self.configure_handler(handlers[name])
File "/usr/lib64/python3.9/logging/config.py", line 745, in configure_handler
result = factory(**kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/utils/log/file_processor_handler.py", line 50, in init
Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True)
File "/usr/lib64/python3.9/pathlib.py", line 1327, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/usr/lib64/python3.9/pathlib.py", line 1327, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/usr/lib64/python3.9/pathlib.py", line 1327, in mkdir
self.parent.mkdir(parents=True, exist_ok=True)
File "/usr/lib64/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/airflow'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/bin/airflow", line 5, in
from airflow.main import main
File "/usr/local/lib/python3.9/site-packages/airflow/init.py", line 68, in
settings.initialize()
File "/usr/local/lib/python3.9/site-packages/airflow/settings.py", line 554, in initialize
LOGGING_CLASS_PATH = configure_logging()
File "/usr/local/lib/python3.9/site-packages/airflow/logging_config.py", line 74, in configure_logging
raise e
File "/usr/local/lib/python3.9/site-packages/airflow/logging_config.py", line 69, in configure_logging
dictConfig(logging_config)
File "/usr/lib64/python3.9/logging/config.py", line 809, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib64/python3.9/logging/config.py", line 571, in configure
raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'processor'