Apache Airflow & llama-index>=0.10.0 Error

[amindadgar]

Hello everyone. I've been writing ETL processes to store my raw data as llama-index Documents in a postgres database using Apache Airflow dags. In apache airflow I have set its default database to postgres and I've been using pgvector image as my postgres database. Previously when I was using llama-index<=0.9.48 everything was working as expected and airflow docker could become up on my local system. When I was migrated my codebase to the newer llama-index>=0.10.0 version, I'm getting a postgres error when the airflow-init service is checking the database health.

The airflow-init service error is

airflow-dags-airflow-init-1       | ....................
airflow-dags-airflow-init-1       | ERROR! Maximum number of retries (20) reached.
airflow-dags-airflow-init-1       | 
airflow-dags-airflow-init-1       | Last check result:
airflow-dags-airflow-init-1       | $ airflow db check
airflow-dags-airflow-init-1       | Traceback (most recent call last):
airflow-dags-airflow-init-1       |   File "/home/airflow/.local/bin/airflow", line 5, in <module>
airflow-dags-airflow-init-1       |     from airflow.__main__ import main
airflow-dags-airflow-init-1       |   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/__init__.py", line 68, in <module>
airflow-dags-airflow-init-1       |     settings.initialize()
airflow-dags-airflow-init-1       |   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/settings.py", line 546, in initialize
airflow-dags-airflow-init-1       |     configure_orm()
airflow-dags-airflow-init-1       |   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/settings.py", line 242, in configure_orm
airflow-dags-airflow-init-1       |     engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args, future=True)
airflow-dags-airflow-init-1       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-dags-airflow-init-1       |   File "<string>", line 2, in create_engine
airflow-dags-airflow-init-1       |   File "/home/airflow/.local/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
airflow-dags-airflow-init-1       |     return fn(*args, **kwargs)  # type: ignore[no-any-return]
airflow-dags-airflow-init-1       |            ^^^^^^^^^^^^^^^^^^^
airflow-dags-airflow-init-1       |   File "/home/airflow/.local/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 612, in create_engine
airflow-dags-airflow-init-1       |     dialect = dialect_cls(**dialect_args)
airflow-dags-airflow-init-1       |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-dags-airflow-init-1       |   File "/home/airflow/.local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 653, in __init__
airflow-dags-airflow-init-1       |     self.executemany_mode = parse_user_argument_for_enum(
airflow-dags-airflow-init-1       |                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-dags-airflow-init-1       |   File "/home/airflow/.local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 1734, in parse_user_argument_for_enum
airflow-dags-airflow-init-1       |     raise exc.ArgumentError(f"Invalid value for '{name}': {arg!r}")
airflow-dags-airflow-init-1       | sqlalchemy.exc.ArgumentError: Invalid value for 'executemany_mode': 'values'
airflow-dags-airflow-init-1       | 
airflow-dags-airflow-init-1 exited with code 1
service "airflow-init" didn't complete successfully: exit 1

I've been trying many times with different airflow versions (ranging from 2.7.3 to the latest 2.8.2) but none was working. Anyone have any idea how this could be resolved?

Note: Here's my latest ETL processes with airflow if you wanted to have a look https://github.com/TogetherCrew/airflow-dags/tree/feat/upgrade-llama-index-dependency

[dosubot[bot]]

Hey @amindadgar, great to see you back! Ran into another quirky tech puzzle, I see? Let's dive into this one together.

Based on the error message you provided, it seems like the issue is related to SQLAlchemy's executemany_mode configuration. The error message Invalid value for 'executemany_mode': 'values' suggests that the executemany_mode is set to 'values', which is not a valid option.

The executemany_mode option is used to control how SQLAlchemy sends multiple-row INSERT and UPDATE statements to the database. The valid options for executemany_mode are None, 'batch', or 'hugememory'.

You can try to set the executemany_mode to None or 'batch' in your SQLAlchemy engine configuration and see if that resolves the issue. Here is an example of how to do it:

from sqlalchemy import create_engine

engine = create_engine(
    SQL_ALCHEMY_CONN,
    executemany_mode='batch',  # or None
    connect_args=connect_args,
    **engine_args,
    future=True
)

Regarding the changes in llama-index>=0.10.0 that could potentially affect the interaction with the Postgres database, I wasn't able to find specific information in the repository. It would be helpful if you could provide more details about the changes you made when upgrading to llama-index>=0.10.0.

If the issue persists after trying the above solution, please provide more details about your SQLAlchemy engine configuration and the exact changes you made when upgrading to llama-index>=0.10.0. This will help in identifying the root cause of the issue and providing a more accurate solution.

For more information about SQLAlchemy's executemany_mode option, you can refer to the SQLAlchemy documentation.

Sources

• llama-index-legacy/tests/initialization/postgres/postgres_init.sql

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[amindadgar]

I think I found the error. airflow=2.8.2 supports sqlalchemy<2.0.0 and llama-index-vector-stores-postgres==0.1.2 supports sqlalchemy>=2.0.0. I don't know why no conflict error happened when installing llama-index-vector-stores-postgres==0.1.2 but in any case another error was rasing because of assigning a deprecated value to executemany_mode of sqlalchemy

The problem is not related to llama-index and apache airflow should update their sqlalchemy dependency.