Installing apache-airflow-backport-providers-odbc failing

[VORUSRIK000]

We have Airflow 1.10.13 version installed with app running healthy (installed using bitnami helm charts). Trying to install apache-airflow-backport-providers-odbc on this version as suggested in documentation, to get support for Azure SQL connectivity from a DAG. Installation is failing with the below error. Any pointers would help.

ERROR: Command errored out with exit status 1:
command: /opt/bitnami/python/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bzou29_4/pyodbc/setup.py'"'"'; file='"'"'/tmp/pip-install-bzou29_4/pyodbc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-uybw5lm_/install-record.txt --single-version-externally-managed --compile --install-headers /opt/bitnami/python/include/python3.6m/pyodbc
cwd: /tmp/pip-install-bzou29_4/pyodbc/
Complete output (10 lines):
running install
running build
running build_ext
building 'pyodbc' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/src
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -s -s -fPIC -DPYODBC_VERSION=4.0.30 -I/opt/bitnami/python/include/python3.6m -c src/buffer.cpp -o build/temp.linux-x86_64-3.6/src/buffer.o -Wno-write-strings
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /opt/bitnami/python/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bzou29_4/pyodbc/setup.py'"'"'; file='"'"'/tmp/pip-install-bzou29_4/pyodbc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-uybw5lm_/install-record.txt --single-version-externally-managed --compile --install-headers /opt/bitnami/python/include/python3.6m/pyodbc Check the logs for full command output.
Running setup.py install for pyodbc: finished with status 'error'
command terminated with exit code 1

[potiuk]

You need to have bulf-tools (for example gcc) installed if you want to install pyodbc. Pyodbc requires build-essentials to be available (usually via apt-get or similar) in your image. You probably will need to customize your image somehow (not sure what image you are using) because - similarly to official production image of Airflow it is optimised for size and it has no build essentials. If you switch to official Airflow image there are some ways you can customize your image (https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html#customizing-the-image)

Another option is installing pyodbc from pre-compiled binary packages, but how to install it is very much image dependent.

See description and further docs here: https://pypi.org/project/pyodbc/

[VORUSRIK000]

Thanks Jarek. Yes, the image we are using from bitnami and they do have a slim version as you mentioned, missing build essentials. Issue is that we are not able to execute apt-get on that image due to some root permission issues.

But this is related to the image and will follow up with that team and get it resolved. Thank you for the pointers and help.

Just a tangent question. Does Airflow provide sort of a helm chart for orchestrating all the containers using kubernetes?

[potiuk]

Yes. It's not yet officially released - we are still in the testing phase but After 2.0 got released I think next few weeks we will have more time to polish it and release. It uses the official image I mentioned above and you can find it here: https://github.com/apache/airflow/tree/master/chart . It's been already battle-tested quite heavily but there probably some specific combinations of parameters that might have problems, but I think this is your best bet to start using it. Maybe there will be some breaking changes before we release it, but I do not expect anything major.

[VORUSRIK000]

Thank you.