Replies: 3 comments
-
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Beta Was this translation helpful? Give feedback.
0 replies
-
Support of MinIO through Amazon Provider not officially supported/tested. |
Beta Was this translation helpful? Give feedback.
0 replies
-
Ok, I understand. Is there another Provider that can be used for MinIO connections? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.8.3
What happened?
I have Airflow 2.8.3 on Kubernetes, deployed through the Helm chart. On the same namespace, I have a MinIO tenant.
My DAGs can write output files to the MinIO using a connection and the S3Hook.
I'm trying to use remote_logging to get the logs to a bucket on the same MinIO tenant, but it's not working.
When I try to read the DAGs logs, the airflow-webserver logs say:
This is the relevant part of the values.yaml:
This is the
conn_logs
connection extras currently:What I've tried to do:
endpoint_url
as inhttps://airflow-storage-hl.airflow.svc.cluster.local:9000
verify
value (/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
)Before setting the
verify
option in the connection, I was getting errors related to the certificate validation. This also was the error when setting the value to'False'
.I have also successfully read from the bucket from the webserver pod:
What you think should happen instead?
The logs should be written to the MinIO 'logs' bucket the same way the DAGs write files to the other buckets in the same MinIO tenant.
How to reproduce
airflow-storage
. Create a Access Key to use in the Airflow ConnectionOperating System
Rocky Linux 9.3
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.19.0
apache-airflow-providers-celery==3.6.1
apache-airflow-providers-cncf-kubernetes==8.0.1
apache-airflow-providers-common-io==1.3.0
apache-airflow-providers-common-sql==1.11.1
apache-airflow-providers-docker==3.9.2
apache-airflow-providers-elasticsearch==5.3.3
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-google==10.16.0
apache-airflow-providers-grpc==3.4.1
apache-airflow-providers-hashicorp==3.6.4
apache-airflow-providers-http==4.10.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-microsoft-azure==9.0.1
apache-airflow-providers-microsoft-mssql==3.5.0
apache-airflow-providers-mysql==5.5.4
apache-airflow-providers-odbc==4.4.1
apache-airflow-providers-openlineage==1.6.0
apache-airflow-providers-postgres==5.10.2
apache-airflow-providers-redis==3.6.0
apache-airflow-providers-sendgrid==3.4.0
apache-airflow-providers-sftp==4.9.0
apache-airflow-providers-slack==8.6.1
apache-airflow-providers-smtp==1.6.1
apache-airflow-providers-snowflake==5.3.1
apache-airflow-providers-sqlite==3.7.1
apache-airflow-providers-ssh==3.10.1
Deployment
Official Apache Airflow Helm Chart
Deployment details
I'm deploying Airflow using the official Helm Chart using ArgoCD. The image is modified as follows:
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions