Problem with pushing Xcom when using SparkKubernetesOperator #39184
Labels
area:core
kind:bug
This is a clearly a bug
needs-triage
label for new issues that we didn't triage yet
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.8.3
What happened?
I'm having an issue when using SparkKubernetesOperator to execute Spark job. Whenever i set do_xcom_push to True, the driver pod is created and run perfectly, but the xcom sidecar pod is nowhere to be found. Airflow log is thus stuck with this message:
In my opinion, there should be something wrong with xcom push for this operator. Any help would be much appreciated. Thanks in advance.
What you think should happen instead?
xcom pushed perfectly with any case of spark, maybe there will be placeholder for spark job with no return value or whatsoever
How to reproduce
submit a basic spark application by using SparkKubernetesOperator to K8s cluster and set do_xcom_push to True
Operating System
Ubuntu 22.04.4 LTS
Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==8.0.1
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: