-
Notifications
You must be signed in to change notification settings - Fork 15.8k
Description
Apache Airflow version
2.10.5
If "Other Airflow 2 version" selected, which one?
No response
What happened?
After remove mapping tasks, subsequent normal tasks are not executed by the scheduler.
The issue was already described in #33164 which is closed with the note "Can't Reproduce".
I also encountered a similar bug, and prepared a small example DAG that reproduces the bug on the latest stable Airflow 2.10.5
What you think should happen instead?
Dag is executed successfully after increasing or decreasing the number of mapped tasks
How to reproduce
import logging
from airflow.models.dag import DAG
from airflow.models.variable import Variable
from airflow.decorators import task
with DAG(dag_id="example_map_removed_bug",
schedule=None, start_date=None,
tags=["example"]) as dag:
@task
def producer():
"The first run generates a list [0, 1, ...] of 5 elements, each subsequent run produces 1 less element."
value = int(Variable.get("test_value", 6))
value = value - 1 if value > 1 else 5
Variable.set("test_value", value)
return list(range(value))
@task
def work(arg):
logging.info("work task. arg = %s", arg)
@task
def finish(data: list[int]):
logging.info("data = %s", str(data))
logging.info("sum = %d", sum(data))
t1 = producer()
t2 = work.expand(arg=t1)
t3 = finish(t1)
t2 >> t3
- Unpause DAG, Trigger DAG, wait for all steps to complete successfully
producer --> work[5] --> finish
- Do: Clear -> Clear existing tasks
DAG starts over, this time the last step (finish) does not run, DAG Run goes into failed state
producer --> work[5] (4 success + 1 removed) --> **F A I L**
Scheduler logs show error:
{dagrun.py:875} ERROR - Task deadlock (no runnable tasks); marking run <DagRun example_map_removed_bug @ 2025-04-04 21:25:51.774187+00:00: manual__2025-04-04T21:25:51.774187+00:00, state:running, queued_at: 2025-04-04 21:31:53.554772+00:00. externally triggered: True> failed
Operating System
Ubuntu 22.04.5 LTS
Versions of Apache Airflow Providers
No response
Deployment
Virtualenv installation
Deployment details
- Python 3.11.0rc1, virtualenv, Sqlite, airflow standalone
- airflow.cfg
[core]
dags_folder = /home/....../dags
donot_pickle = True
load_examples = False
max_templated_field_length = 16384
[database]
sql_alchemy_conn = sqlite:////......./build/airflow/airflow.db
load_default_connections = False
[webserver]
web_server_port = 1080
workers = 2
secret_key = .......
[logging]
base_log_folder = build/airflow/logs
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct