aggregate(download.expand(url=fetch_urls()))
def pull_function(**context): user_id = context['ti'].xcom_pull(task_ids='push_task', key='user_id') print(f"Received user_id")
No xcom_push or xcom_pull needed – the TaskFlow wiring handles it. With traditional operators, you must push/pull manually. xcom in airflow
push = PythonOperator(task_id='push_task', python_callable=push_function) pull = PythonOperator(task_id='pull_task', python_callable=pull_function)
XCom (short for cross‑communication ) is Airflow’s built‑in mechanism for exchanging small pieces of data between tasks. When used wisely, they unlock powerful patterns. When abused, they break your DAGs. Let’s see how to use them correctly. XComs are key‑value pairs stored in Airflow’s metadata database. A task can push an XCom (write a value under a key), and another task can pull that value (read it). aggregate(download
XComs are not designed for large data. Default size limit is 1 MB (configurable, but don’t). Use them for IDs, file paths, dates, small JSON – not DataFrames or images. The Two Ways to Use XComs 1. Implicit XComs via return Any Python function decorated with @task (TaskFlow API) automatically pushes its return value as an XCom.
@task def consume_two(data): return f"Got data['source']" @task def fetch_urls() -> list[str]: return ["http://a.com", "http://b.com"] @task def download(url: str) -> str: # download content return f"content_of_url" When used wisely, they unlock powerful patterns
@task def process(user_data: dict) -> str: return f"Processed user user_data['name']"