How to Create a Data Pipeline with Apache Airflow: A Step-by-Step Guide

Apache Airflow is an open‑source platform to programmatically author, schedule, and monitor workflows. It lets you build complex data pipelines as Directed Acyclic Graphs (DAGs) where each node is a task and edges define dependencies. This tutorial walks through the core steps to create and run your first pipeline.

You will set up Airflow locally, define a DAG with Python, add tasks using operators, and then trigger the pipeline from the web UI. By the end, you’ll have a working foundation for orchestrating any data workflow.

Article illustration

1. Setting Up Airflow

Install Airflow with pip and configure the environment:

  • Create a virtual environment: python -m venv airflow_env
  • Install Airflow: pip install apache-airflow
  • Initialize the metadata database: airflow db init
  • Start the web server and scheduler: airflow webserver -p 8080 & airflow scheduler

Access the UI at http://localhost:8080.

2. Defining Your DAG

Create a Python file in the dags/ folder. Import DAG and operators, set default arguments, and define the schedule:

  • Example: from datetime import datetime and default_args = {'owner': 'me', 'start_date': datetime(2024,1,1)}
  • Instantiate the DAG: dag = DAG('my_pipeline', default_args=default_args, schedule='@daily')

3. Creating Tasks and Dependencies

Tasks are created with operators (e.g., PythonOperator for custom functions):

  • task1 = PythonOperator(task_id='extract', python_callable=extract_fn, dag=dag)
  • task2 = PythonOperator(task_id='transform', python_callable=transform_fn, dag=dag)
  • Set dependencies: task1 >> task2

Run the pipeline by unpausing the DAG in the UI and triggering it manually or letting the scheduler execute it.

4. Monitoring and Logging

Use the Airflow UI to view DAG runs, task logs, and failures. Set up email alerts or retries in default_args for production robustness.

Conclusion: With Airflow you can build scalable data pipelines. Start with a simple DAG, add more tasks, and use connectors (e.g., PostgresOperator, S3Hook) to integrate your data stack.

sarah antaboga
Author: sarah antaboga

Leave a Reply

Your email address will not be published. Required fields are marked *