Thread

10/2/2023, 5:13:53 PM

You can run a dbt model from Apache Airflow by using the BashOperator or the DbtTaskGroup class from the Cosmos package.

Here's an example of how to use the BashOperator to run a dbt model:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime

with DAG('dbt_dag', start_date=datetime(2022, 1, 1)) as dag:

    run_dbt = BashOperator(
        task_id='run_dbt',
        bash_command='dbt run --profiles-dir ~/.dbt/ --project-dir ~/my_dbt_project/'
    )

In this example, dbt run is the command to run the dbt model. The --profiles-dir flag indicates the directory where the profiles.yml file is located, and the --project-dir flag indicates the directory of your dbt project.

Alternatively, you can use the DbtTaskGroup class from the Cosmos package to create a task group from the models in your dbt project. Dependencies between your dbt models are automatically turned into dependencies between Airflow tasks. Here's an example:

from cosmos import DbtTaskGroup, ProjectConfig, ProfileConfig
from airflow import DAG
from datetime import datetime

with DAG('dbt_dag', start_date=datetime(2022, 1, 1)) as dag:

    dbt_task_group = DbtTaskGroup(
        project_config=ProjectConfig("/path/to/my_dbt_project"),
        profile_config=ProfileConfig(
            profile_name="my_profile",
            target_name="my_target",
            profile_mapping=PostgresUserPasswordProfileMapping(
                conn_id="my_postgres_dbt",
                profile_args={"schema": "public"},
            ),
        )
    )

In this example, the DbtTaskGroup class uses the ProjectConfig to specify the directory of your dbt project and the ProfileConfig to specify the profile and target to use for your dbt project. The PostgresUserPasswordProfileMapping class is used to map your Airflow connection to a dbt profile.

Human
10/2/2023, 5:13:26 PM

how can I run a dbt model from Airflow ?

Sources