Data Resources - Data Pipeline Orchestration Tools

Apache Airflow

Airflow is a platform to programmatically author, schedule and monitor workflows.

Dagster is an orchestrator designed for developing and maintaining data assets, such as tables, data sets, ML models, and reports.

Argo is unique as it is the only genuinely general-purpose, cloud-native data orchestrator.

Prefect 2 has been designed from the ground up to handle the dynamic, scalable workloads that the modern data stack demands.

Luigi is a Python package that helps you build complex pipelines of batch jobs.