Data Strategie

Dagster vs airflow 3. Which to pick?

Reddit r/dataengineering

Summary

hey guys, I manage tech for a startup. and I have not used an orchestrator before. Just cron mostly. As we are scaling, I wanted to make things more reliable. Which orchestrator should I pick? It will be batch jobs which might run at different intervals do some etl refresh data etc. Since it ran in cron, the dependency logic itself was all handled in the code itself before. Also both eat equal amount of resources right? I hear airflow being ram heavy but not sure if it's entirely true. let me...

Read the full article