How to optimize Apache Airflow DAGs for high-volume ETL tasks without resource contention?

All answers to this question.

If your tasks are getting stuck, you likely have a bottleneck in your worker pool or your metadata database. First, check your parallelism and dag_concurrency settings in airflow.cfg. However, for 200+ DAGs, moving to the KubernetesExecutor is the most sustainable long-term solution. It allows each task to run in its own isolated pod, which scales dynamically based on your cluster's resources. This eliminates the "noisy neighbor" problem where one heavy Spark job starves other small Python tasks of memory. In a migration I handled in 2024, switching to K8s decreased our average task latency by 45% because we no longer had fixed-size worker nodes sitting idle or being overloaded.

   Answered 2025-10-20 by Amanda Richardson

Does the KubernetesExecutor add too much overhead for small, fast tasks since it has to spin up a new pod for every single operation?

   Answered 2025-11-05 by David Harrison

• David, that is a valid concern. For very short tasks (under 30 seconds), the pod startup time can be an issue. You can mitigate this by using the "LocalKubernetesExecutor," which uses LocalExecutor for simple tasks and shifts heavier jobs to K8s pods. This gives you the best of both worlds: low latency for small tasks and infinite scalability for your massive data processing jobs.

     Commented 2025-11-12 by Michael Stevens

I highly recommend using Task Groups instead of SubDAGs. SubDAGs are notorious for causing deadlocks in the scheduler when concurrency limits are reached.

   Answered 2025-11-20 by Patricia Adams

• Great point, Patricia. We switched to Task Groups last year and the UI is much cleaner, plus we stopped seeing those weird "zombie task" errors we had with SubDAGs.

     Commented 2025-11-25 by Michael Thompson