Dynamic Scheduler In Apache Airflow For Heterogeneous Systems

Abstract

The increasing availability of heterogeneous systems, which combine different types of processing units within a single system, have introduced new opportunities for improving workflow performance. Workflows, consisting of a set of tasks with defined execution dependencies, are widely used in both commercial and scientific applications. However, heterogeneous systems also introduce a key challenge: determining when heterogeneous execution provides benefits over traditional homogeneous CPU-based execution, particularly in environments with varying workflow conditions. This thesis investigates this problem in the context of workflow management systems, with a specific focus on Apache Airflow. A dynamic scheduling approach is proposed, enabling auto-matic selection between CPU only and CPU–GPU execution environments. The study addresses two research questions: which factors and performance metrics can be used to estimate the suitability of workflows for GPU execution, and whether a lightweight decision algorithm can be integrated into Apache Airflow to support dynamic scheduling. The work includes an analysis of heterogeneous computing principles, the architecture of Apache Airflow, and existing approaches to execution environment selection. Based on this analysis, a proof-of-concept plugin is developed that evaluates workflow characteristics and hardware-related metrics to predict the most suitable execution environment. The results demonstrate that dynamic scheduling can be effectively integrated into Apache Airflow, and that relatively simple evaluation methods for execution environment selection can achieve satisfactory performance.