Serverless Data Processing System and its Design Space Consideration

Serverless computing is becoming increasingly important in data-processing applications in science and business. The scheduler is at the centre of serverless data-processing systems, allowing for dynamic decisions on job and data placement. The complex design space, which is influenced by various user, cluster, and workload variables, presents problems for developing high-performance and cost-effective scheduling structures and processes. To make this exploration easier, we present Sched-Probe, a framework that includes a conceptual model and simulator for systematic design space exploration. Using the Sched-Probe framework, we evaluate the performance of three scheduling systems and two techniques using real-world workloads. Our open-source software is now available on ExDe, allowing system designers to collaborate on delving into the complexity of serverless scheduling, paving the way for optimised and efficient data-processing systems.