Design and Implementation of a Scalable and QoS-aware Stream Processing Framework: The Quasit Prototype

Abstract

Today's stream processing scenarios are characterized by large volumes of data, e.g., generated by cyber-physical systems in a smart city, on which continuous analysis tasks need to be performed, often with very different optimal trade-offs between achieved QoS and associated resource consumption. Here we present the novel Quasit model and framework offering runtime support to stream processing applications. Differently from existing literature, Quasit originally allows advanced QoS-based configuration, which can be used to finely tune the framework to fit highly different real-world situations. The paper describes the architecture and development of the Quasit prototype by offering interesting insights and lessons learned about the most important design/implementation choices made, such as the actor-based threading model, or the QoS enabled inter-process communication based on OMG DDS. The reported experimental results, measured over simple real test beds, show that our Quasit framework implementation can provide a good level of horizontal scalability with limited overhead and good exploitation of dynamically available processing resources.