Scale By the Bay 2018

In recent years, stream processing systems have become the de-facto standard for processing a large and growing volume of data from different data sources and in computing interesting insights. Hence, stream processing has become a widely researched area as it poses various challenges with respect to performance, programming abstractions, consistency guarantees, fault-tolerance, resiliency and so on. Different stream processing libraries (e.g., Iteratee, Pipes, Monix, Scalaz-Stream, Akka Streams) have been evolved to address these issues differently and with varying priorities. Also, a wide range of industry-scale distributed stream processing systems has been introduced such as Flink, Heron, Spark Streaming, Samza and so on. 

In this comparative case study, we particularly focus on state-of-the-art distributed stream processing systems (such as Flink, Heron, Spark Streaming, Samza, etc) present different approaches to these systems. In particular, while basing foundation on the commonalities among these approaches and their primary constructs, we build common ground to interact with their semantics. Afterward, we analyze the differences among different approaches, where they excel and where they fall short compared to the others, in addressing the core issues of stream processing. Also, we take a look at how the streaming landscape has evolved in the last two decades, its notable trends and future research directions. 

As an audience, from the talk, I would get a comparative overview of varying approaches to (distributed) stream processing, their programming models, execution models and notable properties that potentially can help in differentiating them and selecting the right tool while enable us to take into account accommodate the unique characteristics and trade-offs of the system into account.