Back To Schedule
Friday, November 16 • 11:30am - 12:00pm
Using Akka Streams for Web-scale Data Ingestion

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Iterable uses Akka Streams to manage streams of hundreds of millions of events daily for some of the world's largest consumer companies. In this talk we'll discuss how we use Akka Streams to ingest streams of event data into Elasticsearch using Kafka. We'll show how Akka Streams makes writing streaming-related code more elegant, flexible, and performant. We'll discuss the various stream operations such as `mapAsync` and `groupedWithin`. We'll look at how Akka Streams deals with common streaming data issues such as idempotency, ordering of updates, and backpressure. Finally, we'll discuss practical issues such as measuring performance, testing, and tuning an Akka Streams-based system in the context of massive data ingestion into a distributed, eventually consistent database.

avatar for Greg Methvin

Greg Methvin

Staff Software Engineer, Iterable
Greg Methvin is probably best known for his work as a maintainer for Play Framework and a contributor to Scala, Akka, and other open source libraries in the Scala community. He currently works on Iterable's data infrastructure team, building a platform to help companies better understand... Read More →
avatar for Jie Ren

Jie Ren

Senior Software Engineer - Data Infrastructure, Iterable
I'm a software engineer currently focused on data infrastructure at Iterable. I am passionate about solving complex technical challenges, as I’ve done throughout my career. I've been working with the Scala, Elastic, Akka Streams, Kafka stack for the last 4 years. Prior to Iterable... Read More →

Friday November 16, 2018 11:30am - 12:00pm PST