Back To Schedule
Friday, November 16 • 11:50am - 12:30pm
Enabling Big Data and Machine Learning for the Masses: Creating a Spark Platform for the Uninitiated

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Medium is expanding its use of big data and machine learning to support its product teams. In doing so, it needs to find a way to leverage both the existing technical stack in which it has invested and the knowledge of its engineering team. Unfortunately, these are somewhat at odds. Medium has heavily invested in Scala and Spark for its ETL pipelines. And while Spark certainly provides functionality to support big data analysis and machine learning, its learning curve is very high and only a few Medium engineers have experience with it. To combat this, Medium is actively developing a platform that eases the learning curve for both big data and machine learning operations. This is not only helping get to machine learning results faster, but also write and maintain ETL pipelines more efficiently. The platform includes tools for development, online and offline testing, machine learning experimentation, and monitoring.

avatar for Tim Kral

Tim Kral

Team Lead of Data Engineering, Medium

Friday November 16, 2018 11:50am - 12:30pm PST