Attending this event?
View analytic
Friday, November 16 • 12:00pm - 12:30pm
Scaling Bayesian Experimentation

Sign up or log in to save this to your schedule and see who's attending!

In the past few years companies like Optimizely, Google, and Uber have noted the limitations of null hypothesis significance testing (NHST) as the framework for experimentation. In particular, statistical problems like multiple comparisons and peeking have been difficult to solve. While some organizations have gone the route of sequential testing, other Bayesian methods provide an alternative to overcome these problems, but are often avoided because of worries about their complexity and computational intensity. We will talk about three challenges with Bayesian statistics for experimentation and how big data, tools like Spark, and a little statistical ingenuity can help us address them. The three challenges we will discuss are (1) coming up with priors for experimentation in a world of big data, (2) building a fast Bayesian computation pipeline that is generalizable to all of the metrics your organization cares about, and (3) overcoming computational inefficiencies when using these statistical methods in a real-time experimentation environment. In the literature on Bayesian statistics, and especially in criticisms of it, you will often run across the difficulty of coming up with priors for statistics. We will show how we were able to automate the computation of priors using data we already had lying around and how we were able to improve on that by slightly changing how we log certain events. The other criticism of Bayesian statistics, and a potential roadblock for implementing it in a big data pipeline, is that it is computationally expensive. This is especially true for more complex models such as a standard revenue distribution which is typically multimodal with a peak at zero and then another near the average receipt. Under a Bayesian methodology, such distributions require multiple parameters to be estimated and do not have analytic (conjugate) priors. The standard approach of using Markov Chain Monte Carlo (MCMC) simulations can be too slow, is not parallelizable, and requires modeling of each metric. We will discuss how we use Spark to efficiently use a statistical method called bootstrapping to handle these computational problems and provide a generalizable solution to Bayesian computation. Lastly, we often want to run our experimentation analysis in real-time so that we can make fast decisions or to inform an n-armed bandit algorithm. We will talk about some approaches we use to decrease the computation needed in a real-time experimentation analysis environment. Although bootstrapping is more efficient than MCMC, it is still more expensive than analytic methods and can be prohibitively costly in real-time. We will talk about a couple of methods we have developed to update bootstrapped data and compare their performance with a naive method.

avatar for Paul Cho

Paul Cho

Data Engineer, Udemy
avatar for Robert J. Neal

Robert J. Neal

Principal Software Engineer, Udemy
Software engineer who prefers Scala. Primarily working in experimentation, statistics, and reinforcement learning.

Friday November 16, 2018 12:00pm - 12:30pm

Attendees (4)

Twitter Feed