Scale By the Bay 2018

It is really bad to block Finagle’s event loop. The most common way this happens is when code explicitly blocks by calling Await.result or Await.ready. But this is not the only way the event loop can get blocked. The event loop can be blocked when long running computations are called within the loop. If you create a service that directly makes a long running computation it will work fine when you test it under no load. But in a production environment it will grind to a halt even though the server does not appear to be overloaded and should be fully capable of handling all the requests quickly. What you will find is happening is that your are exhausting the finagle event loop thread pool while the time taken to run each computation is acceptable the amount of time each request waits around for a network IO thread to become available quickly goes up. In my talk I will discuss how to identify when this problem is occurring and how to correct it so your latency returns back to expected values.