Originally published on The Hotels.com Technology Blog
Ryan Burke | Software Development Engineer, Hotels.com in London
Haystack is an Expedia-backed open source project to facilitate detection and remediation of problems with enterprise-level web services and websites. Haystack uses tracing data to help locate the source of problems, providing the ability to drill down to the precise part of a service transaction where failures or latency are occurring — and find the proverbial “needle in a haystack”. Once you know specifically where the problem is happening, it’s much easier to identify and understand the appropriate diagnostic data, find the problem, and fix it.
Finatra is a web framework created by Twitter built on top of TwitterServerand Finagle, it is the web framework of choice for the majority of Scala core services at Hotels.com. Recently, we wanted to integrate our services with Haystack in order to have distributed tracing information across service boundaries.
Finatra supports out of the box tracing using standard Zipkin
X-B3-* HTTP headers. In order to report this data to Haystack we needed to publish the tracing data to a proxy service we have running which forwards it to both Zipkin and Haystack.
Fortunately for us, zipkin-finagle provides functionality for reporting tracing information over a network. This library allows for tracing information to be sent via HTTP, Scribe, or published to a Kafka topic. Creating a new zipkin tracer is simple once you bring in
zipkin-finagle as a project dependency:
val config = HttpZipkinTracer.Config.builder()
val tracer = HttpZipkinTracer.create(config, statsReceiver)
In the Finatra app’s
HttpServer class you have the ability to set the tracer and label to be used in reporting by overriding the
override def configureHttpServer(server: Http.Server): Http.Server =
After this, sending tracing headers to the service will result in the data being published to Haystack for visualisation. If you’re using Finagle clients to call other services as part of a request, these will automatically be propagated and all your dependencies will show up too.
Dealing with Futures
Finatra and Finagle are designed to operate in a non-blocking asynchronous way, allowing it to scale and keep the overhead of accepting a new request low. There is no global requests thread pool to configure, just don’t block when you’re handling the request. As such, when we are dealing with asynchronous code we don’t have the concept of a single request thread to do things like
MDC, which is how you would normally keep track of per-request state such as tracing information.
When using Scala
Future[T] we need some way to manually keep track of the tracing information between thread boundaries. We found there was no elegant way to do this without creating a wrapper around Future which copies a context between execution threads. Alternatively you can create a custom
ExecutionContext in which the
Future can run that provides the same functionality. Problems arise when you use a third party library or some bit of code that doesn’t allow you to define the ExecutionContext or the return type.
Twitter were an early adopter of Scala and provide a util library which duplicates and builds upon the Scala standard library features. This includes the Twitter Future, a cancellable Future with no ExecutionContext to manage and the built-in ability to keep track of a Context across thread boundaries. The Finatra server uses them at the edge and Finagle clients return Twitter Futures too. If you use them throughout your application instead of the standard Scala Future then you’ll get tracing propagation for free, at the expense of being a little more tied into the Twitter ecosystem.
Twitter Service Loader
One thing to watch out for is the zipkin-finagle library defining a service in the
META-INF/services folder. Finatra uses Guice for dependency injection and if a library defines a file in the services folder then it will auto-magically be created for you and registered in the service registry. This can make it easier to integrate with Zipkin, you can ignore all the code changes above and instead set some environment variables to let the library create and register the service for you.
In my team we tend to prefer explicitly defining behaviour rather than relying on magic components of frameworks to do this for us. It’s why we moved away from Spring, manually wire everything, try to avoid internal shared libraries and write our own request filter logic.
Once we manually wired the tracer using
withTracer we assumed that this would override the one being created from the service loader, but we were wrong. Both were being created and running at the same time, causing the unconfigured default tracer to throw errors (it defaults to sending data to localhost). In order to disable this we have to modify our Docker file to add an additional Java opt:
ENTRYPOINT [“/bin/sh”, “-c”, “exec java $JAVA_OPTS -Dcom.twitter.finagle.util.loadServiceDenied=zipkin2.finagle.http.HttpZipkinTracer -jar service.jar $0 $@”]
This is a bit nasty, we have a hard coded class name in our Docker file and if it ever changes name then it’ll start loading two
HttpZipkinTracer instances again. That’s the cost of being able to define the tracer ourselves.
We are are hiring! If you’re passionate about software engineering and what we do sounds interesting check out our roles!