Kafka Streams and Kafka Connect provide tools to consume and produce from Kafka, but, as services are built out, how do we know how well our system is doing? Is our AMQP broker introducing increasing round-trip time? How many messages are being consumed by our application and where do these messages come from? We could add counters with a Kafka Streams processes, but how would we do the same for Kafka Connectors? Many of these problems can be solved by leveraging interceptors, Prometheus, and OpenTracing. By using interceptors we can quickly instrument new applications and connectors to provide observability into your entire stack. By the end of this talk, you'll learn how to utilize interceptors throughout your Kafka Stream applications and Kafka Connect cluster to provide additional observability. We'll demonstrate how you can use these interceptors to provide both tracing using OpenTracing and monitoring using Prometheus. Additionally, we'll highlight the pros and cons of using interceptors for observability compared to existing methods. As an attendee of Kafka Summit 2018 at SF, I gained insight on how other companies use Kafka with IoT devices and I look forward to the opportunity to share the knowledge our team has gained since attending the summit. Thank you for your consideration.
