After migrating much of our tech stack to Kafka Streams and Kafka Connect, some of our newer streaming applications encountered the same problem: while their primary input data was available in Kafka and their static side input data could be ingested using traditional connectors, they still needed access to semi-static data that has historically been acquired through in-line synchronous requests and responses. External service calls made from within a Kafka Streams application are highly discouraged, as they make error-handling difficult and typically slow down the application. To solve this problem, we used Kafka Connect components to build a large-scale data subscription service which subscribes to, polls, and disseminates auxiliary data to Kafka topics for immediate consumption by our streaming applications.
This talk will cover:
- An overview of our subscription service architecture
- A deep-dive into our Kafka Connector and each of its custom components
- Design considerations and various pitfalls that we encountered along the way
