In a micro-services environment, a clean slate is the ideal starting point for writing Kafka Streams applications. However, the reality is that, in most established organizations, existing legacy systems need to be replaced, while the current client base is maintained. In this talk, we will review our journey in migrating from an existing, legacy system to a more scalable Kafka Streams application, all the while maintaining the stability and integrity of our data for existing customers. We will concentrate on how two technologies – Kafka Connect and KSQL – were instrumental during this migration.
We will cover:
* Our motivation for migrating to KStreams
* How Kafka Connect and KStreams concepts (KTable and GlobalKTable) allowed us to incorporate an external data store into the KStreams app
* How KSQL, using stream-to-stream JOINs, provided a fast and easy way to achieve A/B testing
* Kafka Connect and KSQL internals, pitfalls, and tips
