Until recently, the Messaging team at Twitter had been running an in-house Pub/Sub system, namely EventBus (built on top of Apache DistributedLog and Apache Bookkeeper, and similar in architecture to Apache Pulsar) to cater to our pubsub needs. In 2018, we made the decision to move to Apache Kafka by migrating existing use cases as well as onboarding new use cases directly onto Apache Kafka. Fast forward to today, Kafka is now an essential piece of Twitter Infrastructure and processes over 150M messages per second. In this meetup talk, we will share the learning and challenges in our journey moving to Apache Kafka.
Slides:
https://www.slideshare.net/ConfluentInc/twitters-apache-kafka-adoption-journey/ConfluentInc/twitters-apache-kafka-adoption-journey
