KSQL is an open source streaming SQL engine for Apache Kafka. It provides an easy and completely interactive SQL interface for stream processing on Kafka-no need to write any code in a programming language such as Java or Python. As such, KSQL makes stream processing with Kafka significantly more accessible to many more users inside an organization. And while many users may be familiar with traditional SQL technologies such as Oracle or MySQL from the database world, applying SQL to streams of data has several unique and important differences that need to be understood in order to deploy KSQL as a stream processing technology to production environments.
In this talk we take a deep-dive into key internal concepts and architecture of KSQL as a representative of the recently emerging technologies for “streaming SQL”. We discuss the lifecycle of a query from the time it is submitted by the user, to the time it is executed continuously in the KSQL engines 24×7, and until it is terminated. We also explain standalone and distributed client-server deployment options for KSQL along with how KSQL statements are executed in each of these options. Armed with such knowledge, KSQL users will be able to use and operationalize KSQL in the various supported deployment modes much more smoothly and to prevent many mistakes in deployment and maintenance of KSQL.
