In this presentation, the focus is on elucidating how data is collected and processed utilizing Kafka. The step-by-step walkthrough covers the process of making a request and the seamless passage of data to a Kafka producer. Attention is given to a common challenge: data flowing into Kafka during development, including insights into a encountered bug and its resolution. Furthermore, we touch upon the advantages of leveraging the Kappa architecture and explore the functionality of emerging metastores such as Iceberg.[Recorded Nov 30, 2023]
47 mins