Apache Flink Streaming - Exploring Data Collection and Processing with Kafka in Flink (Day 2 Lecture)

In this presentation, the focus is on elucidating how data is collected and processed utilizing Kafka. The step-by-step walkthrough covers the process of making a request and the seamless passage of data to a Kafka producer. Attention is given to a common challenge: data flowing into Kafka during development, including insights into a encountered bug and its resolution. Furthermore, we touch upon the advantages of leveraging the Kappa architecture and explore the functionality of emerging metastores such as Iceberg.[Recorded Nov 30, 2023]

47 mins

Purchase Required

You need to purchase this content in order to view it

Apache Flink Streaming - Kafka, Postgres, Spark Integration, and Parallelism (Day 2 Lab)

Week 3: Streaming Pipelines with Apache Flink

LLM-Driven Data Engineering Day 1 Lecture