Youtor
Dive into Streaming Checkpoints amp Best Practices 1252023
>> YOUR LINK HERE: ___ http://youtube.com/watch?v=v-GHBmL3KmI
In this video, Vikas Reddy, a Staff Backline Engineer at Databricks, goes over the use and importance of using checkpointing in streaming applications. Asynchronous state checkpointing maintains exactly-once guarantees for streaming queries but can reduce overall latency for some Structured Streaming stateful workloads bottlenecked on state updates. This is accomplished by beginning to process the next micro-batch as soon as the computation of the previous micro-batch has been completed without waiting for state checkpointing to complete. This is explained in a lot more detail (with various examples and demos!) in the 1 hour long video here. • Target Audience - Data Engineers, Spark Engineers, Data/Solutions Architects • ►[Documentation] Asynchronous state checkpointing for stateful queries - https://docs.databricks.com/en/struct... • ►[Documentation] Recover from Structured Streaming query failures with workflows - https://docs.databricks.com/en/struct... • ►[Documentation] Production considerations for Structured Streaming - https://docs.databricks.com/en/struct... • ►[Slides] Slides from the video - https://drive.google.com/file/d/1irws... • ►[Product] Learn more about Databricks here - https://www.databricks.com/ • ►Learn/connect with the speaker here - / avikasreddy • ► Discover more about Databricks in the Skill Builder Series here - • Skill Builder for Databricks • #databricks #spark #data #streaming #structuredstreaming #faulttolerant #checkpointing #data #lakehouse #batch #kafka
#############################
