Building Kafka and Spark Structured Streaming Pipelines using Databricks
>> YOUR LINK HERE: ___ http://youtube.com/watch?v=auyxisjijOc
As part of this topic we will see how to ingest data in real time using Kafka eco system and process using Spark Structured Streaming on top of Databricks. • Here is the link for complete material - http://discuss.itversity.com/t/integr... • Agenda • Here is the agenda for this module or section. We will see end to end streaming analytics pipeline using Kafka and Spark with Scala as Programming Language on top of Databricks Platform. • Define Problem Statement • Setup EC2 Instance • Simulate Web Server Logs • Setup and Start Kafka Broker • Create Kafka Topic and Validate • Ingest Web Server Logs into Kafka Topic • Setup Databricks Cluster • Integrate and Validate with Spark Structured Streaming • Streaming Data Processing using Data Frame APIs • Perform Data Analysis using Spark SQL
#############################
