Cloud Native Data Pipelines • Sid Anand • GOTO 2017

>> YOUR LINK HERE: ___ http://youtube.com/watch?v=G8bNoOjCwMo

This presentation was recorded at GOTO Chicago 2017. #GOTOcon #GOTOchgo • http://gotochgo.com • Sid Anand - Data Architect at Agari • ABSTRACT • Big Data companies (e.g. LinkedIn, Facebook, Google, and Twitter) have historically built custom data pipelines over bare metal in custom-designed data centers. In order to meet strict requirements on data security, fault-tolerance, cost control, job scalability, network topology, and compute and storage placement, they need to closely manage their core [...] • TIMECODES • 0:00 Introduction • 0:16 About Me • 1:06 Agari : What We Do • 4:55 Cloud Native Data Pipelines • 6:26 Desirable Qualities of a Resilient Data Pipeline • 10:23 Use-Case : Message Scoring • 12:58 Architectural Components • 16:55 Tackling Cost • 23:09 ASG : Queue-based • 24:40 ASG - Build Deploy • 27:33 Tackling Operability : Requirements • 28:27 Apache Airflow - Authoring DAGS • 28:56 Apache Airflow - Perf. Insights • 31:15 Apache Airflow - Alerting • 33:29 Stream Processing Architecture • 39:10 What is Avro? • 40:38 Why is Avro Useful? • 43:23 Avro Schema Example • 43:53 Avro Schema Data File Example • 44:26 Avro Schema Streaming Example • 45:01 Avro Schema Registry • Download slides and read the full abstract here: • https://gotochgo.com/2017/sessions/80 • RECOMMENDED BOOKS • Liz Rice • Container Security • https://amzn.to/3oU4iJe • Liz Rice • Kubernetes Security • https://www.oreilly.com/library/view/... • Brendan Burns, Joe Beda Kelsey Hightower • Kubernetes: Up and Running • https://amzn.to/3wrtwlp • John Arundel Justin Domingus • Cloud Native DevOps with Kubernetes • https://amzn.to/3hKZvI5 • Kasun Indrasiri Sriskandarajah Suhothayan • Design Patterns for Cloud Native Applications • https://amzn.to/3yCFxWE • Michael Hausenblas Stefan Schimanski • Programming Kubernetes • https://amzn.to/3qTvKch • Alexander Raul • Cloud Native with Kubernetes • https://amzn.to/3yw9ckc • Nigel Poulton • The Kubernetes Book • https://amzn.to/3dW8ViU • Marko Luksa • Kubernetes in Action • https://amzn.to/3dXk2Im • / gotocon • / goto- • / gotoconferences • #CloudNative #DataPipelines #AWScloud • Looking for a unique learning experience? • Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech • Sign up for updates and specials at https://gotopia.tech/newsletter • SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. • https://www.youtube.com/user/GotoConf...

#############################

New on site