Description:

Airflow plays a crucial role in our data pipeline operations enabling us to schedule and manage data pipelines that run workloads across large Hadoop clusters computing billions of data points every day. This involves a complex and large number of pipelines, each handling different types and volumes of data.

To effectively manage data processing, we utilize Python and Airflow to control the Hadoop cluster lifecycle. Whenever we need to process data, we spin up and spin down Hadoop clusters on demand. Over the years we have distilled out learnings that have stemmed from running complex workloads at scale. These learnings allow us to run the jobs smoothly, efficiently and automatically using Airflow pipelines.

In this talk, we will describe the challenges we encountered and our solutions to overcome them. We will cover the following topics:

• Discussing the challenges of managing the state of a Hadoop cluster from a workflow orchestrator.
• Addressing the complexities associated with error handling during the provisioning of computation clusters.
• Exploring strategies to optimize cost and resource allocation for Spark jobs within a computation cluster.
• Some tips and tricks that are widely applicable to similar workloads and setups.

As a participant, you can expect to take away a distilled set of learnings to run complex workloads on cost-effective Hadoop clusters using Apache Airflow with processes and techniques that provide resilience, predictability, and performance, as well as help you avoid some gotchas.

Prerequisites:

Basic understanding of Airflow and Hadoop

Content URLs:

1. Introduction (3 mins)
   • Importance of Airflow in data pipeline operations. (1 min)
   • Scale and complexity of data pipelines processing billions of data points daily. (2 mins)
2. Data Pipeline Overview (3 mins)
   • Role of Python and Airflow in managing Hadoop cluster lifecycles. (2 mins)
   • Dynamic nature of cluster provisioning and decommissioning. (1 min)
3. Challenges Encountered (7 mins)
   • Challenges of managing the Hadoop cluster state from a workflow orchestrator. (4 mins)
   • Need for effective error handling during cluster provisioning. (3 mins)
4. Solutions Implemented (6 mins)
   • Strategies and solutions adopted to address cluster management challenges. (4 mins)
   • Automation and efficiency achieved through Airflow pipelines. (2 mins)
5. Optimizing Cost and Resource Allocation (4 mins)
   • Insights on optimizing cost and resource allocation for Spark jobs within computation clusters. (2 mins)
   • Techniques to balance performance and cost-effectiveness. (2 mins)
6. Tips and Tricks (2 mins)
   • Practical tips and tricks applicable to similar workloads and setups. (1 min)
   • Best practices for running complex workloads efficiently. (1 min)
7. Q&A (5 mins)

Speaker Info:

Amar currently works as a Solution Consultant at Sahaj Software in Pune. He likes to build things and, more importantly, to fix things. He has worked on a bunch of different kinds of software and is always interested in learning new technologies. He is a past Google Summer of Code Contributor and a maintainer at CRI-O (container runtime for Kubernetes)

During his free time, he likes to go on long walks to observe the beauty of nature. He also enjoys reading about the universe and its origins. While unsure of its truth, he likes to entertain the idea that aliens came from other planets and played a role in our creation.

Speaker Links:

1. Personal website
2. Blog
3. GitHub
4. LinkedIn
5. Twitter