Description:

Apache Airflow has revolutionized the way we manage and automate complex workflows in the world of data engineering and beyond. In this session, we will embark on a journey through the fundamentals and advanced features of Apache Airflow. From understanding its purpose and architecture to exploring DAGs, operators, variables, and the user interface, this talk will equip you with the knowledge and skills needed to harness the full potential of Airflow.

Description:

Apache Airflow has become the backbone of workflow orchestration, enabling developers and data engineers to build, schedule, and monitor data pipelines with ease. This talk will serve as a comprehensive guide to mastering Apache Airflow, covering everything from its basic usage to advanced topics.

We will begin by exploring the core concepts of Apache Airflow, including its purpose and architecture. We'll delve into Directed Acyclic Graphs (DAGs) and how they form the backbone of Airflow's workflow management system. Through practical examples, attendees will learn how to define, schedule, and execute DAGs to automate their workflows effectively.

Next, we'll dive into Airflow's extensive library of operators, which enable users to perform various tasks within their workflows. From simple Bash operators to more specialized operators for interacting with databases, cloud services, and beyond, we'll cover a wide range of use cases and best practices for choosing the right operator for the job.

Attendees will also learn about Airflow Variables, which provide a way to store and retrieve dynamic values within their DAGs. We'll explore how Variables can be used to parameterize workflows and make them more flexible and reusable.

Furthermore, we'll take a deep dive into the Airflow user interface (UI), showcasing its capabilities for monitoring, troubleshooting, and managing workflows. Attendees will gain insights into navigating the UI effectively and leveraging its features to enhance their workflow management experience.

Throughout the session, practical demonstrations and real-world examples will be interspersed to illustrate the concepts discussed and provide attendees with hands-on experience.

Key Takeaways:

1. Understanding of Apache Airflow's purpose, architecture, and core concepts.
2. Practical knowledge of defining and scheduling DAGs.
3. Familiarity with a wide range of Airflow operators and their use cases.
4. Insights into using Airflow Variables to parameterize workflows.
5. Proficiency in navigating the Airflow user interface for monitoring and managing workflows.

Target Audience:

This talk is suitable for developers, data engineers, and technical enthusiasts who are interested in workflow orchestration and automation using Apache Airflow. Prior knowledge of Python and basic understanding of data engineering concepts would be beneficial but not required.

Speaker Info:

Speaker's Bio:

Akhil Garg is a Lead Software Engineer with EPAM Systems. With 10+ years of experience in IT and Python projects development, Akhil Garg is passionate about Development, technical management, leadership, giving talks in various meetups and conferences. He is also leading a community within his organisation where he organises talks, workshops and guide others to write blogs. As an active member of the tech community, Akhil enjoys sharing his knowledge and insights through speaking engagements, workshops in various meetups. His previous talks have been in EPAM Systems, PyDelhi monthly meetup, FOSS United meetup, CNCF meetup, Apache Flink Meetup etc. He has talked on topic like Microservice architecture, Hasura, GIT, Apache Airflow, Encryption techniques etc.

Speaker Links:

Linkedin handle https://www.linkedin.com/in/akhilgarg1990