Description:

Abstract

In this talk, we will dive into the world of dynamic workflow automation using Managed Workflow for Apache Airflow(MWAA), a powerful and flexible aws service to programmatically author, schedule, and monitor workflows using Python. We will take you through the entire journey of deploying an end-to-end workflow using MWAA and Amazon ECS containers. Furthermore, we will explore how to orchestrate this deployment seamlessly using ECSOperator, and then wrap it all up by setting up a robust CI/CD pipeline using Bitbucket and Terraform which also includes logging through Sumologic and triggers through AWS lambda.

Contents

Exploring Managed Apache Airflow (MWAA)(5 mins)

• Introduction to Managed Apache Airflow (MWAA)
• Considerations of using MWAA Deploying MWAA environment
• Integrating with AWS services

Container Orchestration with ECSOperator(5 mins)

• Leveraging Amazon ECS for container orchestration
• Introducing ECSOperator: Automating task and service management
• Integrating ECSOperator with Airflow DAGs with Python

End-to-End Deployment Walkthrough(5 mins)

• Step-by-step deployment of a sample workflow using MWAA and ECSOperator
• Monitoring and managing the deployed workflow

Debugging Python inside MWAA(5 mins)

• How to check MWAA logs
• How to debug inside MWAA

QnA(5 mins)

Prerequisites:

Knowledge of functional programming in Python(>=3.7)

Speaker Info:

A python professional with 10 years of industry experience developing scalable python projects using multiple backend frameworks. Passionate about data and its transformation across various stages of data pipeline and data lifecycle. A core python enthusiast currently working as Senior python developer with Seneca Global IT Services, Hyderabad

Speaker Links:

LinkedIn- Sourav Roy