Description:

Abstract: Celery, a powerful and flexible task queue library for Python, has become a go-to choice for handling asynchronous processing in various applications. While it offers great flexibility, its complexity often leads to confusion and challenges in designing efficient workflows. In this PyCon talk, we will delve into my blog post "The Curious Case of Celery Workflows", exploring its insights and expanding on practical strategies to build scalable and robust Celery workflows.

Description: Celery has proven itself as a crucial component in many Python projects, enabling developers to offload time-consuming tasks to background workers. However, as applications grow in complexity and scale, crafting efficient Celery workflows becomes essential to maintain high-performance and avoid potential pitfalls.

In this talk, we will begin by revisiting the key points from my blog post "The Curious Case of Celery Workflows", addressing common issues faced by developers in real-world scenarios. We will cover following topics,

1. Introduction to Celery: We will begin with an overview of Celery and its role in managing asynchronous task execution in Python applications.
2. Chord and Group: Leveraging advanced Celery features like Chord and Group to orchestrate complex workflows and manage dependencies between tasks. Also we will discuss 5 practical cases of Celery workflow creation. Tasks execution in Series (Chain) Tasks execution in Parallel (Group) Tasks execution both in Series and Group (Combination of Chain, Group and Cord) Generating Group tasks from Chain task (Map) Conditional Tasks execution (Map)
3. Best Practices for Task Workflow Design: Sharing tips on designing tasks workflows that are modular, reusable, and easy to maintain.
4. Error Handling and Retries: Discussing strategies to handle errors gracefully within Celery workflows and implementing effective retry mechanisms.
5. Monitoring and Scaling: Demonstrating how to monitor Celery workflows and scale the system to handle increased workloads seamlessly. Real world scenario: Concluding the talk with an in-depth discussion on video processing workflow.

By the end of this talk, attendees will have a comprehensive understanding of Celery workflows and be equipped with the knowledge to build scalable, robust, and efficient asynchronous task queues for their Python applications.

Target Audience:

1. Python developers familiar with Celery basics.
2. Developers working on web applications and services handling complex, time-consuming tasks.
3. Engineers interested in improving the efficiency and reliability of their asynchronous processing.

Prerequisites:

1. Proficiency in Python
2. Celery Basics
3. Asynchronous Concepts
4. Distributed Systems
5. Familiarity with Task Queues

Content URLs:

1. Blog: https://dev.to/akarshan/the-curious-case-of-celery-work-flows-39f7
2. PPT: https://drive.google.com/file/d/1hWo2eb2qNThcvSml-nztLO90yFV-lh2u/view?usp=sharing

Speaker Info:

Akarshan is a passionate Software Developer on a mission to make a significant impact on the world by delivering innovative solutions to everyday challenges through the seamless blend of creativity and technical skills. With a strong affinity for Python, Akarshan has honed his skills in various frameworks like Django, Flask, and FastAPI, crafting robust and scalable applications. He has a keen interest in system design, consistently striving to architect efficient and reliable solutions.

Akarshan is also an avid writer. He regularly shares his knowledge and experiences with the tech community through his insightful technical blogs. His blogs cover a wide range of topics, including Python programming, system designing, Kubernetes, and backend development, providing valuable insights and practical tips to fellow developers.

Akarshan currently works as a Software Development Engineer (SDE-2) at Innovaccer, a HealthTech Unicorn.

Speaker Links:

My Blogs:

1. Create Awaitable Constructors with AsyncMixin
2. Deploying RabbitMQ in Kubernetes with Custom Docker Image
3. Unlocking 100% of NGINX Ingress Controller
4. The Curious Case of Celery Work-flows
5. Mocking Redis in Python's unittest
6. System Designing: Data Stores Decisions ft. COWIN
7. Discussing Bloom Filters
8. Let's Brainstorm Backtracking
9. Android blogs written for contributions in FOSSASIA (GSoC'17)

My Open Source Contributions

10. Drivelert: Drivelert is an android application that helps to combat drowsy driving. Github link Playstore link
11. Pocket Science Lab: Github link Product Link GSoC gist link