Creating Highly Scalable Fault tolerant Distributed Task processing systems with Celery, Python, Rabbitmq and Kubernetes

Deepanshu Lulla (~deepanshu1)


2

Votes

Description:

Title: Creating Highly Scalable Distributed Task Processing Systems with Celery, Python, RabbitMQ, and Kubernetes

Abstract:

Distributed task processing has become crucial in this era of big data and real-time analytics. This talk will explore the process of building a highly scalable, distributed task-processing system leveraging the power of Celery, Python, RabbitMQ, and Kubernetes. Attendees will learn how to configure, deploy, and scale a fault-tolerant task processing system that can efficiently handle immense workloads, ensuring optimal resource utilization and seamless application performance.

Outline:

  1. Introduction:

    • Overview of the importance and challenges of distributed task processing in modern applications.
    • Brief introduction to Python, Celery, RabbitMQ, and Kubernetes and their roles in building distributed task processing systems.
  2. Task Processing with Celery and Python:

    • Understanding Celery: Architecture, workers, tasks, and brokers.
    • Exploring the use of Python with Celery for creating tasks.
  3. RabbitMQ: Message Broker for Scalability:

    • Understanding the role of RabbitMQ in the system.
    • Advantages of using RabbitMQ as a message broker with Celery.
  4. Kubernetes: Orchestrating our Distributed System:

    • Basics of Kubernetes and its significance in maintaining distributed systems.
    • Kubernetes configurations for deploying our Celery-RabbitMQ setup.
    • Scaling and managing the system effectively with Kubernetes.
  5. Real-world Case Study:

    • Demonstrating a real-world use case: Building a task queue system for a high-traffic e-commerce website.
    • Showing how the system effectively manages and scales with fluctuating demands.
  6. Best Practices and Pitfalls:

    • Discussing some best practices when implementing distributed task processing systems with these technologies.
    • Highlighting common pitfalls and how to avoid them.
  7. Q&A:

    • Addressing queries and discussions about the session.

This presentation aims to equip attendees with practical knowledge to efficiently design, build, and manage a highly scalable distributed task processing system using Python, Celery, RabbitMQ, and Kubernetes. The session is designed for software engineers, system designers, and DevOps practitioners who are interested in distributed systems and task processing.

Prerequisites:

Python Kubernetes basics Distributed systems basics

Speaker Links:

https://deepanshululla.com/

Section: Distributed Computing
Type: Workshops
Target Audience: Advanced
Last Updated: