Building an autoscale micro-services architecture using Celery on Kubernetes

Ayush Agarwal (~thisisayush)




Any Django Developer today has used celery as a means to run tasks in background, and traditionally the celery workers for these tasks have been deployed across different or same servers with N concurrency, while still managing the resources (nodes, number of workers) manually when needed. With hacks and techniques developed or explored over years people found their own ways to automate it.

In this talk, I cover how modern day technolgies can help reduce the overhead and help you create a native autoscaling architecture that scales from 0 to N on demand, helping you save time, cost and effort spent in maintenance.

The talk will cover: 1. Overview of Celery and Tasking in Django 2. Having integrated and completely independent celery tasks and how they can be invoked from Django. 3. Deploying celery tasks in kubernetes 4. Using Keda and RabbitMQ to setup autoscalers in Kubernetes 5. Live Demonstration and real life uses


  1. Basics on Django and Celery
  2. Basics of Docker
  3. YAML Syntax

Speaker Info:

I'm Ayush, I am a Full Stack Engineer with 5+ years of experience. I also run a Software Development and Consulting Agency in Noida (Remote First). I've created over 100 Products in the timespan (micro and large) and have worked with a lof of founders (early age startups) and helped them in building scalable architectures and products.

Speaker Links:

Web Handle: @thisisayush (X: @thisisayushaa) LinkedIn: Github:

Section: Python in Web and Applications
Type: Talk
Target Audience: Intermediate
Last Updated: