Celery tasks + distibuted locks using AMQP

Manu Singhal (~manu31)


Celery is a tool to manage and execute asynchronous tasks in a distributed manner. At Mammoth, we use celery extensively for task management. We faced a situation where several tasks attempted concurrent modification to a shared resource resulting in an inconsistent end state of the resource. This is a typical binary semaphore problem, only applied in a distributed context. Celery, out of the box doesn’t provide a solution to this as a primitive. We implemented a binary semaphore using AMQP (Advanced Message Queuing Protocol) and using it with Celery which solved this problem.

Following is an outline of the talk:

  1. Problem context - Distributed tasks - shared resources [3-4 minutes]
  2. Celery and AMQP - fundamentals [8-10 minutes]
  3. Using AMQP to implement distributed locks in python [6-8 minutes]
  4. Controlling celery tasks using distributed locks [1-2 minutes]
  5. Responsiveness vs Throughput tradeoff. Semaphores and AWS auto-scaling [1-2 minutes]
  6. QnA

Who is this talk for?

  • Developers with experience in web framework and working on distributed task management.

Key takeaways

  • Understanding about distributed locks /semaphores
  • Inter-task communication in celery using AMQP


Familiarity with messaging protocols, preferably AMQP. Good to have familiarity with Celery

Content URLs:

The technique to design distributed locks using AMQP is described in this blog article.

Presentation link

Speaker Info:

Manu SInghal is a graduate from the BITS, Pilani with Masters in Computer Science and has 4 years of experience of full stack development.

Currently, he is working as a Senior Software Engineer at a tech startup - Mammoth Software Inc (https://www.mammoth.io).

Email: manusinghal19@gmail.com

Speaker Links:

Id: 1422
Section: Web development
Type: Talks
Target Audience: Beginner
Last Updated: