Celery tasks + distibuted locks using AMQP
Manu Singhal (~manu31) |
Celery is a tool to manage and execute asynchronous tasks in a distributed manner. At Mammoth, we use celery extensively for task management. We faced a situation where several tasks attempted concurrent modification to a shared resource resulting in an inconsistent end state of the resource. This is a typical binary semaphore problem, only applied in a distributed context. Celery, out of the box doesn’t provide a solution to this as a primitive. We implemented a binary semaphore using AMQP (Advanced Message Queuing Protocol) and using it with Celery which solved this problem.
Following is an outline of the talk:
- Problem context - Distributed tasks - shared resources [3-4 minutes]
- Celery and AMQP - fundamentals [8-10 minutes]
- Using AMQP to implement distributed locks in python [6-8 minutes]
- Controlling celery tasks using distributed locks [1-2 minutes]
- Responsiveness vs Throughput tradeoff. Semaphores and AWS auto-scaling [1-2 minutes]
Who is this talk for?
- Developers with experience in web framework and working on distributed task management.
- Understanding about distributed locks /semaphores
- Inter-task communication in celery using AMQP
Familiarity with messaging protocols, preferably AMQP. Good to have familiarity with Celery
The technique to design distributed locks using AMQP is described in this blog article.
Manu SInghal is a graduate from the BITS, Pilani with Masters in Computer Science and has 4 years of experience of full stack development.
Currently, he is working as a Senior Software Engineer at a tech startup - Mammoth Software Inc (https://www.mammoth.io).