Celery tasks + distibuted locks using AMQP
Manu Singhal (~manu31) |
Description:
Celery is a tool to manage and execute asynchronous tasks in a distributed manner. At Mammoth, we use celery extensively for task management. We faced a situation where several tasks attempted concurrent modification to a shared resource resulting in an inconsistent end state of the resource. This is a typical binary semaphore problem, only applied in a distributed context. Celery, out of the box doesn’t provide a solution to this as a primitive. We implemented a binary semaphore using AMQP (Advanced Message Queuing Protocol) and using it with Celery which solved this problem.
Following is an outline of the talk:
- Problem context - Distributed tasks - shared resources [3-4 minutes]
- Celery and AMQP - fundamentals [8-10 minutes]
- Using AMQP to implement distributed locks in python [6-8 minutes]
- Controlling celery tasks using distributed locks [1-2 minutes]
- Responsiveness vs Throughput tradeoff. Semaphores and AWS auto-scaling [1-2 minutes]
- QnA
Who is this talk for?
- Developers with experience in web framework and working on distributed task management.
Key takeaways
- Understanding about distributed locks /semaphores
- Inter-task communication in celery using AMQP
Prerequisites:
Familiarity with messaging protocols, preferably AMQP. Good to have familiarity with Celery
Content URLs:
The technique to design distributed locks using AMQP is described in this blog article.
Speaker Info:
Manu SInghal is a graduate from the BITS, Pilani with Masters in Computer Science and has 4 years of experience of full stack development.
Currently, he is working as a Senior Software Engineer at a tech startup - Mammoth Software Inc (https://www.mammoth.io).
Email: manusinghal19@gmail.com