How We Built a Celery-Powered Task Queue Service to Scale Infra Jobs
Bhargav Bardipurkar (~bhargav5) |
5
Description:
Abstract: In this talk, we'll explore how Celery, a powerful distributed task queue, can be leveraged to build micro services that handle mundane tasks efficiently. We'll delve into the concept of Bring Your Own Cloud (BYOC) for deploying machine learning models on Kubernetes, breaking down each step as a job. We'll also discuss how Taskman, a service accessible via API, fits into this ecosystem, and share patterns and best practices for observability in Celery. By the end of this session, you'll have a clear understanding of how to build, deploy, and manage distributed micro services effectively.
Outline:
Introduction to Celery: (2 min) Overview of Celery and its role in distributed task management. Benefits of using Celery for automating mundane tasks.
Concept of BYOC: (3 min) Explanation of Bring Your Own Cloud (BYOC). Steps to deploy machine learning models on Kubernetes using BYOC.
Building the Microservice: (3 min) Breaking down the deployment process into discrete jobs. Example: ICE (Infrastructure Creation Engine) creating Pulumi templates with infrastructure details. Automating infrastructure creation and management.
Job Queue as a Service Using celery (we call it Taskman): (8 min) Introduction to Taskman and its API integration. Use cases and benefits of using Taskman for task management.
Patterns and Best Practices: (2 min) Common patterns in using Celery for distributed microservices. Best practices for building and managing tasks.
Observability in Celery: (2 min) Challenges of observability in Celery. Tools and techniques to monitor and debug Celery tasks.
Integration with Tune Chat: (2 min) Demonstrating how Tune Chat can be integrated into this workflow. Behind the scenes of Chat file upload and celery workers getting triggered etc Practical use cases and examples.
Q&A: (3 min) Open floor for questions and discussion.
Format: Talk: Introduction and key concepts. Demo: Live demonstration of deploying a machine learning model using BYOC, Taskman, and Celery. Q&A: Interactive session with the audience.
Prerequisites:
Knowledge about kubernetes and Celery
Speaker Info:
Bhargav is a Senior Software Engineer at Tune AI. With over 5 years of experience in the industry