UnicornTask: Streamlined Task Orchestration for Python Web Applications using gunicorn
Derrick Joseph (~derrick) |
5
Description:
Abstract: UnicornTask is an innovative micro-task orchestration framework designed to integrate seamlessly with your Python web applications. By leveraging Gunicorn's pre-fork worker model, UnicornTask offers a streamlined and efficient approach to managing both web requests and out-of-band compute/IO heavy tasks. This talk will introduce UnicornTask, its architecture, and the advantages it brings to Python developers
Detailed Description: Gunicorn uses a pre-fork worker model to spawn child worker processes, with the parent process managing these child workers using basic Unix primitives. UnicornTask extends this model by spawning task execution child processes alongside WSGI processes that serve your application's web requests. This allows for efficient handling of out-of-band compute/IO heavy tasks.
In UnicornTask, each task or operation becomes an individual Gunicorn worker that listens on a message-broker queue instead of a socket (like WSGI workers). The message-broker interfacing is implemented using Kombu, which is also the message-broker interface for Celery. This provides significant flexibility regarding the choice of message-broker platform.
UnicornTask retains the familiar configuration parameters of Gunicorn, ensuring zero setup hassle. Gunicorn's use of native Unix primitives facilitates seamless scaling, and horizontal scalability is as simple as spinning up additional hosts running UnicornTask that share the same message broker. Implementing a task is straightforward, requiring just a subclass or a decorator, with worker management handled by Gunicorn.
UnicornTask was developed in-house to address the specific needs and challenges faced in managing out-of-band tasks within our Python web applications. This bespoke solution has proven highly effective in enhancing performance and scalability.
This session aims to provide Python developers with a comprehensive overview of UnicornTask, demonstrating its potential to enhance the performance and scalability of their web applications.
Content URLs:
Speaker Info:
Derrick Joseph is a Senior Software Engineer at LinkedIn, bringing nearly a decade of experience to the industry. He launched his career at a small startup, where he honed his skills and gained invaluable hands-on experience. Derrick then moved to Saama AI Research in Chennai, where he focused on designing MLOps frameworks. During his tenure at Saama, he delivered a talk on implementing CSP pattern using asyncio and how that helps in deploying Tensor based Deep Learning models at PyCon India 2017, highlighting his innovative work in this area.
Currently, Derrick is part of LinkedIn's Site Engineering team, where he develops tools to manage LinkedIn's extensive fleet of machines. His work involves addressing scalability challenges and optimizing infrastructure.
In addition to his professional work, Derrick is an active volunteer with KochiPython, contributing to the local Python communityw.
In this talk, Derrick will present the framework he developed to solve for scalability in Python applications at LinkedIn leveraging the Actor pattern using gunicorn. His insights will provide valuable takeaways for developers and engineers facing similar challenges in their own organizations.