Let's build a thread-safe HTTP Connection Pool in 30 minutes
Tushar (~tushar5526) |
0
Description:
This talk is inspired after I went through the internals of the library urllib3
which is used in famous libraries like requests
for connection pooling and other features. Python's standard library urllib.request
does not provide support for pooling/re-using a connection. Each request made using urllib.request
establishes a new connection to the server and closes it after the response is received, but for multiple requests to the same server, it is better to use the same connection as with each new connection a separate socket connection is created and by reusing sockets the requests will take up lesser resources on the server's end and provide faster response time at client's end. Connection pooling combined with Threading can provide a significant performance boost.
The talk will cover making our own thread-safe HTTP connection pool in a progressive manner using the built-in http
library and how to make it thread-safe, followed by a benchmarking example that compares making requests to a local HTTP server with builtin http
, simple connection pool and using threading alongside connection pool.
The outline for the talk is as follows:
- Introduction to Connection Pooling and why we need it? [4 minutes]
- What does "thread-safe" means? [2 minutes]
- Introduction to
Queue
for making thread-safe programs [3 minutes] - Building a simple Thread safe connection pool [12 minutes]
- Benchmarking using the Simple Connection Pool, to see it all happen [4 minutes]
- Q&A [5 minutes]
Prerequisites:
As this is a beginner-focused talk, the only pre-requisite is Python.
Speaker Info:
Tushar is a recent graduate and an engineer at Canonical, working with the Launchpad team. He is also an alumnus of GitHub Campus Expert. He has been part of the MLH Fellowship and Processing Foundation Fellowship and is currently the maintainer of p5py - a creative coding library written in Python. He loves playing around with open-source projects in his free time. You can find him at - https://linktr.ee/tushar_5526
Speaker Links:
OpenSource contributions
- Processing Foundation Fellow
- MLH Fellowship with GitHub as the organization
- Contributed/Contributing to projects like p5py, urllib3, flagsmith, fury-gl, openhorizon and many more!