Let's build a thread-safe HTTP Connection Pool in 30 minutes

Tushar (~tushar5526)




This talk is inspired after I went through the internals of the library urllib3 which is used in famous libraries like requests for connection pooling and other features. Python's standard library urllib.request does not provide support for pooling/re-using a connection. Each request made using urllib.request establishes a new connection to the server and closes it after the response is received, but for multiple requests to the same server, it is better to use the same connection as with each new connection a separate socket connection is created and by reusing sockets the requests will take up lesser resources on the server's end and provide faster response time at client's end. Connection pooling combined with Threading can provide a significant performance boost.

The talk will cover making our own thread-safe HTTP connection pool in a progressive manner using the built-in http library and how to make it thread-safe, followed by a benchmarking example that compares making requests to a local HTTP server with builtin http, simple connection pool and using threading alongside connection pool.

The outline for the talk is as follows:

  • Introduction to Connection Pooling and why we need it? [4 minutes]
  • What does "thread-safe" means? [2 minutes]
  • Introduction to Queue for making thread-safe programs [3 minutes]
  • Building a simple Thread safe connection pool [12 minutes]
  • Benchmarking using the Simple Connection Pool, to see it all happen [4 minutes]
  • Q&A [5 minutes]


As this is a beginner-focused talk, the only pre-requisite is Python.

Content URLs:

The example I will be building is inspired by the implementation of connection pool in this library. https://github.com/urllib3/urllib3

Speaker Info:

Tushar is a recent graduate and an engineer at Samagra Governance working in the space of making an impact through tech. He is also an alumnus of GitHub Campus Expert and has been part of the MLH Fellowship and Processing Foundation Fellowship and is currently the maintainer of p5py - a creative coding library written in Python. He loves playing around with open-source projects in his free time. You can find him at - https://linktr.ee/tushar_5526

Speaker Links:

LinkedIn, Twitter, GitHub

OpenSource contributions

  • Processing Foundation Fellow
  • MLH Fellowship with GitHub as the organization
  • Contributed/Contributing to projects like p5py, urllib3, flagsmith, fury-gl, openhorizon and many more!

Section: Core Python
Type: Talk
Target Audience: Beginner
Last Updated: