Concurrency and parallel execution in Python and how the GIL affects it
Simran Jain (~simran0) |
2
Description:
Abstract: Python is the preferred programming language by many developers because of its ease of use and the versatility provided by its rich library. Since it supports so many use cases, including data processing, web development, data science, machine/deep learning and other compute-intensive tasks, there is a need to ensure concurrency and parallelism for better performance and responsiveness of the task being performed. This talk will focus on why and how concurrency/parallelism works in Python. It will also highlight how the GIL works with multithreading and multiprocessing modules, as well as what the trade-off is between each of these approaches through some Python client-server use case examples.
Outline of the talk:
- Concurrency
- Need for concurrency in Python
- Concurrency vs. parallelism
- Locks and semaphores
- Intro to the GIL and its significance
- Concurrency via multiprocessing and multithreading (explanation of how these two work at the Operating System level)
- Multithreading in Python (thread-based parallelism)
- How it works in Python’s threading module
- How the GIL hinders multithreading (not truly parallel)
- Why we need multithreading when the GIL is present
- Multiprocessing in Python (process-based parallelism)
- Why do we need multiprocessing?
- How does it overcome the limitations brought about by the GIL
- How it works in Python using the multiprocessor module
- Python client-server illustrations (code and diagrams) https://github.com/simran2jain/python-examples
- Single client and server
- Multiple clients (simulated using multithreading) and multithreaded server
- Multiple clients (simulated using multithreading) and multi-process server
Prerequisites:
Basics of Python
Content URLs:
- My GitHub repository link for illustrations
- Python documentation on concurrency
Speaker Info:
Simran Jain works as a software engineer at Bloomberg, where she is responsible for designing and developing high performance, scalable, and reliable full stack workflows for Bloomberg Terminal applications. Prior to joining the firm, she earned a master’s degree in computer science and engineering from the National Institute of Technology (NIT), Trichy.
In this role, she uses Python to develop testing frameworks. She also has some experience using Python for machine learning/deep learning projects.
Simran has delivered a couple of lightning tech talks internally at Bloomberg. She has also presented her research work virtually during the 2022 3rd International Conference for Emerging Technology (INCET) on behalf of NIT Trichy. Speaking at PyCon will be an opportunity for her to deliver his first full-fledged technical talk, about which he is very excited.
Email: sjain705@bloomberg.net LinkedIn profile: Simran Jain
Speaker Links:
Published work: https://dl.acm.org/doi/abs/10.1145/3083201.3083202 and https://ieeexplore.ieee.org/document/9824941