The Python concurrency story
Madhukar Mishra (~madhukar93) |
You’ve heard a lot about concurrency. Asyncio has been in the standard library for a while, and concurrency is picking up mindshare. Why does the world suddenly care so much about concurrency?
How did people write concurrent code before asyncio? Do we still need multithreading, multiprocessing, Gevent, Tornado, etc now that asyncio is here?
You’ve also heard about the GIL. Supposedly, it doesn’t let you write parallel programs - so why does Python have it?
We’ll also discuss the kinds of problems that can be solved faster with concurrency and the kinds of problems that definitely can’t.
Let's answer all these questions and more in this light, demo-driven talk.
- why is concurrency important (2 min, total: 2 min)
- The free lunch is over
concurrency is not parallelism (2 min, total: 4 min)
- The all important difference
Why you should care
Threading, GIL and the dilemma of concurrent programming in python. (6 min, total: 10 min)
- code demo of basic threading features and some nifty benchmarking
- GIL in action
Why the GIL ?
multiprocessing and leveraging multiple cores effectively (5 min, total: 15 min)
- leveraging multiple cores in python
code demo of basic multiprocessing features and some nifty benchmarking
sync vs async (1 min, total: 16 min)
- async vs sync
- async vs concurrent
- asyncio - the best way forward? (6 min, total: 22 min)
- event loop and co-routines
code demo of basic asyncio implementation with some nifty benchmarking
conclusion: making the right choice (3 min, total 25 min)
- pick the right tool for the right job
- honorable mentions (third party libs): gevent, tornado
code samples to solve a similar problem to compare and contrast approaches
- A talk for the Gurgaon Golang meetup group on Go's concurrency - https://github.com/madhukar93/concurrency_patterns/blob/master/csp/presentation.slide
I am a full stack dev at Grofers where Python is our first choice for everything from backend services to infrastructure orchestration. I have worked on a wide range of solutions using Python, among them are APIs that power our consumer ecosystem at scale, inventory/content management systems, and other high throughput ETL services in Python.