The curious case of slow/fast grequests code
Saurabh Hirani (~saurabh) |
If you have made a single HTTP request in Python, you have probably used the requests module.
If you have made concurrent HTTP requests in Python, you have probably used the grequests module.
I ran into an issue in production which made a grequests based code execute HTTPS GET requests serially without utilising the concurrency goodness promised by grequests. What was more interesting is that depending on the Python modules installed on my system, the same codebase ran fast or slow.
Tumbling down that rabbit hole made me understand Python profiling, function tracing, understanding how gevent works and opening a pull request to fix the issue with a gevent based Python module.
In this talk, I want to share my learnings with the audience through a working dockerized demo which showcases the following stages for the same codebase while explaining why each stage behaves the way it does:
- Stage-0 - Trigger which led to the exploration - May run slow/fast for Python 2.7 and slow/fast for Python 3.7 depending on the modules installed on your system.
- Stage-1 - Baseline with dockerized environment - Runs fast both for Python 2.7 and Python 3.7
- Stage-2 - Detection of slowness and verification - Runs slow both for Python 2.7 and Python 3.7
- Stage-3 - Python 2.7 fix - Runs fast for Python 2.7 but slow for Python 3.7
- Stage-4 - Python 3.7 fix - Patching a gevent based Python module which fixes the issue and makes Python 3.7 fast again.
Please check out the preview video to get a sneak peek into the issue and watch the same codebase run slow/fast depending on the Python version chosen.
Blog posts describing the different stages of the issue:
Demo setup for the talk:
PR raised to fix the issue described in the talk:
- Saurabh Hirani normally doesn't talk about himself in third person, but when he does, he is writing a speaker bio.
- He enjoys programming and infrastructure automation.
- He is currently working as a Site Reliability Engineer at Autodesk and has spoken at Python meetups, Rootconf conferences and firmly believes in sharing hard earned knowledge that everyone can benefit from.