Design Patterns to make your Python APIs fast
AbdealiJK (~AbdealiJK) |
Description:
Abstract
Whether you are writing Web APIs with Python using FastAPI, Flask, Django, Tornado, or any framework really - there are many times when you would be thinking
Darn, why is this API so slow !
I've faced this numerous times in my career. And while at the start when I asked around for advice, I frequently heard things like "Yep, what were you expecting with python - Just use Go or Java". But I've always loved the pythonic ways of doing things, and so took it upon myself to dig deep into libraries, packages, documentation, etc. to identify how I can make my Python APIs faster !! I found various approaches and solutions on how to tackle my performance issues - like the N+1 ORM Problem, how abstraction can sometimes lead to performance penalties, runtime vs compile time efficiency, SQL query writing, etc. And over time through numerous StackOverflow answers, blog articles, and benchmarks - have identified a few patterns that most of us get into due to the way we learn from the internet.
This talk goes into details about good practices to follow, and also discuss the right time to follow them (to avoid premature optimization!!) Find out common design flaws in your architecture and solve them to make your APIs run super fast⚡
Patterns to discuss
Database Query Performance
- How to analyze your SQL query performance and identify if the database is the culprit
- N+1 problem in ORM queries
- Loading strategies and how to automate them with your VIewSchema
API Performance
- Organizing APIs to avoid unnecessary data load from an RDBMS
- Creating JSON payload structures which can be efficient
- Async vs Sync API and performance considerations
Syntax abstraction issues
- Properties vs Functions - class properties and object properties
- When not to rely on the ORM
Initialization and import time
- Identifying import time performance overheads
- Global vs Local initialization and imports
- Runtime vs Compile time logic - potential benefits of typing
Prerequisites:
- Basic knowledge of Python Syntax
- Basic knowledge about RDBMS including Foreign Keys and indexes
- Should know any 1 Python API framework - like FastAPI, Flask, Django, Tornado, CherryPy, etc.
- Have an understanding of Rest API concepts and common components like - ORM Models, View Schema, etc.
Content URLs:
Some blogs that are good reads in this topic:
- Why ORMs are slow
- The N+1 select query problem
- asyncio python and API performance
- Core python performance improvements with pyston-lite
- Swagger OpenAPI Spec
I have given smaller lightning talks on performance in many different parts. Here are some slides from those talks that I plan to refer to:
- sqlalchemy performance and solving the N+1 problem - Talk given in Bangpypers 2022
- How JIT solutions improve python core language performance - Talk given in PyDelhi 2023
Speaker Info:
I have been working with Python for the part 14 years - Since 8th grade in school Worked with opensource projects for the past 10 years - Since GSoC while I was at IIT Madras. Some notable projects I have contributed to include flask, marshmallow, sqlalchemy, GNOME, Wikipedia, etc. I’ve dabbled a lot with Javascript and Python - and love mixing boundaries between the two languages to learn the best practices from both sides.
I work as the CTO at Corridor Platforms and work towards creating a Risk Management software which is used by some of the largest banks in the world. My main objective is to create a stable and performant system while allowing developers to have a good developer experience. I strongly believe that software needs to be fun to develop and have a great dev-exp for developers, while catering to “business” needs of Performance, Maintainability, and Stability.
Why I am passionate about this topic: I've been using Python since Python2 and I really like it ! I have gotten fed up of explaining to people about the benefits of Python and why it is "not as fast" as Java, C, etc. I believe each tool has it's benefits, and if you can use it correctly then it can be an amazing tool in your repertoire. When comparing pure performance - obviously python is slower than Java/Go/C. But when writing APIs, I have found that the "slowness" of python is typically because of bad coding practices that can be solved with some rigor and discipline (which I know does not come naturally with Python!)
Speaker Links:
Recent talks I have done