Making Monorepos work with Python
AbdealiJK (~AbdealiJK) |
2
Description:
As a python project grows (within 2-3 years), you will go down either of these 2 paths:
- Create a monolith
- Modularize your code into smaller packages
In the current world, you will be affected by multiple other libraries you use. And modularity is a requirement for any good project.
But managing multiple modular packages becomes tough over time.
- How do you ensure coding standards (quality, styling, etc) are consistent across them ?
- How would we ensure all the packages work correctly without spending hours and hours of CI time ?
- How can common logical pieces be modularized further and still be DRY ?
These are common issues I have faced by the 2-3 year mark in any active project. And if not solved quickly can easily cause your project to get messy very quickly.
This talk aims to discuss these common issues and how a monorepo structure which is widely popular in other communities like NodeJS can also be applied to python.
We also discuss how the crux of the issue:
- Making your code structure machine understandable
- How this structured information can then be used to optimize workloads
- How this structured information can be used to automate tasks
And also go into discussing how monorepo tools like pants, bazel, nx, etc. leverage this code structure information to simplify your life as a maintainer
Prerequisites:
- Reasonable experience with python projects
- Tools used in development: git, ruff, flake8, pytest, etc.
- Some experience with python packaging and dependency handling
Content URLs:
I have given this talk in PyCon DE & PyData Berlin Slides from that time are in this Google slides You can also see the video from that talk on youtube
Technical requirements:
The talk will be 10 mins of theory and 20 mins more practical. I generally prefer showing things than talking about them.
The participants should know the basics of how to work with python and common tooling used - like linters, formatters, package managers, CI pipelines.
I believe these all have become mainstream and everyone knows about them so I don't feel like there is a major barrier to take away something from this talk.
Speaker Info:
My experience with Python:
- I have been working with Python for the part 13 years - Since 8th grade in school
- Worked with opensource projects for the past ~10 years - Since GSoC while I was at IIT Madras. Some notable projects include flask, marshmallow, sqlalchemy, GNOME, Wikipedia.
- Leading a team of 20 developers for the past 5 years to build a product - Mainly ensuring nobody can break our product by mistake :)
Why I am passionate about this topic:
When creating my own project I realized that the maintenance overhead was increasing steeply.
I am someone who believes that developer experience is very important, because if you don't enjoy developing your product then everything else doesn't really matter as much.
I spent a significant amount of time thinking through how to improve developer and maintainer experience and would love to share the learnings I've had in this regard.
Speaker Links:
Github: https://github.com/AbdealiLoKo
Recent talks I have done