Let's Hunt a Memory Leak
Python being a high level interpreted language, it never bothers us to deal with garbage collection. However, because we do not explicitly free memory like we do in C language, python has to clean it for us and hence there is garbage collection involved with python. While garbage collection works most of the times and never becomes an issue, when there is a memory leak in the code, we have no choice but to dig deeper to uncover it.
In this talk we will first look at how python manages memory and how does garbage collection works with python. We will look at some hands on examples to confirm the python memory management behavior and we will see why is it important to be aware about this behavior.
Further, we will look at a simulated memory leak scenario which we faced in production with Flask environment and we will look at one of the ways we used to hunt it down.
- Introduce Python Memory Management
- look into concept of memory arena, pool and blocks
- Look at an example to understand memory management behavior
- Introduction to the memory leak
- and confirm the existence of leak
- Explore possible solutions
- core dump analysis
Understand python memory management and be able to debug memory related issues better.
Let's understand how python deals with memory and garbage collection, with that knowledge we will hunt a production grade memory leak!
Sanket is currently a Site Reliability Engineer with LinkedIn where he is working on Capacity Engineering initiative. He previously worked with Directi where he took care of Hadoop infrastructure, metrics, monitoring, and incident management pipelines. He is also into cycling and blogging.
Man page: http://sanket.superuser.blog
Previous talk: Serverless meets CI/CD, a hands on session at Large Scale Production Engineering Forum, Bangalore
Upcoming talk: Let's Build a Distributed File System at SRECon Asia, 2019 at Singapore