Lessons from optimising inter-process communication: Path to zero-copy

Kartheek S (~kartheek)


4

Votes

Description:

This talk aspires to be an insightful journey into the world of Inter-Process Communication (IPC) in Python, where we will take a neutral and unbiased look at various IPC techniques like sockets, shared memory and many more. The focus is on performance and efficiency over other things since the talk was inspired by a batch processing workload where I was looking for ways to efficiently send large messages between processes. We'll delve into the trials and errors I encountered along the way.

Throughout the session, we will benchmark and analyse different IPC methods, evaluating their strengths and shortcomings, and discussing their suitability in scenarios involving substantial data transfers. I hope that on attending this talk, you'll gain a deeper understanding of how to optimise your Python applications for IPC, especially when handling significant message sizes.

The talk is suitable for all Python developers, from beginners to seasoned professionals, as it equips you with the knowledge and tools to make informed IPC decisions for your future projects. So buckle up for the ride and I'll see you at the venue 🚀

Prerequisites:

  • Python fundamentals
  • IPC concepts (pipes, shared memory, sockets etc)
  • Familiarity with modules like concurrent.futures , multiprocessing and threading
  • Basic OS concepts

Speaker Info:

Meet Kartheek, Head of Engineering at ScaleGenAI, with extensive experience in Python, Deep Learning, and Cloud/System Architecture. Holding a bachelor's degree in Information Technology, Kartheek has led various projects, including MLOps and highly available backend systems. His expertise, in Python, lies in concurrency and OOP, and he is currently focused on reducing Deep Learning model training time by 10-200x at ScaleGenAI.

Speaker Links:

Section: Concurrency
Type: Talks
Target Audience: Intermediate
Last Updated: