Python and Parallelism - When to and When not to

anirudhsriniv


Description:

Introduction

We have all heard about how modern programs make effective use of multi threaded CPUs on our PCs. When our python programs take a long time to run, we immediately think about how to parallelize it and make it faster. As we google for a solution, we enter the dark and murky mess that is multiprocessing in python. Gathering from my experience in parallelizing code for reinforcement learning (running multiple OpenAI gym environments in parallel), I present a talk on de-mystifying the concept of multiprocessing in Python and how to determine if it is the right technique to apply for the task you have

Outline

  1. How are processes scheduled
  2. How does parallelism help
  3. Multithreading
    • Why it doesn't work
    • The GIL
  4. Multiprocessing - The actual way to do it
    • Pool - the simple way
    • Processes and Pipes - more flexibility
  5. When not to use multiprocessing
  6. When to use multiprocessing
  7. Common mistakes while using multiprocessing
  8. Performance comparisons

Who is this task for

People who want to understand the versatility of python

People who want to learn about parallelism and if it is the right technique for his/her task

Prerequisites:

  • Basic Python
  • Curiosity to Learn

Content URLs:

BabyAI: An example of parallelizing OpenAI gym environments across multiple processes : Link

Python Docs: Multiprocessing

Speaker Info:

My name is Anirudh Srinivasan. I graduated from BITS Pilani Hyderabad Campus in 2019 with a degree in Computer Science and I am a Machine Learning researcher. I am currently a Research Fellow at Microsoft Research, Bangalore, where I am working on Natural Language Understanding problems. I previously interned at MILA - Quebec Artificial Intelligence Institute in Montreal, Canada, where I worked on Reinforcement Learning and on the BabyAI Project, a project at MILA to study techniques for Grounded Language Learning (an instruction following task).

Speaker Links:

Github: Genius1237

Blog: github.io

Id: 1283
Section: Core Python
Type: Talks
Target Audience: Intermediate
Last Updated: