Concurrency in Python 3.0 - Writing concurrent and parallel programs in Python

Anand B Pillai (~anand5)


42

Votes

Description:

This workshop is influenced and partly derived from my PyDelhi workshop "Concurrency in the Python 3.0 world" given this year.

From my experience, most Python developers aren't still aware of the fundamental principles of concurrent programming, parallel computing and how to identify problems that yield well to data parallelilsm.

While the PyDelhi workshop focussed on introducing different types of concurrent programming techniques available in Python, this workshop wil be focussed more on concurrency using multiple processes - via the multiprocessing library and concurrent futures modules.

The focus is to enable the attendees to pick up the skills to write programs that scale to more than 1 CPU core.

We will start off with an introduction to concurency and parallelism plus why the GIL behaves the way it does with an example.

  1. What is Concurrency - 5 min
  2. Concurrency vs Parellelism - 5 min
  3. The GIL weakness - Multithreading vs Multiprocessing - 10 min
    1. Prime number example with multiple threads & multiple processes
    2. Illustrating how GIL forces computing to 1 core

Next will be an introduction to Multiprocessing & Concurrent futures with some simple examples.

  1. Quick introduction to multiprocessing - 10 min
  2. Concurrent futures - Introduction (10 min)
    1. ThreadPool vs ProcessPool executors

Now time for example problems which can be scaled using data parallel technique.

  1. "Data Parallel" problems - 5 min
  2. Matrix Multiplication - 20 min
    1. Serial
    2. Parallel version using Multiprocessing
  3. Mandelbrot Fractals - 20 min
    1. Serial
    2. Parallel version using Multiprocessing
  4. A simple web crawler - 20 min
    1. Serial
    2. Concurrent version using Multiprocessing
    3. Concurrent version using concurrent futures

[Break] - 5 min

Next I will demonstrate a Maze solver in Python and then show how to speed it up using parallel processing. (Note that this is experimental and maybe be replaced with something else in the actual talk !)

  1. Maze solver in Python (30 min)
    1. Linear
    2. Speed up using Multiprocessing

Next the aspect of timing and measuring performance of your parallel/concurrent code.

(30 mins)

  1. Why wall clock time is not all that matters
  2. Using simple "time" command
  3. Using "timeit" module
  4. Debugging concurrent code

(If time allows)

  1. Generators & concurrent futures - Some advanced examples - 15 min

Prerequisites:

  1. Python programming fundamentals - Knowledge of generators is useful indeed.
  2. Some awareness of Python GIL.
  3. Some awareness of concurrent computing.
  4. The code & discussions will be based on Python 3.x - the latest at that time.

This workshop is NOT for those who are just starting out as Python programmers.

Content URLs:

The latest workshop code and presentation is at.

https://github.com/pythonhacker/pyconindia2017concurrency

I may add more examples in the next few days but what is there now will not change!

Speaker Info:

Anand has been a long time advocate and community leader of the Python programming language in India and Bangalore.

Anand founded the BangPypers community in Feb 2005 as the result of a meeting of Bangalore Pythonistas. Discussions in this community went on to build other communities and laid the foundation of PyCon India and similar conferences.

He has 17+ years of software development experience having worked in a variety of technical roles in a number of software companies. He is currently working as Senior Architect at Yegii Inc., a startup from MIT, where he spends his time writing web crawlers to perform focussed and deep crawls for structured and unstructured data and to develop the next generation AI search engine for knowledge discovery.

His interests are high performance computing architectures, large-scale web crawling, information extraction & security.

Anand is also the author of a book discussing Python and Software Architecture for Packt Publishing, published in April 2017.

Section: Concurrency
Type: Workshops
Target Audience: Intermediate
Last Updated:

Interesting topic for myself. I am trying my hand on high performance computing especially coupled partial differential equations of complex physical systems for simulating nanoparticle behaviour. Parallelising my codes is a logical step and will save a lot of time. Hence, looking forward to attend the workshop. By any chance, if you take a workshop somewhere sooner, please let me know at sandeep.nagar@gmail.com. I am also proposing to give a talk on solving differential equations using python's FEniCS package.

Sandeep Nagar (~sandeep2)

Thanks for the comment Sandeep. I am touching the fundamentals of data and compute parallelization in this talk. What I am not covering are libraries which provide specialized compute parallelization - such as MPI, PyCUDA etc. For parallelizing FEM computations I think such libraries are perhaps far more useful than the fundamental approach in my talk - using multiprocessing and concurrent futures. If given time I may touch upon two libraries namely joblib and PyMP - but the focus of the talk is not on specialized parallel/concurrent processing libraries.

I will check out your proposal on FeniCS.

Anand B Pillai (~anand5)

Hi,

Is the presentation going to be the same(https://github.com/pythonhacker/pydelhi2017concurrency/blob/master/slides.html)?

If not then could you please upload your presentation for review by the Pycon India Team.

Thanks!

Gourav Chawla (~Gouravchawla)

Interesting topic. Can you please provide some links for Prerequisites? This will be helpful. Thanks.

Ayush Kesarwani (~ayush2)

@Goruav - Yes the presentation will be same or slightly modified but mostly the same. The content (notebook) will be different. I wont be able to share content now. Will do after a week.

Anand B Pillai (~anand5)

@Ayush - Pre-requisites are given in the proposal itself. If you got any specific questions on them, ask me and I will be glad to answer.

Anand B Pillai (~anand5)

This was one of the most well received talks in PyDelhi Conf 2016 +1 for this talks.

Akshay Arora (~akshayaurora)

I will be updating code within a week with the new examples.

Anand B Pillai (~anand5)

To people who are attending this workshop.

The final code is pushed to the following Repository on Github.

https://github.com/pythonhacker/pyconindia2017concurrency

Please check the repository out. There is no special software required - except a couple of dependencies mentioned in the requirements file. You just need Python 3.x. These code examples are tested on Python 3.5.2 and it should work on any version of Python >= 3.5.2.

There is no ipython-notebook for this Workshop. I will be demoing the examples in the console. Code will be demoed using my favorite Editor called Emacs .

Anand B Pillai (~anand5)

Login to add a new comment.