Description:

Generators Explained

Generators have a variety of uses including lazy iteration, maintaining state and pausing function execution, pipelining data through multiple stages, and concurrency as a whole. They form a rich and powerful part of the core Python language, and a working knowledge of how to use them will serve as an indispensable tool for every Python programmer.

Objective

The first part of the talk aims to provide a conceptual understanding of what generators are, how to make them in Python, and how to use them to make your code more efficient and sleek. The second part focuses on understanding how they form the core foundation of concurrency in Python.

• The talk will begin with an introduction of the yield keyword, and how to create generators with it. This will be followed by showing how they can be used for lazy iteration and saving memory. A brief example in a real world application related to data loading in machine learning will also be presented.
• The next part will focus on the ability of the yield statement to accept values, and will explain how this can be used to make stateful functions with minimal and elegant code. The yield from keyword will be introduced and the particularly mind-bending example of binary tree traversal will be used to showcase its power.
• The next part will point out important differences between yield from and the yield keyword. We will then use it to create _co-routines_ - functions that can transfer control between each other mid-execution.
  • A subtle and important distinction between _parallelism_ and _concurrency_ will be highlighted with a simple example through co-routines built using yield from.
  • The concept of an event loop will also be introduced through this example.
• The talk will conclude with a discussion of the async and await keywords, and how the asyncio library can be used to streamline writing concurrent Python code.

Target audience

Beginners who are familiar with the basics of Python. Exposure to basic concepts like iteration using for loops and while loops is required. Knowledge of classes will help. A brief introduction to concepts behind concurrent programming will be a bonus.

Outline

• What are generators? (~5 min)
  • Recap of standard iteration
  • Iteration using generator functions - Lazy Iteration
    • Infinite series
• Generator Expressions and Data Pipelining
• How do they work? (~7 min)
  • Sending data back into generators
    • Example - Running metric evaluation
  • Stacking generators and the yield from statement
    • Example - binary tree traversal
• A foray into async and co-routines (~10 min)
  • Understanding yield from and how it differs from yield
  • A simple example of an event loop
  • Co-routines using async and await
  • Creating and running tasks concurrently using the asyncio library

Prerequisites:

Beginners who are familiar with the basics of Python. Exposure to basic concepts like iteration using for loops and while loops is required. Knowledge of classes will help. A brief introduction to concepts behind concurrent programming will be a bonus.

Content URLs:

Video recording of a talk with similar content I gave at PySangamam 2018 - link

This talk at PySangamam was planned for a duration of 20 minutes + questions, so I was not able to cover material on async and await. I will be able to include this for the 25 minute talk duration at PyCon.

Online slides for the talk here: link

Draft materials I used for the talk: link

Speaker Info:

My name is Rajat Vadiraj Dwaraknath and I’m currently a 4th year undergraduate student pursuing a degree in Electrical Engineering at IIT Madras. I’ve always been fascinated when seemingly unrelated concepts from different fields and subjects like mathematics and physics link together to form an insightful explanation of the world around us. I realized that a strong link could be made between these two fields through computational mathematics -- and consequently through programming. While I began my journey into the vast world of programming through Java, I was soon converted by the elegance and power of Python. Once I discovered the efficiency of Python in converting ideas and concepts in my head to actual working code with minimal effort, I haven't looked back. As a member of the Computer Vision and Intelligence group at IIT Madras, I have used Python for deep learning and computer vision applications, and also given numerous talks and sessions in workshops related to these fields. I also use it constantly in my personal projects, research and even coursework as well. For instance, I used Python extensively in my research internship at Stanford University related to applying deep learning methods in solving and modeling partial differential equations.

Speaker Links:

I maintain a blog about machine learning and math here: link

My GitHub page: link