Generators Explained
Rajat Vadiraj Dwaraknath (~rajatvd) |
Description:
Generators Explained
Generators have a variety of uses including lazy iteration, maintaining state and pausing function execution, pipelining data through multiple stages, and concurrency as a whole. They form a rich and powerful part of the core Python language, and a working knowledge of how to use them will serve as an indispensable tool for every Python programmer.
Objective
The first part of the talk aims to provide a conceptual understanding of what generators are, how to make them in Python, and how to use them to make your code more efficient and sleek. The second part focuses on understanding how they form the core foundation of concurrency in Python.
- The talk will begin with an introduction of the
yield
keyword, and how to create generators with it. This will be followed by showing how they can be used for lazy iteration and saving memory. A brief example in a real world application related to data loading in machine learning will also be presented. - The next part will focus on the ability of the
yield
statement to accept values, and will explain how this can be used to make stateful functions with minimal and elegant code. Theyield from
keyword will be introduced and the particularly mind-bending example of binary tree traversal will be used to showcase its power. - The next part will point out important differences between
yield from
and theyield
keyword. We will then use it to create _co-routines_ - functions that can transfer control between each other mid-execution.- A subtle and important distinction between _parallelism_ and _concurrency_ will be highlighted with a simple example through co-routines built using
yield from
. - The concept of an event loop will also be introduced through this example.
- A subtle and important distinction between _parallelism_ and _concurrency_ will be highlighted with a simple example through co-routines built using
- The talk will conclude with a discussion of the
async
andawait
keywords, and how theasyncio
library can be used to streamline writing concurrent Python code.
Target audience
Beginners who are familiar with the basics of Python. Exposure to basic concepts like iteration using for loops and while loops is required. Knowledge of classes will help. A brief introduction to concepts behind concurrent programming will be a bonus.
Outline
- What are generators? (~5 min)
- Recap of standard iteration
- Iteration using generator functions - Lazy Iteration
- Infinite series
- Generator Expressions and Data Pipelining
- How do they work? (~7 min)
- Sending data back into generators
- Example - Running metric evaluation
- Stacking generators and the
yield from
statement- Example - binary tree traversal
- Sending data back into generators
- A foray into async and co-routines (~10 min)
- Understanding
yield from
and how it differs fromyield
- A simple example of an event loop
- Co-routines using
async
andawait
- Creating and running tasks concurrently using the
asyncio
library
- Understanding
Prerequisites:
Beginners who are familiar with the basics of Python. Exposure to basic concepts like iteration using for loops and while loops is required. Knowledge of classes will help. A brief introduction to concepts behind concurrent programming will be a bonus.
Content URLs:
Video recording of a talk with similar content I gave at PySangamam 2018 - link
This talk at PySangamam was planned for a duration of 20 minutes + questions, so I was not able to cover material on async
and await
. I will be able to include this for the 25 minute talk duration at PyCon.
Online slides for the talk here: link
Draft materials I used for the talk: link
Speaker Info:
My name is Rajat Vadiraj Dwaraknath and I’m currently a 4th year undergraduate student pursuing a degree in Electrical Engineering at IIT Madras. I’ve always been fascinated when seemingly unrelated concepts from different fields and subjects like mathematics and physics link together to form an insightful explanation of the world around us. I realized that a strong link could be made between these two fields through computational mathematics -- and consequently through programming. While I began my journey into the vast world of programming through Java, I was soon converted by the elegance and power of Python. Once I discovered the efficiency of Python in converting ideas and concepts in my head to actual working code with minimal effort, I haven't looked back. As a member of the Computer Vision and Intelligence group at IIT Madras, I have used Python for deep learning and computer vision applications, and also given numerous talks and sessions in workshops related to these fields. I also use it constantly in my personal projects, research and even coursework as well. For instance, I used Python extensively in my research internship at Stanford University related to applying deep learning methods in solving and modeling partial differential equations.