What happens behind execution of an `import` statement?

Shivashis Padhi (~shivashis)


2

Votes

Description:

TL;DR

This talk follows this template to explain what goes on in the background when you execute from spam import ham, or any other import statement. Additionally, it discusses import hooks and how to use them.

Introduction

Every Python script involves import statements. And if you develop software with Python, you might already have come across import and path related errors like ImportError, ModuleNotFoundError, etc.

It's important to know the import mechanics, both for general knowledge, and to develop custom plugins to tweak the import system. In this talk, we'd discuss in detail about how the import system works, its components and what tasks they perform respectively.

Who should attend this talk?

This talk is most suitable for beginner to intermediate level Pythonistas.

How does the talk proceed

  • The talk starts with packages in Python, and how to organize Python files in order to make them distributable, and importable.
  • It proceeds to discuss about $PATH and $PYTHONPATH, and how they are used to enable system-wide imports, and how capable are Python's path based import handlers, if they let a programmer import from remote URIs, etc.
  • Different types of packages such as regular packages, namespace packages, frozen modules, etc. are discussed.
  • After that, some key components are introduced i.e sys.modules and how modules are refreshed and regulated in runtime, what finders and loaders are, and what 'import protocol' is, sys.meta_path, sys.path_hooks
  • The talk then goes in detail about finders, the default finders in Python, spec object, and its functioning.
  • After finders, loaders are discussed with sample code, explaining the steps involved from processing a spec, and creating, executing, and loading a module to sys.modules.
  • After finders and loaders, 'import hooks' are discussed i.e how Python allows injection of custom components to handle a pre-defined import statement.
  • This follows a demonstration of import hooks, implementation of a meta_path finder Class which protects importing of modules from an http server with a token exchange. (So the source code is protected, and it can be logged who requested the source code by a signature to act against misuse.)
  • The demonstration is followed by listing some important use cases of import hooks, and how folks have used it in the past.

What can someone get out of this talk?

  • An understanding of the import system's mechanics, which helps to quickly debug and fix path/import related errors.
  • Knowledge of import hooks, to build custom import related plugins.

Outline

Time Duration - Topic

0 - 4 Introduction to packages and how to organize packages

4 - 7 Types of packages in python, and the default ones the import system supports.

7 - 12 Components of import system - sys.modules - finders and loaders - chronology of import related tasks

12 - 15 Finders, examples of finders, and how they function, find_spec function

15 - 17 sys.meta_path, sys.path_hooks in detail

17 - 19 Introduction to PEP 302, and import hooks

19 - 22 explain loaders with template code

23 - 25 code walkthrough and demonstration of an implementation of import hooks

25 - 26 briefly cover 'importing of submodules' topic with an example

26 - 28 present use-cases of import hooks, and usage in industry, conclude.

28 - 30 Q&A

Prerequisites:

  • Syntactic knowledge of Python
  • Should have come across import/$PYTHONPATH related errors

If you don't know either and still like the talk idea, please follow this gist to 'intentionally' come across the errors :)

Video URL:

youtube link

Speaker Info:

Shivashis Padhi is a senior, majoring Computer Science and Engineering at National Institute of Technology, Tiruchirappalli, India.

With ~3 years of experience as a student software developer he has worked with multiple organizations(Python Software Foundation - GSoC'20, Delta Force), large and small scale startups(Grofers, Flytbase Labs, Gmetri), he lives by a simple policy, 'learn and build stuff to make the world a better place'. He's currently an intern with the Data Engineering team at Grofers building geospatial data-management tools. He spends some time contributing to MSS with an amazing set of people.

He also gave his first ever technical talk, about maps and Python during PyCon India'19.

He has written more than 15000 SLOCs of Python, and after being frustrated 100s of times when there were import/path related errors, he decided to read the official docs on import. Going through the docs saved him countless hours of debugging errors related to the import system.

During PyCon India'20, he wishes to share his learnings and findings on this topic with fellow Python practitioners.

Speaker Links:

homepage

blog

github / twitter

Section: Core Python
Type: Talks
Target Audience: Beginner
Last Updated: