Introduction to PEP 3118 - The Python Buffer Protocol





In use cases involving manipulation of large amounts of data, like scientific computing applications, copying data across libraries or different objects, for example from a numpy array to a pytorch array can be expensive. In these cases, it will be better to share a pointer to the underlying memory instead of explicitly copying the data. To this end, Python needs a mechanism to share a view of the underlying data across the libraries and at the same time, capture the complexities of the data, like data type, shape and dimensions. Here comes the Python buffer protocol. The talk delves into the need for buffer protocol, the mechanism provided by PEP 3118 for sharing buffers and discusses how it is implemented in practice.

Outline of the talk:

  • Buffers and Arrays (3 min)
  • Motivation for the buffer protocol (3 min).
  • Discussion on PEP-3118 (3 min)
  • Technical details about buffer protocol implementation (7 min).
  • Code demo on using the buffer protocol (10 min). (total: ~25 min)

Talk takeaways

The key takeaway for the audience is the introduction of Python C-API to beginner audiences in a simple easy-to-follow demo and for intermediate/advanced users, it introduces the buffer protocol. The Python C-API provides rich and powerful functionalities which can be leveraged in applications like scientific computation and to access the python interpreter. Apart from it, the other takeaways are:

  • Understand the difference between a buffer and an array.
  • Get to know the technical details of the buffer protocol.
  • Get hands-on experience with the buffer protocol via a code-demo.


Familiarity with C programming language.

Video URL:

Content URLs:

Speaker Info:

Arun is a machine learning software engineer working on building web-based machine learning platform for computational drug discovery at Deep Forest Sciences. He is a contributor to the open-source library DeepChem, a scientific python library for applying deep learning in molecular data. Before Deep Forest, he worked as a project assistant at RISE Lab, Indian Institute of Technology, Madras for about a year where his work focused on detecting malwares from network traffic using predictive algorithms. Arun is also an active volunteer at FOSSUnited, a non-profit aimed at strengthening the open-source software ecosystem in India by helping the team organise the monthly FOSSUnited meetups at Bangalore. His technical writing can be found in his webpage and his code at github.

Speaker Links:

Section: Core Python
Type: Talk
Target Audience: Intermediate
Last Updated: