Jupyterhub - serving python notebooks at all scales. Why we at HackerEarth went for this to provide notebooks as a code editor for our users?

Padamja Gupta (~padamja53)


6

Votes

Description:

We at HackerEarth, have 4M+ community of developers, and our B2B product helps recruiters hire the best technical talent. The boom in ML and AI has raised a lot of requests for the notebook as a code editor just like Kaggle so that developers feel comfortable during assessments. With such dynamic demand and huge scale managing, jupyter notebooks would have been a difficult task but thanks to JupyterHub, it became quite easy. I would like to share why JupyterHub, a multi-user Hub that spawns, manages, and proxies multiple instances of the single-user Jupyter notebook server can be used in a class of students, a corporate data science group, or a scientific research group.

To give a little more context towards the problems solved by this open-source project, consider

  • a professor or a trainer who has to invest time to get his attendees onboarded with environment setup, or
  • developers in most organizations using notebooks for exploring huge datasets are restricted to the computing resources of their local machines, or
  • the case we had in hand where we wanted to spawn isolated multiple notebooks on demand

All of these issues are beautifully addressed by a single initiative, JupyterHub. We feel more and more people should know about this solution.

Outline of the talk

  • Introduction - 5 mins - What is Jupyterhub? For complete beginners, will also add a little bit on what are jupyter notebooks
  • The infrastructure of Jupyterhub - 15 mins
    • Infra for Jupyter notebooks for a small group of people(1-100) - 5 mins
    • Kubernetes Infra for Jupyter notebooks for a dynamic much larger pool of people - 10 mins
  • Customization in Jupyterhub - 10 mins
    • Custom authentication as well as Spawners - 7 mins
    • Custom database and custom storage - 3 mins
  • A demo of how we integrated this into our product. - 5 mins

Prerequisites:

Familiarity with Jupyter notebooks and a basic understanding of how a web application work.

Video URL:

https://youtu.be/ttw3_wVcN88

Speaker Info:

Padamja is building features at HackerEarth and has two years of industry experience. Python is her mother tongue and she wants to contribute towards more and more developer-friendly initiatives.

Speaker Links:

LinkedIn

Github

Twitter

Section: Developer tools and automation
Type: Talks
Target Audience: Intermediate
Last Updated: