Jupyterhub - serving python notebooks at all scales. Why we at HackerEarth went for this to provide notebooks as a code editor for our users?
Padamja Gupta (~padamja53) |
Description:
We at HackerEarth, have 4M+ community of developers, and our B2B product helps recruiters hire the best technical talent. The boom in ML and AI has raised a lot of requests for the notebook as a code editor just like Kaggle so that developers feel comfortable during assessments. With such dynamic demand and huge scale managing, jupyter notebooks would have been a difficult task but thanks to JupyterHub, it became quite easy. I would like to share why JupyterHub, a multi-user Hub that spawns, manages, and proxies multiple instances of the single-user Jupyter notebook server can be used in a class of students, a corporate data science group, or a scientific research group.
To give a little more context towards the problems solved by this open-source project, consider
- a professor or a trainer who has to invest time to get his attendees onboarded with environment setup, or
- developers in most organizations using notebooks for exploring huge datasets are restricted to the computing resources of their local machines, or
- the case we had in hand where we wanted to spawn isolated multiple notebooks on demand
All of these issues are beautifully addressed by a single initiative, JupyterHub. We feel more and more people should know about this solution.
Outline of the talk
- Introduction - 5 mins - What is Jupyterhub? For complete beginners, will also add a little bit on what are jupyter notebooks
- The infrastructure of Jupyterhub - 15 mins
- Infra for Jupyter notebooks for a small group of people(1-100) - 5 mins
- Kubernetes Infra for Jupyter notebooks for a dynamic much larger pool of people - 10 mins
- Customization in Jupyterhub - 10 mins
- Custom authentication as well as Spawners - 7 mins
- Custom database and custom storage - 3 mins
- A demo of how we integrated this into our product. - 5 mins
Prerequisites:
Familiarity with Jupyter notebooks and a basic understanding of how a web application work.
Video URL:
https://youtu.be/ttw3_wVcN88
Speaker Info:
Padamja is building features at HackerEarth and has two years of industry experience. Python is her mother tongue and she wants to contribute towards more and more developer-friendly initiatives.