Jupyterhub - serving python notebooks at all scales. Why we at HackerEarth went for this to provide notebooks as a code editor for our users?
Padamja Gupta (~padamja53) |
We at HackerEarth, have 4M+ community of developers, and our B2B product helps recruiters hire the best technical talent. The boom in ML and AI has raised a lot of requests for the notebook as a code editor just like Kaggle so that developers feel comfortable during assessments. With such dynamic demand and huge scale managing, jupyter notebooks would have been a difficult task but thanks to JupyterHub, it became quite easy. I would like to share why JupyterHub, a multi-user Hub that spawns, manages, and proxies multiple instances of the single-user Jupyter notebook server can be used in a class of students, a corporate data science group, or a scientific research group.
To give a little more context towards the problems solved by this open-source project, consider
- a professor or a trainer who has to invest time to get his attendees onboarded with environment setup, or
- developers in most organizations using notebooks for exploring huge datasets are restricted to the computing resources of their local machines, or
- the case we had in hand where we wanted to spawn isolated multiple notebooks on demand
All of these issues are beautifully addressed by a single initiative, JupyterHub. We feel more and more people should know about this solution.
Outline of the talk
- Introduction - 5 mins - What is Jupyterhub? For complete beginners, will also add a little bit on what are jupyter notebooks
- The infrastructure of Jupyterhub - 15 mins
- Infra for Jupyter notebooks for a small group of people(1-100) - 5 mins
- Kubernetes Infra for Jupyter notebooks for a dynamic much larger pool of people - 10 mins
- Customization in Jupyterhub - 10 mins
- Custom authentication as well as Spawners - 7 mins
- Custom database and custom storage - 3 mins
- A demo of how we integrated this into our product. - 5 mins
Familiarity with Jupyter notebooks and a basic understanding of how a web application work.
Padamja is building features at HackerEarth and has two years of industry experience. Python is her mother tongue and she wants to contribute towards more and more developer-friendly initiatives.