Deploying AirBnB’s Knowledge Repo at scale
Abhirath Batra (~abhirath54) |
Knowledge Repo, an open-source knowledge sharing platform by Airbnb, has rapidly become a tool of choice for Data Scientists to share analyses and insights while ensuring reproducibility, a key foundation of any scientific research. It allows for analyses done in Jupyter Notebooks and R-Markdowns to be curated in an immutable, presentable, searchable and shareable manner allowing seamless consumption of insights and knowledge.
The problem :
However, Knowledge Repo has been made to work comfortably for internal teams, with a selected set of “repositories” loaded on deployment. It would also need an internal server admin of sorts to manage the server.
What we did : The poster will talk about our fork of Knowledge Repo which is modified to :
- Use Database backend to remove the need to use KRs(Knowledge Repositories)
- Add KRs to a deployed server on the fly.
- Add posts to KRs directly
- Attach this as a microservice to a platform(case in point : Polly[HTML_REMOVED]TM[HTML_REMOVED])
How: It will also talk about how these features were implemented by:
- Leveraging the impeccable Object Oriented design of KR
- Modifying Database object and connection cycles
- URL blueprinting to allow reverse proxy
- Jupyter notebook
- Object Oriented Programming
- Some web development
I'm Abhirath Batra, working at Elucidata as a Bioinformatics Engineer. I've been interested in using computing to solve problems in science and have worked on molecular simulations, ML and data to solve questions in biological systems