Jupyter for DevOps

prateekj1993


3

Votes

Description:

DevOps teams focus on standardizing development environments and automating delivery processes to improve delivery predictability, efficiency, security and maintainability. This includes control of the production environment, better understanding of the production infrastructure, and autonomy to support their own applications. With DevOps, nothing gets “thrown over the wall.”

This is how an application development and support flow looks like -

  1. A developer "Jim" creates an application
  2. "Jim" lists down (most commonly on a Wiki) the most common issues that could arise in the application, and notifies support/ops team about the same
  3. An issue comes in the application. Support engineer "Bob" tries to find the relevant steps to fix the issue
  4. "Bob" tries to execute the steps to fix the issue
  5. If the steps doesn't solve the issue, "Bob" contacts "Jim" and asks him for help

So, Wikis become the crux of the operational issue flow. We believe this current approach is problematic and inefficient in the following regards:

Verbose: Wikis are geared towards explaining the problem rather than solving it

Subjective: Since Wikis are in plain English, there may be differences in interpretations of the instructions based on perceptions

Too technical: The user needs to be familiar with various tools to debug the issue successfully

Error prone: The wiki content could have commands, which the user may need to modify to solve the issue; and hence this approach is error prone (typing, syntactical, copy-paste, configuration mistakes to name a few)

Monotonous: Every time an issue happens, the user needs to reread the Wiki to ensure that he's executing up-to-date steps

Resolution time: Resolution time is higher w.r.t. the time taken to resolve issue in an automated manner [HTML_REMOVED]

These challenges become even more pronounced for companies that have development team in one region, support team in other, and clients/users in another. This was true for Market Data team in AQR. So, the time zone difference meant that collaboration with Clients, Users and Support was a big challenge along with many others. We additionally wanted to adopt automation that would make sure the ops duties are integrated into the application development workflow and hence could ease operational workload. We turned to finding a tool that could help us achieve all this. And voila, we found out that the solution was something we were using all along, Jupyter notebooks -

Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and other rich media.

Jupyter ships with the IPython kernel, which allows you to write your programs in Python, but there are currently over 100 other kernels for languages like Java, Go, C#, Scala, Erland, R, Ruby, Haskell, Javascript. It is currently used by a lot of data scientists for things like data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

At first glance, it doesn't seem very obvious as to how Jupyter can help us here. But as we think a bit deeper, it makes all the sense in the world. We decided to use Jupyter Notebook to create our production support runbook as it is -

  • A one stop shop for interaction with multiple languages and tools
  • More developer friendly and powerful as executable code can be mixed with rich text
  • Can be made very intuitive even for a non technical user (with various widgets and other UI elements)
  • Provides ease of usage, reproducibility, and reuse (notebooks are portable if running in multiple environments, such as development, staging and production)
  • Always provides objective results, rather than results based on instructions interpretation
  • Promotes partnership of Developers with Support Team (just like partnership of QA with Developers in Agile)

In this talk, we will demonstrate how we used Jupyter as a framework that can significantly increase the operational efficiency in an organization and can be an integral part of an organization's DevOps flow.

Prerequisites:

We expect anyone attending this talk to have a basic understanding of Python and Jupyter Notebooks

Video URL:

https://www.dropbox.com/s/voy0yoimk2xy2tq/VID_20200729_162917.mp4?dl=0

Content URLs:

https://www.dropbox.com/s/dail65blh7q17t2/PyCon.slides.html?dl=0

(Please download the file and then view it since it is an HTML file)

Speaker Info:

Prateek Jain is a Software Engineer proficient in multiple languages with a soft spot for Python. He is currently working as an Associate at AQR Capital India, and has a total experience of 5 years in the FinTech domain. He is a passionate technology enthusiast and sees python as a way to solve complex problems in a rather easy manner

Section: Developer tools and automation
Type: Talks
Target Audience: Intermediate
Last Updated: