How Python helped us to automate workflows for Data migration

Harshita Sharma (~harshita-sharma011)


Description:

Data is the new age currency. Keeping your data and workloads safe through backups is important, more so in a Kubernetes environment where service interrupting events are inevitable.

Performing backups and migration manually is time-consuming and is not reliable.

It is important to have an automated Disaster recovery plan to decrease the risks and downtime.

Also, having a workflow that makes it easy to migrate your kubernetes workload where your old cluster setup is not in a healthy state or you want to create a fresh setup on a different cloud provider/ on-premise.

Harshita and Satyam created an automation module that is highly-configurable and reliable. We use ansible for our setups and pytest to write various test scenarios to make sure the whole backup and migration process works as expected and to check data integrity after each restore. Using pytest we can leverage its rich plugin architecture and libraries such as pytest-ansible-playbook, pytest-ansible. This has made backing up and migration of kubernetes workloads easier than ever.

In this talk, we will explain how we used Kubernetes a Container Orchestration platform, Open Source cloud-native storage solution OpenEBS as our underlying storage solution and Heptio’s disaster recovery tool Velero for backup and restore of our workloads with automated framework written in Python, followed by a quick demo of how things work.

We will be leveraging Restic which uses a clever deduplication function for data.

Basic outline of the talk:

  1. Why an automation module for backup/restore of kubernetes workloads is the need of the hour? [2 minutes]
  2. Why heptio velero is a reliable, fast, secure option to backup/restore your workloads? [3 minutes]
  3. How we are using python to effectively automate the workflow? [3 minutes]
  4. A quick demo to show the end to end migration of workload from one cluster to another using velero with restic [5-7 minutes]
  5. Q/A - [2 minutes]

Prerequisites:

A beginner’s knowledge of ,Python, Ansible, Pytest, Kubernetes.

Speaker Info:

Speaker: Harshita Sharma

Harshita is computer science and engineering graduate , batch of 2014-2018. She is working as a software engineer at MayaData Inc. as a part of the E2E team. She is a tech enthusiast and enjoy learning new technologies. She loves to use Python to automate the manual tasks.

Co-Speaker: Satyam Zode

Satyam is an open source enthusiast and works as a Senior Software Engineer at MayaData Inc which is a company behind the leading open source cloud native storage project OpenEBS. Satyam spends his time on writing Go and work with Open source projects like OpenEBS, Kubernetes and Weave Scope. He is a maintainer of Weave Scope project. He is an occasional speaker at Kubernetes Bangalore, Kubernetes Chennai meetup, CNCF Bangalore Meetup, OpenEBS meetup, RootConf 2019. He is Certified Kubernetes Application Developer.

Speaker Links:

Harshita Sharma:

  • Github: https://github.com/harshita-sharma011/
  • Twitter: https://twitter.com/Harshit81162445
  • LinkedIn: https://www.linkedin.com/in/harshita-sharma1729/

Satyam Zode:

  • Github: https://github.com/satyamz
  • Twitter: https://twitter.com/SatyamZode
  • LinkedIn: https://in.linkedin.com/in/satyamz

Past speaking experience:

  1. Bangalore Kubernetes meetup:

    • https://www.meetup.com/kubernetes-openshift-India-Meetup/events/250069202/
    • Delivered hands on session: Kubernetes 101
  2. Chennai Kubernetes meetup

    • https://www.meetup.com/Chennai-Kubernetes/events/249792060/
  3. RootConf 2019:

    • https://hasgeek.com/rootconf/2019/proposals/birds-of-feather-bof-session-on-taking-your-home-b-WpAFkm6P7pmBWi76viYFw3

Section: Developer tools and automation
Type: Talks
Target Audience: Intermediate
Last Updated: