Automating testbed experiments, data collection and visualization with Python

Sunit Kumar Nandi (~sunitknandi)


16

Votes

Description:

During my M.Tech. programme at IIT Guwahati, I observed that researchers in both industry and academia work with testbeds, both real and virtual, for making advancements in Computer Science, whether it is in algorithms, networking protocols or data science by running exhaustive experiments. I realised that Python, being a very versatile language, can be used to do everything related to experimental research and analysis, without requiring the usage of any other scripting language. Based on my experiences, I am presenting a talk to explain how to build automated testbed experiments, data collection and analysis with Python and a few libraries, avoiding big and bulky frameworks as much as possible.

My talk is structured as follows:

  1. Building a testbed for computing and networking experiments
  2. Using Python and paramiko to provison entities (PCs, smartphones, Raspberry Pis, routers, switches, etc.) in the testbed
  3. Running tests on the entities with subprocess and paramiko
  4. Collecting log files and other trace data from testbed entities
  5. Parsing log files and trace information to collect statistics with basic text processing and regex and storing them in appropriate Python data structures like lists, tuples and dictionaries for easy access
  6. Analysing collected statistics with Python math and generating reports
  7. Visualizing graphs from statistics with python-gnuplot or matplotlib

I hope that after attending my talk, you will be able to automate your testbed experiments to the extent of spending less time on experimentation and data collection and more time on actual research and publishing papers.

Prerequisites:

There are minimal prerequisities for my talk. You need to have knowledge of the following:

  1. Basic algorithms and data structures
  2. Computer networks, especially IP addressing
  3. Python basics

Also you need to have:

  • The willingness to learn and experiment

Content URLs:

Contents related to the talk will be added later.

Speaker Info:

Hello everyone! I am Sunit Kumar Nandi, a Trainee Teacher at National Institute of Technology, Arunachal Pradesh. I have completed my M.Tech. at IIT Guwahati this year and am also enrolling for Ph.D.

I am deeply interested in computer networking, telecommunications, operating systems and distributed systems design. I use Python for most of my daily work involving a great deal of experimentation. Apart from that I contribute to SuperX OS, a Linux distribution with KDE, based out of Assam, India. I love BSD and Linux based systems and have been involved with them since my childhood. As a result, I have had 14 years of experience with managing Linux servers, networking equipment and designing automated systems in the simplest way possible. In my free time, I spend my efforts running Techno FAQ, an e-magazine for science, technology, education and business.

Speaker Links:

You can follow me on:

My open source contributions:

Packages I maintain for Arch Linux:

Other projects I run:

Section: Developer tools and Automation
Type: Talks
Target Audience: Intermediate
Last Updated: