Infrastructure automation using open source python components
by Bhupendra Singh (speaking)
Objective
Linkedin has a large network infrastructure supporting applications serving millions of customers. Python and its large set of modules have played a big role in managing this fast growing infrastructure. This talk will give an insight into the automation system built to manage it.
Description
Linkedin is one of the fastest growing internet companies. In our
environment, each day brings a new challenge. This provides a unique opportunity where one the one hand, the day-job demands keeping the 'lights on' by attending to infrastructure issues and then spend the rest of the day, thinking of the next bottleneck in the infrastructure that needs to be scaled.
With Python and its vast resource of libraries around open source
utilities like celery, redis, flask, queuing and multiprocessing solutions, we have developed a platform that:
1. Manages configuration of a heterogeneous infrastructure.
2. Keeps track of a constantly changing configuration.
3. Automatically updates the system monitoring this elastic infrastructure.
4. Provides a data-analysis platform that identifies operational issues
close to real-time and provides capacity models for architects to scale
this infrastructure.
5. Visualization tools that aggregates millions of data-points, identify anomolies and generate highly customized UI's which has enabled operations to continually reduce MTTD and MTTR.
This has helped a lean infrastructure team to keep up with its
exponential growth.
Speaker bio
Pradeep Hodigere has been working in the infrastructure management space, at companies with massive infrastructure, building systems that manage the application, systems and network tiers. Currently, he is leading network infrastructure automation at linkedin using python and related open source components.
1
▼
Need to document more on the flow with some example as how the data would be collected and displayed.
1
▼
Can you give us more info on what tools you are planning to cover ?
1
▼
We are going to cover internally developed custom tools for monitoring and configuring network devices.
1
▼
You've indicated that these are open source. Please provide links to the internally developed tools.
1
▼
Internally developed tools are not open sourced yet as they are very specific to internal processes. These tools, however, have been implemented in python using open source tools and libraries. This talk can be seen as a case study in automating critical infrastructure operations in a large organization.