Description:

The tech talk is going to discuss Salt as a case study of an on going project at LinkedIn to evaluate Helix to manage clusters of Salt Masters to alleviate problems caused by a very lopsided salt master to salt minion ratio. The tech talk will use salt as a case study to demonstrate how high volume python applications can be written to be managed by Helix.

Apache Helix is a generic cluster management framework used for the automatic management of partitioned, replicated and distributed applications hosted on a cluster of nodes. Helix automates reassignment of resources in the face of node failure and recovery, cluster expansion, and reconfiguration. Apache Helix was originally developed at LinkedIn to manage some of our high volume applications

Saltstack's Salt is a remote execution and orchestration tool, counted in the company of other tools such as puppet, chef ansible etc. LinkedIn has one of the largest deployments of salt anywhere in the world.

Topics to be covered:

• Covering basics of cluster management and traditional cluster management solutions
• Introduction to Finite State Machines
• Introduction to Apache Helix and Zookeeper and related concepts
• Introduction to Python libraries used for Helix and Zookeeper
• Real Code examples to demonstrate implementation
• Potential use cases outside Salt

Prerequisites:

The talk assumes some familiarity or curiosity about high volume applications and general understanding of clustering, partitioning etc. Some reading of the following links might be helpful but is not necessary

Helix:

• Apache Helix Home
• Blog on development of Helix at LinkedIn
• Blog on use of Helix at LinkedIn
• A great Quora answer related to development of Helix
• Helix on linkedin@github

Salt:

• Salt Home
• Salt Documentation
• Salt at Github

Python Libraries:

• PyHelix on pypi
• Kazoo, the zookeeper library, on pypi

Content URLs:

Presentation URL: Google Docs

Content of this presentation is still evolving.

Speaker Info:

I am a Site Reliability Engineer at LinkedIn and I speak python along with a few dialects of Java. Unlike other engineers, I don't convert coffee to code but I do viciously attack with sarcasm.

• vim vs emacs: vim
• tmux for dear life
• MacOS(Yes that's the new name) vs Windows: Ubuntu
• Part time arm chair analyst of everything

You can find me on LinkedIn: Akhil Malik @ LinkedIn

Speaker Links:

• My Last year's talk in PyCon 2015 on Scaling Python @ LinkedIn where I talked about developing a python based torrent cluster to move huge amounts of data: Youtube