Simple Linux Cluster With Python and Beanstalkd
by Rachee Singh (speaking)
- Scientific Computing
- Session type
- Technical level
To construct a simple Linux cluster which is reasonably fault tolerant for executing parallel algorithms across multiple nodes.
- Introduction to distributed message queues
- Setting up Beanstalkd
- Setting up the cluster with the help of Beanstalkd
- Testing the cluster [with 2 or more nodes].
- Demonstration of computation of Pi across nodes and the associated speedup
- Making the cluster fault tolerant
- Running parallel matrix multiplication on the cluster
Tutorial participants will need to have the following tools running on their systems:
- Python 2.7.x
- GCC, or any other reliable C compiler.
- Git, verify the ability to clone Github repositories.
- pip, installed globally instead of in a virtualenv.
This talk discusses the details of setting up one’s own Beowulf cluster using Python. It presents an alternative to the traditional MPI APIs based approach to setting up a cluster. It is amazing how simple and elegant this alternative is! This simple cluster will then be used as a breadboard for parallel computing algorithms.
Setting up a cluster is a cumbersome task if one intends to use MPI/OpenMPI and their many variants . The issues range from setting up ssh between nodes to the unfriendly nature of C/C++. This is where Python’s ease-of-use can be leveraged.
Emaad Manzoor hacks on distributed computing, language processing and computer vision, hoping to engineer the Oracle someday. He currently builds prescient systems for the trend detection platform at Yahoo!. More on LinkedIn.