+1 -1 +19
Vote on this proposal

Simple Linux Cluster With Python and Beanstalkd

by Rachee Singh (speaking)

Section
Scientific Computing
Session type
Tutorial
Technical level
Intermediate

Objective

To construct a simple Linux cluster which is reasonably fault tolerant for executing parallel algorithms across multiple nodes.

  • Introduction to distributed message queues
  • Setting up Beanstalkd
  • Setting up the cluster with the help of Beanstalkd
  • Testing the cluster [with 2 or more nodes].
  • Demonstration of computation of Pi across nodes and the associated speedup
  • Making the cluster fault tolerant
  • Running parallel matrix multiplication on the cluster

Description

Update

Tutorial participants will need to have the following tools running on their systems:

  1. Python 2.7.x
  2. GCC, or any other reliable C compiler.
  3. Git, verify the ability to clone Github repositories.
  4. pip, installed globally instead of in a virtualenv.

This talk discusses the details of setting up one’s own Beowulf cluster using Python. It presents an alternative to the traditional MPI APIs based approach to setting up a cluster. It is amazing how simple and elegant this alternative is! This simple cluster will then be used as a breadboard for parallel computing algorithms.

Setting up a cluster is a cumbersome task if one intends to use MPI/OpenMPI and their many variants . The issues range from setting up ssh between nodes to the unfriendly nature of C/C++. This is where Python’s ease-of-use can be leveraged.

Speaker bio

Rachee Singh uses Python for everything ranging from configuring DNS servers to testing networking protocols. More details about her work: blog and Github

Emaad Manzoor hacks on distributed computing, language processing and computer vision, hoping to engineer the Oracle someday. He currently builds prescient systems for the trend detection platform at Yahoo!. More on LinkedIn.

Comments


  • 1

    [-] Anand B Pillai 910 days ago

    Hi Rachee, this sounds better suited for a tutorial.


  • 1

    [-] Rachee Singh 910 days ago

    Hi Anand, Heeding to your suggestion, changed it to a tutorial session. Thanks.

Login with Twitter or Google to leave a comment →