Distributed Computing in Docker using Python and Flask: DockerComp

Archit Sharma (~arcolife)


3

Votes

Description:

The Pitch

During a recent docker hackathon, I had pitched in a proposal of using docker for doing distributed computing, for the purpose of solving challenging problems that require volunteer computing or crowdsourced efforts. This prototype still needs to go out there and be put to use, since I could only imagine the wonderful use cases (refer to README file on github repo link)

TODO

We would start by doing a basic demo of how to use this, then brainstorming features and application areas and fixing some bugs, then adding a few daemons to manage the workloads and clients. If time permits, add security layers, better networking, a management dashboard, task queue implementation, possible integration with orchestration platforms and such.

Would appreciate efforts from the community to contribute to the same.

Cheers

Prerequisites:

  • Good Python concepts
  • API development in Python-Flask
  • Basic usage of docker (building, running, stopping and interacting with a container).
  • Have Linux OS pre-installed. (docker is not supported on Windows currently). Welcome to add support for OSX.

Optional:

  • Shell scripting
  • Would appreciate if you also have good networking concepts and/or could quickly develop workloads for sample data analysis tasks.

Content URLs:

  • Github: http://github.com/arcolife/dockerComp/
  • Youtube Presentation: https://www.youtube.com/watch?v=lIp2nrOnKFs
  • Wiki: https://github.com/arcolife/dockerComp/wiki

Speaker Info:

Data/Web/Performance Engineer by day; Guitarist by evening; Scholar by night; Socrates by dawn

Speaker Links:

  • Work - http://work.arcolife.in/
  • Blog - http://blog.arcolife.in/
  • Twitter - http://twitter.com/arcolife
  • Github - http://github.com/arcolife/

Section: Others
Type: Dev Sprint
Target Audience: Intermediate
Last Updated: