Distributed Computing in Docker using Python and Flask: DockerComp
Archit Sharma (~arcolife) |
During a recent docker hackathon, I had pitched in a proposal of using docker for doing distributed computing, for the purpose of solving challenging problems that require volunteer computing or crowdsourced efforts. This prototype still needs to go out there and be put to use, since I could only imagine the wonderful use cases (refer to README file on github repo link)
We would start by doing a basic demo of how to use this, then brainstorming features and application areas and fixing some bugs, then adding a few daemons to manage the workloads and clients. If time permits, add security layers, better networking, a management dashboard, task queue implementation, possible integration with orchestration platforms and such.
Would appreciate efforts from the community to contribute to the same.
- Good Python concepts
- API development in Python-Flask
- Basic usage of docker (building, running, stopping and interacting with a container).
- Have Linux OS pre-installed. (docker is not supported on Windows currently). Welcome to add support for OSX.
- Shell scripting
- Would appreciate if you also have good networking concepts and/or could quickly develop workloads for sample data analysis tasks.
- Github: http://github.com/arcolife/dockerComp/
- Youtube Presentation: https://www.youtube.com/watch?v=lIp2nrOnKFs
- Wiki: https://github.com/arcolife/dockerComp/wiki
Data/Web/Performance Engineer by day; Guitarist by evening; Scholar by night; Socrates by dawn
- Work - http://work.arcolife.in/
- Blog - http://blog.arcolife.in/
- Twitter - http://twitter.com/arcolife
- Github - http://github.com/arcolife/