Downloaded 19 times
![12 Send message http://docs.celeryproject.org/en/latest/userguide/calling.html kick.apply_async([‘neuron_x’]) kick.apply_async([‘neuron_x’], routing_key = ‘neuron_h1’)](https://image.slidesharecdn.com/buildingdistributedsystemwithceleryondockerswarm-pyconjp2016-160925085320/75/Building-Distributed-System-with-Celery-on-Docker-Swarm-PyCon-JP-2016-12-2048.jpg)
![33 Deploy parallel computing • Deploy containers over Docker Swarm – deploy containers • docker-compose up –d • docker-compose scale celery= n (for example) • Send task messages to worker containers results = [ doSomething.apply_async([data]) for data in dataset ] • But, Docker Swarm needs root password-less SSH](https://image.slidesharecdn.com/buildingdistributedsystemwithceleryondockerswarm-pyconjp2016-160925085320/75/Building-Distributed-System-with-Celery-on-Docker-Swarm-PyCon-JP-2016-33-2048.jpg)
![36 Summary • Easy to use – doSomething.apply_async ([xxx], routing_key = ‘host_NY’) – getData.apply_async(routing_key = ‘host_NY’) .get() • Adventages of a distributed system – Decoupling, DI/IC (dependency injection / inversion of control ) – Distributed = Shared – Load-Balancing • Caution: – See ref: Distributed systems theory for the distributed systems engineer • Take a look of Canvas in Celery – Chain – Group](https://image.slidesharecdn.com/buildingdistributedsystemwithceleryondockerswarm-pyconjp2016-160925085320/75/Building-Distributed-System-with-Celery-on-Docker-Swarm-PyCon-JP-2016-36-2048.jpg)
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
- 1. Building Distributed System withCelery on Docker Swarm Wei Lin PyCon JP 2016 2016/09/22
- 2. 2 About Me • WeiLin @Wei_1144 Wei1234c@gmail.com https://github.com/Wei1234c • Mostly worked in the fields of: – marketing – strategy-planning – project-management • Materials of this talk: https://github.com/Wei1234c/PyCon-JP-2016-talk
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11. Send message toa specific queue
- 12.
- 13.
- 14.
- 15.
- 16. Assign an uniqueworker for a specific queue
- 17. 17 Starting the Workerhttp://docs.celeryproject.org/en/latest/userguide/workers.html https://github.com/celery/celery/issues/3065 $celery -A IoT worker -n neuron_h1 -Q neuron_h1 --concurrency=1
- 18.
- 19.
- 20.
- 21.
- 22. 22 Celery worker containers asneurons h h1 h2 h3 kick.apply_async( routing_key = ‘neuron_h1’) getData.apply_async(routing_key = ‘neuron_h1’) .get()
- 23. 23 “Celery + Docker-Swarm” So? I think…
- 24.
- 25. 25 Think OO • “From”to Inherit • Dockerfile to Encapsulate • Docker Image as Class • Docker Container as Object
- 26. 26 Docker Swarm &Celery doSomething.apply_async (routing_key = ‘host_NY’) Tokyo Taipei New York getData.apply_async(routing_key = ‘host_NY’) .get()
- 27.
- 28. 28 OOAD Station C StationB Station A
- 29.
- 30.
- 31.
- 32. 32 How to doParallel Computing 1. Run and deploy Docker containers across Docker Swarm cluster, with a Celery worker process running in each container. 2. When initiate worker processes, don’t assign “Queue” parameter, so ALL the workers will listen to the default queue in broker. 3. Sending bunch of messages to broker without setting routing_key parameter, the messages will be routed to the default queue. 4. In a parallel fashion, all workers will pick up messages from the default queue and do the tasks.
- 33. 33 Deploy parallel computing •Deploy containers over Docker Swarm – deploy containers • docker-compose up –d • docker-compose scale celery= n (for example) • Send task messages to worker containers results = [ doSomething.apply_async([data]) for data in dataset ] • But, Docker Swarm needs root password-less SSH
- 34.
- 35. 35 One for all,All for one https://github.com/Wei1234c/OneForAll_AllForOne/blob/master/celery_projects/OneForAll_AllForOne.md • run a Docker image to join distributed parallel computing cluster, that’s all. docker run –d wei1234c/musketeer (for example)
- 36. 36 Summary • Easy touse – doSomething.apply_async ([xxx], routing_key = ‘host_NY’) – getData.apply_async(routing_key = ‘host_NY’) .get() • Adventages of a distributed system – Decoupling, DI/IC (dependency injection / inversion of control ) – Distributed = Shared – Load-Balancing • Caution: – See ref: Distributed systems theory for the distributed systems engineer • Take a look of Canvas in Celery – Chain – Group
- 37. 37 Source Code • Celeryon Docker Swarm – using “Word Count” as an example of parallel computing. – https://github.com/Wei1234c/CeleryOnDockerSwarm/tree/master /celery_projects • IoT as Brain – simulate an artificial neural-network of 6 neurons. – https://github.com/Wei1234c/IOTasBrain/tree/master/celery_proj ects • One for all, all for one – follow the SETI@home paradigm. – on each machine, running a single Docker container to join the cluster. – https://github.com/Wei1234c/OneForAll_AllForOne/tree/master/c elery_projects
- 38. 38 References References : • Distributedsystems theory for the distributed systems engineer – http://the-paper-trail.org/blog/distributed-systems-theory-for-the-distributed-systems- engineer/ • Celery user guide – http://docs.celeryproject.org/en/latest/userguide/ • Docker document – https://docs.docker.com/ • MQTT Message Type and Flows – http://blog.maxkit.com.tw/2014/02/mqttmessage-type-and-flows.html • Celery on Docker Swarm – https://github.com/Wei1234c/PyCon-JP-2016- talk/blob/master/Celery%20on%20Docker%20Swarm%20- %20PyCon%20JP%202016.md • IoT as Brain – https://github.com/Wei1234c/PyCon-JP-2016- talk/blob/master/IoT%20as%20Brain%20-%20PyCon%20JP%202016.md