Downloaded 14 times
Uploaded byAlluxio, Inc.
The Future of Computing is Distributed
The document discusses the evolution and current challenges of distributed computing, highlighting the increasing demand driven by deep learning and the limitations posed by the end of Moore's Law. It emphasizes that the shift towards AI-centric applications necessitates a move to distributed systems for scalability and integration with existing workloads. The document also presents Ray as a universal framework for distributed computing, promoting its ecosystem of libraries and tools for scalable machine learning and AI applications.
The Future of Computing is Distributed
- 1. Distributed computing is distributed IonStoica December 4, 2020
- 2.
- 3. Distributed computing stillthe exception… Inaccessible to most developers Few universities teach distributed computing This will change… Distributed computing will be the norm, rather than the exception
- 4. What is differentthis time? The rise of deep learning (DL) The end of Moore’s Law Apps becoming AI centric
- 5. (https://openai.com/blog/ai-and-compute//) DL demands growingfaster than ever 35x every 18 months
- 6. (https://openai.com/blog/ai-and-compute//) Not only esotericapps ResNet 50 ResNeXt 101 im age processing translation
- 7. GPT-3 (175B) Turing Proj.(17B) GPT-2 (8.3 B) GPT-1 (1.5B) BERT GPT-1 Transformer ResNet-50 20x every 18 months Not only processing, but memory
- 8. GPT-3 (175B) Turing Proj.(17B) GPT-2 (8.3 B) GPT-1 (1.5B) BERT GPT-1 Transformer ResNet-50 200x every 18 m onths! Not only processing, but memory
- 9. What is different? Therise of deep learning (DL) The end of Moore’s Law Apps becoming AI centric
- 10. The end ofMoore’s Law From 2x every 18 months to 1.05x every 18 months.
- 11. (https://openai.com/blog/ai-and-compute//) Hardware cannot keepup 35x every 18 months CPU
- 12. (https://openai.com/blog/ai-and-compute//) Hardware cannot keepup 35x every 18 months Moore’s Law (2x every 18 months) CPU
- 13. What about specializedhardware? Trade generality for performance Not enough! Just extending Moore’s Law
- 14. (https://openai.com/blog/ai-and-compute//) Specialized hardware notenough 35x every 18 months Moore’s Law (2x every 18 months) CPU GPU TPU
- 15. Turing Proj. (17B) GPT-2(8.3 B) GPT-1 (1.5B) BERT GPT-1 Transformer ResNet-50 20x every 18 months (https://devblogs.nvidia.com/training-bert-with-gpus/) Memory dwarfed by demand GPU memory increased by just 1.45x every 18 months No way out but distributed!
- 16. What is different? Therise of deep learning (DL) The end of Moore’s Law Apps becoming AI centric
- 17. Apps becoming AIcentric... ... and integrating other distributed workloads
- 18. HPC Big Data Micro- services Before: Isolateddistributed workloads
- 19. HPC AI Big Data Micro- services AI integrateswith other dist. workloads
- 20. HPC AI Training, Simulations Big Data Micro- services Distributedtraining using MPI Leverage industrial simulators for RL AI integrates with other dist. workloads
- 21. HPC AI Training, Simulations Micro- services Big Data Log processing, Featurization Online learning,RL AI integrates with other dist. workloads
- 22. HPC AI Training, Simulations Big Data Log processing, Featurization Micro- services Serving, Business logic Inference, online learning,web backends AI integrates with other dist. workloads
- 23. (https://devblogs.nvidia.com/training-bert-with-gpus/)Distributed apps becomingthe norm No way to scale AI, but going distributed HPC AI Training, Simulations Big Data Log processing, Featurization Micro- services Serving, Business logic Apps becoming AI centric (and integrating with other distributed workloads)
- 24. No general solutionfor e2e AI apps DL Big Data MicroservicesHPC ?
- 25. Natural solution... Stitch togetherexisting frameworks Distributed systems Model Serving Training Distributed systems Distributed systems Hyperparam. Tuning Distributed systems Data processing Simulation Distributed systems Business logic Distributed systems
- 26. RAY Universal framework fordistributed computing
- 27. Model Serving Training Hyperparam. Tuning Data processing Simulation Business Logic Libraries Universal framework fordistributed computing (Python and Java) *
- 28.
- 29. Three key ideas Executeremotely functions as tasks, and instantiate remotely classes as actors ○ Support both stateful and stateless computations Asynchronous execution using futures ○ Enable parallelism Distributed (immutable) object store ○ Efficient communication (send arguments by reference)
- 30. def read_array(file): # readndarray “a” # from “file” return a def add(a, b): return np.add(a, b) a = read_array(file1) b = read_array(file2) sum = add(a, b) Function class Counter(object): def __init__(self): self.value = 0 def inc(self): self.value += 1 return self.value c = Counter() c.inc() c.inc() Class
- 31. @ray.remote def read_array(file): # readndarray “a” # from “file” return a @ray.remote def add(a, b): return np.add(a, b) a = read_array(file1) b = read_array(file2) sum = add(a, b) @ray.remote class Counter(object): def __init__(self): self.value = 0 def inc(self): self.value += 1 return self.value c = Counter() c.inc() c.inc() Function → Task Class → Actor
- 32. @ray.remote def read_array(file): # readndarray “a” # from “file” return a @ray.remote def add(a, b): return np.add(a, b) id1 = read_array.remote(file1) id2 = read_array.remote(file2) id = add.remote(id1, id2) sum = ray.get(id) @ray.remote(num_gpus=1) class Counter(object): def __init__(self): self.value = 0 def inc(self): self.value += 1 return self.value c = Counter.remote() id4 = c.inc.remote() id5 = c.inc.remote() Function → Task Class → Actor
- 33. @ray.remote def read_array(file): # readndarray “a” # from “file” return a @ray.remote def add(a, b): return np.add(a, b) id1 = read_array.remote(file1) id2 = read_array.remote(file2) id = add.remote(id1, id2) sum = ray.get(id) file1 file2 Node 1 Node 2 • Blue variables are Object IDs • Similar to futures read_array id1 Return id1 (future) immediately, before read_array() finishes Task API
- 34. @ray.remote def read_array(file): # readndarray “a” # from “file” return a @ray.remote def add(a, b): return np.add(a, b) id1 = read_array.remote(file1) id2 = read_array.remote(file2) id = add.remote(id1, id2) sum = ray.get(id) file1 file2 Node 1 Node 2 read_array id1 read_array id2 Dynamic task graph: build at runtime • Blue variables are Object IDs • Similar to futures Task API
- 35. @ray.remote def read_array(file): # readndarray “a” # from “file” return a @ray.remote def add(a, b): return np.add(a, b) id1 = read_array.remote(file1) id2 = read_array.remote(file2) id = add.remote(id1, id2) sum = ray.get(id) • Blue variables are Object IDs • Similar to futures file1 file2 Node 1 Node 2 read_array id1 read_array id2 add id Node 3 Every task scheduled, but not finished yet Task API
- 36. @ray.remote def read_array(file): # readndarray “a” # from “file” return a @ray.remote def add(a, b): return np.add(a, b) id1 = read_array.remote(file1) id2 = read_array.remote(file2) id = add.remote(id1, id2) sum = ray.get(id) • Blue variables are Object IDs • Similar to futures file1 file2 Node 1 Node 2 read_array id1 read_array id2 add id Node 3 ray.get() block until result available Task API
- 37. @ray.remote def read_array(file): # readndarray “a” # from “file” return a @ray.remote def add(a, b): return np.add(a, b) id1 = read_array.remote(file1) id2 = read_array.remote(file2) id = add.remote(id1, id2) sum = ray.get(id) • Blue variables are Object IDs • Similar to futures file1 Node 1 Node 2 Node 3 read_array file2 read_array add sumTask graph executed to compute sum Task API
- 38.
- 39. universal framework for distributedcomputing Native Libraries Most popular scalable RL library ● PyTorch and TF support ● Largest # of algorithms Best Distributed Library Ecosystem
- 40. universal framework for distributedcomputing Native Libraries A popular hyperparameter tuning library: “For me, and I say this as a Hyperopt maintainer, Tune is the clear winner down the road. Tune is fairly well architected and it integrates with everything else, and it’s built on top of Ray so it has all the benefits stemming from that as well. … In 2020, I would certainly bet on Tune.” -- Max Pumperla, HyperOpt creator Best Distributed Library Ecosystem
- 41. universal framework for distributedcomputing Native Libraries Just launched. Promising start but a long way to go. Best Distributed Library Ecosystem
- 42. universal framework for distributedcomputing Native Libraries Best Distributed Library Ecosystem
- 43. universal framework for distributedcomputing Native Libraries 3rd Party Libraries Best Distributed Library Ecosystem ●Horovod: most popular distributed training library. ●Optuna and hyperopt: popular hyperparameter search libraries
- 44. universal framework for distributedcomputing Native Libraries 3rd Party Libraries The two most popular NLP libraries using Ray to scale up Best Distributed Library Ecosystem
- 45. universal framework for distributedcomputing Native Libraries 3rd Party Libraries ModelArts Best Distributed Library Ecosystem Major ML cloud platforms embedding Ray/RLlib/Tune
- 46. universal framework for distributedcomputing Native Libraries 3rd Party Libraries ModelArts Best Distributed Library Ecosystem Dask running on Ray ● Faster, more resilient, and more scalable
- 47. universal framework for distributedcomputing Native Libraries 3rd Party Libraries ModelArts Popular experiment tracking platform ● One-line integration with Ray Tune. Best Distributed Library Ecosystem
- 48. universal framework for distributedcomputing Native Libraries 3rd Party Libraries ModelArts Intel’s unified Data Analytics and AI platform. ● Integrated Ray together with Spark, TF, etc ● Use cases include streaming ML with customers such as Burger King and BMW. Best Distributed Library Ecosystem
- 49. universal framework for distributedcomputing Native Libraries 3rd Party Libraries ModelArts Best Distributed Library Ecosystem
- 50. universal framework for distributedcomputing Native Libraries 3rd Party Libraries ModelArts Your apps here! Best Distributed Library Ecosystem
- 51.
- 54. Refactored actor management Placement groups JavaAPI Direct calls between workers Azure support in cluster launcher Original dashboard Refactor worker to C++ Initial port to Bazel Significant community contributions
- 55. Summary Ray: universal frameworkfor distributed computing Comprehensive ecosystem of scalable libraries universal framework for distributed computing Native Libraries 3rd Party Libraries ModelArts Your apps here! https://github.com/ray-project/ray
- 56.