- Notifications
You must be signed in to change notification settings - Fork 326
Closed
Description
Is your feature request related to a problem? Please describe.
Currently there are no shuffle support in Mars, therefore a lot of functions cannot be added, such as fancy indexing.
Describe the solution you'd like
Add shuffle support for Mars. To reduce potential burden on graph processing, a proxy node is added and edge count is reduced from a * b to a + b.
Shuffle output operands inherit a base class called ShuffleMap and input operands ShuffleReduce. ShuffleMap exports data parts into separate parts by keys. ShuffleReduce accepts data parts by key and calculate combinations.
Local execution, scheduler and worker needs updates to accept such kind of graph and data organization.
Shuffle will first be applied on irregular reshapes.
