Skip to content

Add shuffle support #134

@wjsi

Description

@wjsi

Is your feature request related to a problem? Please describe.
Currently there are no shuffle support in Mars, therefore a lot of functions cannot be added, such as fancy indexing.

Describe the solution you'd like
Add shuffle support for Mars. To reduce potential burden on graph processing, a proxy node is added and edge count is reduced from a * b to a + b.

image

Shuffle output operands inherit a base class called ShuffleMap and input operands ShuffleReduce. ShuffleMap exports data parts into separate parts by keys. ShuffleReduce accepts data parts by key and calculate combinations.

Local execution, scheduler and worker needs updates to accept such kind of graph and data organization.

Shuffle will first be applied on irregular reshapes.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions