Skip to content

Some operands can be submitted and run when some inputs are ready #118

@wjsi

Description

@wjsi

Is your feature request related to a problem? Please describe.
Some operands, such as summation over a long series of chunks, can be started and run partially when some data are ready. This may effectively reduce the memory size during computing.

Describe the solution you'd like
Several changes are needed.

  • Separate job submission and running in worker. This enables a job be executed multiple times.
  • Partial-execution support should be added to .
  • Jobs in workers are triggered given data arrival. When executed, these jobs are released from slots.
  • Remove OperandState.FREE and use methods like ref-counting to manage these data.
  • Reflect actual progress in Mars web for these operands.

Describe alternatives you've considered
Streaming-based implementation is also considered. The reason not to use this solution is that the streaming does not share the data storage model currently implemented in worker. It may be implemented later.

Additional context
N/A.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions