- Notifications
You must be signed in to change notification settings - Fork 326
Open
Labels
Description
Is your feature request related to a problem? Please describe.
Some operands, such as summation over a long series of chunks, can be started and run partially when some data are ready. This may effectively reduce the memory size during computing.
Describe the solution you'd like
Several changes are needed.
- Separate job submission and running in worker. This enables a job be executed multiple times.
- Partial-execution support should be added to .
- Jobs in workers are triggered given data arrival. When executed, these jobs are released from slots.
- Remove
OperandState.FREEand use methods like ref-counting to manage these data. - Reflect actual progress in Mars web for these operands.
Describe alternatives you've considered
Streaming-based implementation is also considered. The reason not to use this solution is that the streaming does not share the data storage model currently implemented in worker. It may be implemented later.
Additional context
N/A.