[wip] ProgressBar with callable remove_when_done

[kenodegard]

Driven by the conversation in #974

Working to add the ability to provide a callable for remove_when_done which is run in the background and once complete, if the return value is True, then the counter in question is deleted.

Example use case:

from prompt_toolkit.shortcuts.progress_bar import ProgressBar import time import random def remove_when_done(counter): if counter.total % 2: time.sleep(2.5) return True with ProgressBar() as group: for label in "ABCDEFGH": for _ in group(range(random.randrange(5, 10)), label=label, remove_when_done=remove_when_done): time.sleep(.1)

@jonathanslenders I'm still trying to wrap my head around Python's async, I don't think I'm understanding tasks correctly so I haven't been able to get this concept to work yet. The above example generates the progress bars as expected and the background tasks appear to be created however the actual removals themselves are not occurring. I'm also getting a RuntimeError when the application attempts to finalize/cleanup:

RuntimeError: Task <Task pending name='Task-1' coro=<...> cb=[_run_until_complete_cb() at ...]> got Future <Task cancelling name='Task-12' coro=<ProgressBarCounter._remove_when_done_async() running at ...>> attached to a different loop 

I've been testing with 3.8.

[kenodegard]

Ah, so progress bars are juggling two event loops, the main loop (_loop) and the application loop (_app_loop). The _app_loop is only set as the event loop in the separate app thread. Since the counters exist in the main thread it's grabbing the wrong event loop.

A slight modification was introduced to Application.create_background_task to where the event loop to use can be passed in. Alternatively, I see that Application is storing what appears to be its event loop:

python-prompt-toolkit/prompt_toolkit/application/application.py

Lines 613 to 618 in 7d2ef19

	async def _run_async() -> _AppResult:
	" Coroutine. "
	loop = get_event_loop()
	f = loop.create_future()
	self.future = f # XXX: make sure to set this before calling '_redraw'.
	self.loop = loop

So should we be using self.loop instead? Of passing loop or using get_event_loop?

[jonathanslenders]

In the case of the progress bars, there should be only one event loop running, which is the one in the UI thread. get_event_loop() should never be called in the main thread. If that happens, then that's a bug.
I'll have a look.

[jonathanslenders]

Two things:

• I don't really understand why remove_when_done would have to run in an executor? It should not be hard to compute, so it can run in the event loop.
• I think this can be a prompt_toolkit.filters.FilterOrBool. (Use to_filter or is_true, like in other places.)

[kenodegard]

This allows for more creative removals. I'm personally interested in removal policies like keeping the last X counters and keeping each counter for Y minutes.

[kenodegard]

Pinging this - just don't want to lose steam on this one. @jonathanslenders, let me know if there is anything I can change or further clarify.

[jonathanslenders]

I have a few change requests here.

But, I'm still wondering too why we need a callback for this. Could you please share some example code for how this would be used? I think it would be good to have examples in the examples/ directory anyway.

[jonathanslenders]

This doesn't correspond with the type signature, I think it's fine to pass self, but the Callable in __init__ should match.

[kenodegard]

hm I think I got this figured out

[jonathanslenders]

I'm still thinking about whether this is the right thing to do.

By default, I'd like as much as possible for prompt_toolkit to have asynchronous APIs, and possibly create an synchronous abstraction on top of that. If we expect a blocking synchronous function here, then we exclude people that would like to include it in an async app.

[kenodegard]

I borrowed almost all of this code from how validation and ThreadedValidators are implemented. I do see that the current implementation appears to have some issue with cleaning up tasks quickly upon completion (you'll see this if you run the a-lot-of-parallel-tasks.py example that I updated).

Would it be better to allow remove_when_done to be either synchronous or asynchronous and then it's up to the user to design any expensive async functions correctly (with asyncio.sleep, etc.)?

[jonathanslenders]

This is nice. That's a great use case.

I'm still not convinced though that a thread is the right approach here. We will get many threads that never finish and can't be joined anywhere. I'm thinking about whether we could reuse the prompt_toolkit filters in this case too, and do something like:

@Condition def ten_seconds_passed() -> bool: counter = get_counter() # context local return time.time() - counter.stop_time > 10 @Condition def at_least_five_bars() -> bool: pb = get_progress_bar() # context local return len(pb.counters) > 10 for i in pb(range(total), label=label, remove_when_done=ten_seconds_passed & at_least_five_bars):

If this condition evaluates during every render, we can remove the bar when this turns False.

[kenodegard]

closing this in favor of #1042