win32 console debug

[stephenrauch]

Debugging applications on windows under a debugger/ide can be difficult
because the IDE may use its own 'console' which Prompt Toolkit is unable
to use productively. This PR makes it possible to set an environment
variable which allows Prompt Toolkit to create a console window and
attach the appropriate handles to allow the program to use the console
productively.

[jonathanslenders]

Hi @stephenrauch,
Thank you for this! It looks like you spend on lot of time on this patch.
I need a bit of time to read through all of the code before merging it, and probably I'll have some remarks.
But it sounds definitely useful!
Jonathan

[jonathanslenders]

Hi @stephenrauch,

I just review the code. In general it looks good. But I'd like a few changes. (Hopefully you don't mind.)

• Most importantly, importing win32_console_debug has a side effect. It executes _check_environment_variables. We should not execute any code during an import. It's better to call this function ourself. (See: https://chrismorgan.info/blog/say-no-to-import-side-effects-in-python.html )
• Further, I prefer to avoid introducing new environment variables. I would prefer that users who need this functionality, simple call a function. Something like create_win32_console(debug=False). If they like to expose this API through an environment variable, then they can still do it themself.

If you don't have the time to fix any of this, then I'll do it.
I just tested this change in my Windows machine with Pycharm, and it works wonderful!

Thanks,
Jonathan

[stephenrauch]

I will gladly make the changes. Give me a few days.

Cheers,

-- Stephen

[stephenrauch]

@jonathanslenders

So I have finally updated the pull request. I wanted to use it for a while to help make sure it was working ok. I am glad I did, as it needed a ^C handler. There are now no side effects of the import. I did however leave the Environment Variable as an option for invoking the functionality, for this reason: Editing the source code of my program between actually running it in the console, and running it under the debugger is rather a PITA. This way I can assign the environment variable in my debug environment, and it is largely a NOP when not debugging. The use of the environment variable is optional, as discussed in the docs, the functionality can be invoked without it.

Hopefully this meets with your desires. If not, or you need anything else, please LMK.

I have also added (a needed) ^C handler.

[jonathanslenders]

Thanks! This should become part of the next release.
Right now, I'm working on two things. prompt_toolkit=2.0 (which will come with many improvements) and the next release for the current version.

[pe224]

@stephenrauch
Could you give me a hint on how to get your console to work?
I invoke it by

from prompt_toolkit.terminal.win32_console_debug import ensure_console ensure_console() 

On the invoking side, I get

Found debug console pid 5468, pipe name: 'FIND_CONSOLE: ipy_repl.py'` 

A console window opens up which outputs

debug window pipe name: \\.\pipe\FIND_CONSOLE: ipy_repl.py Connected at 2017-07-02 22:24:27.539 Error in sys.excepthook: Original exception was: 

So there is a connection followed by an exception.
The exception seems to occur when h_pipe = win32pipe.ConnectNamedPipe(pipe) is called the 2nd time in the final while-loop of your stub program.
I'm on Python 3.6, 64bit.

[stephenrauch]

@pe224 So I am reading right that there is no exception information printed? Also, what windows version?

[pe224]

@stephenrauch
Correct.
It's Windows 7. I can reproduce the error with Windows 10, Python 3.5, 32bit as well.

I also found a minor bug: if the path to python.exe contains spaces, there's an error,
e.g. C:\Program Files\Python\...
The spaces need to be escaped before the start command is sent to Popen,
i.e. start cmd /k C:\"Program Files"\Python\...

EDIT, Some new insights:
If I start a python shell from cmd.exe and call win32_console_debug._attach_console() it works - in this case I cannot test ensure_console as I'm already in a console and hence a new one will not be created.
If I either try it from

• a python shell in sublimeREPL (a plugin for Sublime Text 3, it essentially opens python.exe via Popen and pipes stdin and stdout)
• from ipython (launched via cmd.exe)

the error mentioned occurs.

[jonathanslenders]

Hi @stephenrauch,

Excuse me for coming back just now.

Regarding this pull request, I'm not sure what to do. I still think it's definitely valuable, but I'm not sure I have the resources and knowledge to maintain this piece as part of prompt_toolkit while it is evolving. I also think that it's perfectly feasible to implement all of this on top of prompt_toolkit, like it is right now. What about creating a prompt_toolkit_win32_debugger package or something like that?
Probably this requires also a couple of changes in order to be compatible with the 2.0 release.

[MrCredible]

This pull request was never merged. Is there another solution for debugging while using an IDE?

[jonathanslenders]

@Spectrem12,

PyCharm has now a built-in console emulator. It's not enable by default, but you can do this in the settings:

I'm not sure, but VS Code probably has a similar option. Since Windows now has ConPTY (see https://devblogs.microsoft.com/commandline/windows-command-line-introducing-the-windows-pseudo-console-conpty/ ), any IDE can render a console.

So, I don't think there's a need for this anymore.

@stephenrauch : hope you're fine too if we close this?

[MrCredible]

I should have added more detail to my initial question.

I have the emulator option checked but it limits the debugging and output capabilities or maybe I just don't know about pycharm.

All the output is monochrome. The output buffer is limited. And you are not able to type inline python syntax within the emulator.

I appreciate any solutions or work around you may have to offer.

[jonathanslenders]

I'm not sure about the debugging capabilities, because something is broken in my PyCharm right now, but colors are definitely working fine (notice I run it from the PyCharm run button, it's not the "terminal" pane).

[MrCredible]

Yes, I am also using the run button.

Are those colors defined within the IDE itself or with an extension?

I am using Grep Console and it no longer functions using the emulated window (not exactly prompt-toolkit issue) , but it causes other setbacks such as being able to pause debugging and input python commands inline.
As well, as I am not able to have PyCharm open up my separate log files that have streams going to, where I am also using Grep Console highlight.

The terminal emulator is severely lacking in functionality.

Is there a workaround to not force the terminal emulator?