Port RTL/MJIT to Windows MINGW.

[larskanis]

Thank you for working on ruby's performance! I really like the way you wrote the code. As a proof of concept I did a Windows port of your invention. See the commit message for description of all single changes.

I tested with gcc on x86_64 only. RTL works per miniruby.exe and per ruby.exe. MJIT doesn't work per miniruby, because miniruby is static linked, but the MJIT binaries are linked to the external ruby DLL - not the miniruby-internal symbols. However MJIT works per ruby.exe, since it's linked to the one and the same ruby DLL. It passes the make check equally to the non RTL/MJIT version.

The compiler currently outputs a lot of warnings on Windows. I didn't try to resolve these for now, because there is a fundamental issue to be solved first: "long" is 32 bit only on Windows. The long type is used a lot in the RTL and MJIT code for instance as offsets to pointers. Interestingly enough unsigned long is used as positive and negative offset per typedef lindex_t. But adding a 32 bit negative offset to a 64 bit pointer is very different to adding 64 offset to 64 pointer.

I think there are two alternatives:

1. Use a 64 bit type as lindex_t: This is what I used in the attached pull request. This has the advantage, that all arithmetic operations with this type are the same on all supported platforms. However there seems to be no cross platform printf placeholder for 64 bit integers other then PRIdVALUE and co.

2. Use platform specific long size as lindex_t: This is how lindex_t was defined before my patch and how "long" is used in conjunction with FIXNUM in MRI. printf() can be used with simple %l in this case. The downside is, that it limits possible sizes of offsets, although there is no need to do so at runtime. It also makes arithmetic operations platform specific, so that additional type casts are required.

When the "long" type issue is solved, it makes sense to fix the compiler warnings.

Another portability issue is the use of pthread. While it is available on Windows MINGW, it is not on Visual-C. Possibly this can be solved by using ruby's cross platform rb_thread* functions. Since pthread works flawlessly on MINGW, I didn't change anything for now.

The minimizer task of rb_mjit_header is awfully slow. This is due to the slow process starts on Windows. This task runs almost 10 times slower compared to Linux. It's even worse with activated virus scanning. The algorithm is quite efficient, but since each call to gcc yields to only one single bit of result, it requires roughly 2000 such calls. I think the longer term solution could be a pure ruby minimizer, instead of relying on the gcc parser.

[vnmakarov]

As a proof of concept I did a Windows port of your invention. See the commit message for description of all single changes.

Lars, thank you for doing this. Windows port is at the end of my list but your work could help me to port RTL/MJIT earlier than I planned. Currently I am focused on exploring an alternative way of RTL code generation but I'll try to read your code and try it on Windows in a couple weeks.

I tested with gcc on x86_64 only. RTL works per miniruby.exe and per ruby.exe. MJIT doesn't work per miniruby, because miniruby is static linked, but the MJIT binaries are linked to the external ruby DLL - not the miniruby-internal symbols. However MJIT works per ruby.exe, since it's linked to the one and the same ruby DLL. It passes the make check equally to the non RTL/MJIT version.

Testing a JIT is a challenge especially asynchronous one. I still have no a good approach to it. Probably you tested only a part of test code (mostly testsuite infrastructure code) because MJIT has no time to generate code which executed only once (most of tests) and I know that currently MJIT has a lot of bugs which can not be easily fixed.

The compiler currently outputs a lot of warnings on Windows. I didn't try to resolve these for now, because there is a fundamental issue to be solved first: "long" is 32 bit only on Windows. The long type is used a lot in the RTL and MJIT code for instance as offsets to pointers. Interestingly enough unsigned long is used as positive and negative offset per typedef lindex_t. But adding a 32 bit negative offset to a 64 bit pointer is very different to adding 64 offset to 64 pointer.

I think there are two alternatives:

Use a 64 bit type as lindex_t: This is what I used in the attached pull request. This has the advantage, that all arithmetic operations with this type are the same on all supported platforms. However there seems to be no cross platform printf placeholder for 64 bit integers other then PRIdVALUE and co. Use platform specific long size as lindex_t: This is how lindex_t was defined before my patch and how "long" is used in conjunction with FIXNUM in MRI. printf() can be used with simple %l in this case. The downside is, that it limits possible sizes of offsets, although there is no need to do so at runtime. It also makes arithmetic operations platform specific, so that additional type casts are required. 

When the "long" type issue is solved, it makes sense to fix the compiler warnings.

Using existing lindex_t for negative was a quick hack. I am going to fix this. I was focusing to 64-bit platforms because it is the most widely used case these days.

Another portability issue is the use of pthread. While it is available on Windows MINGW, it is not on Visual-C. Possibly this can be solved by using ruby's cross platform rb_thread* functions. Since pthread works flawlessly on MINGW, I didn't change anything for now.

It is on my list too. Eric Wong proposed the same couple months ago.

The minimizer task of rb_mjit_header is awfully slow. This is due to the slow process starts on Windows. This task runs almost 10 times slower compared to Linux. It's even worse with activated virus scanning. The algorithm is quite efficient, but since each call to gcc yields to only one single bit of result, it requires roughly 2000 such calls. I think the longer term solution could be a pure ruby minimizer, instead of relying on the gcc parser.

Yes, I know. People told me about this a few times. It is also irritating for me when minimizing starts. I am going to work on this issue too.

Thank you for working on ruby's performance! I really like the way you wrote the code. As a proof of concept I did a Windows port of your invention. See the commit message for description of all single changes.

I'll definitely look at your code and incorporate your code (or part of it) into repository. I need to set up a Windows machine for testing and porting MJIT (VM is not good for performance work).

Lars, thank you very much again. Your comments are very helpful. Your patch just shows that Windows port is doable and I should pay more attention to it.

[larskanis]

This has mostly been merged to MRI as part of the initial MJIT patch.