[libc][GSoC 2024] Half precision in LLVM libc

Community GSoC
1

Description: Half precision is an IEEE 754 floating point format that has been widely used recently, especially in machine learning and AI. It has been standardized as _Float16 in the latest C23 standard, bringing its support to the same level as float or double data types. The goal for this project is to implement C23 half precision math functions in the LLVM libc library.

Expected Results:

  • Setup the generated headers properly so that the type and the functions can be used with various compilers (+versions) and architectures.
  • Implement generic basic math operations supporting half precision data types that work on supported architectures: x86_64, arm (32 + 64), risc-v (32 + 64), and GPUs.
  • Implement specializations using compiler builtins or special hardware instructions to improve their performance whenever possible.
  • If time permits, we can start investigating higher math functions for half precision.

Project Size: Large

Requirement: Basic C & C++ skills + Interest in knowing / learning more about the delicacy of floating point formats.

Difficulty: Easy/Medium

Confirm Mentor: Tue Ly, Joseph Huber

6 Likes
2

@lntue This project sounds great!

I have good familiarity with both C and C++, are there any open issues in libc that you could point me to that are related? It would be really nice to gain some familiarity with libc in the process. In the meantime, I will brush up my knowledge on floating points, so that I can work on any related issues.

Thanks!

3

Hi @lntue,

This is Shijin. I am a Ph.D. candidate in Computer Engineering, in my fourth year at Northeastern University. I’m greatly interested in efficient ML, including the computation acceleration on hardware. The half-precision on ML has already shown great efficiency in practice. In my very recent project (which uses half-precision for a secured neural network), I observed that the half-precision model will not induce an observable accuracy drop compared to its full-precision counterpart.

To my understanding, this project call is about generic half-precision math implementation on various architectures. Given reliability and generality are expected, the performance (for sure :D) and the scalability towards more complex math functions should also be the sweet point. I believe this opportunity will demonstrate a promising implementation to me, on the current half-precision computing paradigm.

For the requirement, I have familiarity with C/C++ and the knowledge of floating-point, based on my coursework before and my teaching assistance experience in undergraduate courses (about embedded system design and CPU realization). My knowledge of LLVM is still basic and general, yet I can quickly wrap up the required coding specifications on LLVM libc.

I am enthusiastic about this project and looking forward to contributing. Please feel free to reply with any questions or consideration. Thanks!

4

I would certainly be interested in contributing to this. I have familiarity with C/C++ programming and I’m interested in deepening my knowledge of floating point implementations within LLVM.

5

@lntue @jhuber6 I have read all of What Every Computer Scientist Should Know About Floating Point Arithmetic and parts of Elementary Functions: Algorithms and Implementation thoroughly, and now I have a fair understanding of how floating point arithmetic is performed and how such numbers are stored in memory.

I was reading the code in libc and looking for the actual implementation of the floating arithmetic support wanted to confirm if my thoughts are in the right direction.

If I understand correctly, the code that is implemented to support Half Precision floating points is to be placed in libc/src/__support/FPUtil/ by extending existing files, for internal use of the compiler?

After that has been done, we must modify some of the existing header files, specifically math.h to extend the functions to support half precision floating values.

All the architecture specific code is added in the respective folders correspondingly named, located in the same __support/ directory, and similarly for math functions in the math directory.

Please correct me if you find me in error anywhere, and I would really appreciate some pointers and advice on how I can proceed and what else I should be careful/mindful about.

Thanks a lot!

6

@vinayakdev @Sean_D @nasherm1 : Thanks for your interest in the project!

Let me add some more details to the stages I mentioned in the expected results:

  1. Setup the generated headers properly so that the type and the functions can be used with various compilers (+versions) and architectures:

  2. Implement generic basic math operations supporting half precision data types that work on supported architectures: x86_64, arm (32 + 64), risc-v (32 + 64), and GPUs.

  3. Implement specializations using compiler builtins or special hardware instructions to improve their performance whenever possible.

    • This step will require some investigation, instruction / function selection, and performance benchmark.
    • Hopefully the other project working on revamping performance testing in LLVM libc can be used at this point.

  4. Investigating higher math functions for half precision:

    • This is to me is actually the most exciting part, and I expect most of the functions will be much simpler than their single precision counter part in src/math/generic.
    • One great thing about half precision functions is that we can really able to test everything exhaustively, even the FMA function.

In addition to the 2 references that @vinayakdev mentioned, I would like to mention some of the following sources if you want to look further:

  • Handbook of Floating-Point Arithmetic by J.-M. Muller et. al.
  • Symposium on Computer Arithmetic has many great papers on floating point arithmetic and related fields.

Let me know if you have any question.

3 Likes
7

@lntue Thanks for replying!
I read through the resources that you mentioned, specifically Handbook of Floating-Point Arithmetic, and I found it to go into a fair amount of detail regarding the implementation of floating points, specifically in software.

Reading through the commit

I found that _Float128 is currently not supported by Clang, but it turns out that _Float16 is for a good number of targets (Clang Language Extensions — Clang 19.0.0git documentation).

So, if I understand correctly, what actually is to be done is that we have to check for the macros that compilers define for their half-precision data types(or alternatively check for the number of digits in the significand/mantissa) and assign the types accordingly, in possibly a file named float16.h.

Rest of the headers are generated through adding the required build support to the respective CMakeLists.txt where required, and adding the necessary entries to the API TableGen files.

This task, if no unforeseeable difficulties arise, should be manageable.

In the next task, we would have to implement basic math functions for the Float16 type, support for which is to some extent available in src/__support/FPUtil/ folder. This would be where most of the knowledge gained from the references can be applied, specifically in adding support for rounding functions, which might be required for the implementation of others.
Again, we would need to define the operations separately and add them to the build process as we did for the first stage.

Using hardware instructions to implement specialisations could actually be the trickiest and most time consuming part, as performance and accuracy are both required. However, the last part of implementing higher math functions seems to be the most interesting! It would be really a fun process to add support for binary16 format to them, and then testing for the correct output.

If everything that I wrote above seems good, can I then start working on the proposal and share a first draft here? It would be really great to hear your suggestions and advice, to create as good a proposal as I possibly can!

Thanks a lot!

8

I’m replying to show my interest in contributing to this project. If I list down my mostly related experiences to this project:

  • Using LLVM to build a procedural language frontend (toy language) - GitHub - Amila-Rukshan/winzigc-compiler: The WinZigC programming language
  • I have learned IEEE 754 single and double precision floats under a university module called numerical analysis.
  • I have gained C++ experience by contributing to the envoy proxy project.
  • I have recently read floating point representation in Donald Knuth’s MMIX machine architecture.

I will also check the more related information given in the reply above.

9

I’m Dat Le, a Computer Science student at the University of Florida. I have just recently found out about GSOC 2024 and I am reaching out with keen interest in the LLVM libc project focused on implementing C23 half precision math functions. Despite being relatively new to LLVM, my solid background in C++ and a strong desire to dive into the nuances of floating-point formats fuel my enthusiasm for this project.

I’m particularly drawn to the goals of enhancing cross-compatibility, developing basic and specialized operations for half precision types, and the prospect of advancing into higher math functions. These areas not only align with my academic interests but also present a valuable opportunity for hands-on learning and contributing to meaningful advancements in AI and machine learning applications.
If anyone has any suggestions or guidances please let me know down there

PS: I do know that it is really really late in the cycle already so any help would be appreciated