Security Evaluation

RIPE testbed

We evaluated all approaches against the RIPE security testbed.¹ RIPE is a synthesized C program that tries to attack itself in a number of ways, by overflowing a buffer allocated on the stack, heap, or in data or BSS segments. RIPE can imitate up to 850 attacks, including shellcode, return-into-libc, and return-oriented programming.

To evaluate security of approaches, we disabled all other security features:

• Linux ASLR was disabled via sudo bash -c 'echo 0 > /proc/sys/kernel/randomize_va_space'
• All compiler optimizations were disabled via -O0
• Compiler-inserted stack canaries were disabled via -fno-stack-protector
• Compiler-enforced fortify-source was disabled via -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0
• Executable stack was enabled via compiler flag -Wl,-z,execstack

Even under these relaxed security flags all compilers were susceptible only to a small number of attacks. Under native GCC, only 64 attacks survived, under ICC – 34, and under Clang – 38.

Results

Surprisingly, a default GCC-MPX version showed very poor results, with 41 attacks (or 64% of all possible attacks) succeeding. As it turned out, the default GCC-MPX flags are sub-optimal. First, we found a bug in the memcpy wrapper which forced bounds registers to be nullified, so the bounds checks on memcpy were rendered useless.² This bug disappears if BNDPRESERVE is manually set to one. Second, the MPX pass in GCC does not narrow bounds for the first field of a struct by default, in contrast to ICC which is more strict. To catch intra-object overflows happening in the first field of structs one needs to pass the -fchkp-first-field-has-own-bounds flag to GCC. When we enabled these two flags, all attacks were prevented; all next rows in the table were tested with these flags.

Other results are expected. MPX versions without narrowing of bounds overlook 14 intra-object overflow attacks, where a vulnerable buffer and a victim object live in the same struct. The same attacks are overlooked by AddressSanitizer, SoftBound, and SAFECode. Interestingly, AddressSanitizer has 12 working attacks, i.e., two attacks less than other approaches. Though we did not inspect this in detail, AddressSanitizer was able to prevent two shellcode intra-object attacks on the heap.

We performed the same experiment with only-writes versions of these approaches, and the results were exactly the same. This is explained by the fact that RIPE constructs only control-flow hijacking attacks and not information leaks (which could escape only-writes protection).

Up to table of contents

RIPE Logs

Below are the logs which show which attacks worked under each approach.

• Native versions:
  • GCC
  • ICC
  • Clang
• MPX versions:
  • GCC default
  • GCC
  • GCC only-writes
  • GCC no narrow bounds
  • GCC no narrow bounds only-writes
  • ICC
  • ICC only-writes
  • ICC no narrow bounds
  • ICC no narrow bounds only-writes
• AddressSanitizer versions:
  • full
  • only-writes
• SoftBound
• SAFECode

Up to table of contents

Bugs in Benchmark Suites

During our experiments, we found 6 real out-of-bounds bugs (true positives). Five of these bugs were already known, and one was detected by GCC-MPX and was not previously reported.

The bugs found are:

1. incorrect black-and-white input pictures leading to classic buffer overflow in ferret (PARSEC);
2. wrong preincrement statement leading to classic off-by-one bug in h264ref (SPEC);
3. out-of-bounds write in perlbench (SPEC);
4. benign intra-object buffer overwrite in x264 (PARSEC);
5. benign intra-object buffer overread in h264ref (SPEC);
6. intra-object buffer overwrite in perlbench (SPEC).

A more refined summary table as well as descriptions of the aforementioned bugs can be found in the Usability page.

Up to table of contents