Fix asserting arithmetic on big-endian platforms (#2092)

This fixes a latent issue on big-endian platforms that was exposed by the merge of PR#2074, which does an `mlib_assert_mul(int32_t, ...)`. Broken: If the target type of a ckdint-asserting-arithmetic macro is smaller than `intmax_t`, then the return value of the function will be bogus on big-endian platforms, because we neglected to adjust the temporary bits used to store the result. Fix: If the platform is big-endian, we bit-shift the temporary result down by the number of bytes of different between `uintmax_t` and the destination type size.

Author: vector-of-bool

src/common/src/mlib/ckdint.h

@@ -629,6 +629,13 @@ static inline uintmax_t(_mlib_assert_ckdint)(size_t dst_sz,
  b_str);
  abort();
  }
+ if (!mlib_is_little_endian()) {
+ // We unconditionally set the leading bytes of `tmp`, but big-endian expects
+ // the lower place values to be in the later bytes. If the target int is
+ // smaller than intmax, we must shift all the bits over to their proper
+ // position. This expression is trivially constant-folded by an optimizer.
+ tmp >>= ((size_t)CHAR_BIT * ((sizeof tmp) - dst_sz));
+ }
  return tmp;
 }
 

src/common/tests/test-mlib.c

@@ -768,6 +768,13 @@ _test_ckdint_partial(void)
  mlib_check(mlib_mul(&a, 3, INTMAX_MAX));
  mlib_check(a, eq, INTMAX_MAX - 2);
  }
+
+ {
+ // Check asserting form with a dest type smaller than intmax, which triggers
+ // a special branch on big-endian systems.
+ int32_t const i = mlib_assert_mul(int32_t, -908, 1000);
+ mlib_check(i, eq, -908000);
+ }
 }
 
 static void