[ruby/openssl] Fix modular square root test with LibreSSL >= 3.8
If x is a modular square root of a (mod p) then so is (p - x). Both answers are valid. In particular, both 2 and 3 are valid square roots of 4 (mod 5). Do not assume that a particular square root is chosen by the algorithm. Indeed, the algorithm in OpenSSL and LibreSSL <= 3.7 returns a non-deterministic answer in many cases. LibreSSL 3.8 and later will always return the smaller of the two possible answers. This breaks the current test case.
Instead of checking for a particular square root, check that the square of the claimed square root is the given value. This is always true. Add the simplest test case where the answer is indeed non-deterministic.
[ruby/openssl] Fix modular square root test with LibreSSL >= 3.8
If x is a modular square root of a (mod p) then so is (p - x). Both
answers are valid. In particular, both 2 and 3 are valid square roots
of 4 (mod 5). Do not assume that a particular square root is chosen by
the algorithm. Indeed, the algorithm in OpenSSL and LibreSSL <= 3.7
returns a non-deterministic answer in many cases. LibreSSL 3.8 and
later will always return the smaller of the two possible answers. This
breaks the current test case.
Instead of checking for a particular square root, check that the square
of the claimed square root is the given value. This is always true. Add
the simplest test case where the answer is indeed non-deterministic.
https://github.com/ruby/openssl/commit/93548ae959