Edit - I tried repeating the full process this morning, and now I get an error as expected. See my answer below.
I also looked at %ALLUSERSPROFILE%\Application Data\Microsoft\Crypto\Keys and the permissions for the most recently modified private key "file" are as expected (i.e. no access for anyone).
Edit - I tried repeating the full process this morning, and now I get an error as expected. However, if I use yesterday's certificate thumprint I still get no error but they are set up the same. I've rebooted in case something was cached, but it made no difference.