This is extremely common in the field of cryptography. It was popular to design key exchange systems in the 1980s without proof of security, just giving intuitive arguments which were often quite convincing. It turns out a lot of these systems became broken later.

Distributed systems are difficult to have good intuition about because they are complicated objects. However, we have a lot of (wrong) intuition around them because they are physical systems and we think we understand them. Hence, it is easy to make mistakes. There are many cases where systems work contrary to intuition.

One of my favourite examples is bitcoin. The original paper gives an argument about why the system is secure and why the honest strategy is a Nash equilibrium, by illustrating that one particular attack doesn't work in the favour of an adversary. This is more than just a hand-wavy argument: It is the full calculation of a particular attack, analytically and with numbers. Furthermore, that particular attack seems the most reasonable to do.

However, it was later shown that there are strategies which are better than the honest strategy -- the "selfish mining" attacks. The latter paper describes an alternative attack, quite more complicated, than what the original author envisioned. The attack is contrary to intuition. The bitcoin protocol was later fully analyzed on the backbone paper and it was proven secure for any adversarial strategy. However, the selfish mining strategy remains a better strategy than the honest strategy (and in fact the backbone paper shows a tight bound on this classes of attacks), so bitcoin is not incentive-compatible.

Another example around bitcoin-related conjectures is that, as time goes by and coinbase rewards are decreased, fees will make up for the incentives to continue running the blockchain. This notion is so ingrained in the bitcoin community that the bitcoin wiki literally states that "In the future, as the number of new bitcoins miners are allowed to create in each block dwindles, the fees will make up a much more important percentage of mining income." This conjecture has been shown to be false.

This is extremely common in the field of cryptography. It was popular to design key exchange systems in the 1980s without proof of security, just giving intuitive arguments which were often quite convincing. It turns out a lot of these systems became broken later.

Distributed systems are difficult to have good intuition about because they are complicated objects. However, we have a lot of (wrong) intuition around them because they are physical systems and we think we understand them. Hence, it is easy to make mistakes. There are many cases where systems work contrary to intuition.

One of my favourite examples is bitcoin. The original paper gives an argument about why the system is secure and why the honest strategy is a Nash equilibrium, by illustrating that one particular attack doesn't work in favour of an adversary. This is more than just a hand-wavy argument: It is the full calculation of a particular attack, analytically and with numbers. Furthermore, that particular attack seems the most reasonable thing to do.

However, it was later shown that there are strategies which are better than the honest strategy -- the "selfish mining" attacks. The latter paper describes an alternative attack, quite more complicated, than what the original author envisioned. The attack is contrary to intuition. The bitcoin protocol was later fully analyzed on the backbone paper and it was proven secure for any adversarial strategy. However, the selfish mining strategy remains a better strategy than the honest strategy (and in fact the backbone paper shows a tight bound on this class of attacks), so bitcoin is not incentive-compatible.

Another example around bitcoin-related conjectures is that, as time goes by and coinbase rewards are decreased, fees will make up for the incentives to continue running the blockchain. This notion is so ingrained in the bitcoin community that the bitcoin wiki literally states that "In the future, as the number of new bitcoins miners are allowed to create in each block dwindles, the fees will make up a much more important percentage of mining income." This conjecture has been shown to be false.