I believe this is because of the say the security group resource works. I've resolved this same issue in several modules (both CP and my own). If the SG resource defines an ingress rule, it "changes" every apply. Also, this makes it so that you can't add additional ingress rules from another module because when this one runs it will remove them.

Moving the ingress rule to a security_group_rule resouce resolves these issues.