SSH config for two daemons on same server

My server exposes two ssh ports: one for the server itself, and one for a git daemon (gitea).

My local ~/.ssh/config:

Host server hostname 1.2.3.4 port 22 user foo identityfile ~/.ssh/id_rsa_server Host gitea hostname 1.2.3.4 port 2222 user git identityfile ~/.ssh/id_rsa_gitea 

I can ssh into the server using $ ssh server.

But I cannot ssh or perform git operations using gitea - it returns public key errors. It seems to choose the wrong key, even though I specified it in the config file. I think it chooses the first in the list.

Workarounds:

• $ GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa_gitea -F /dev/null" git ...
• $ git config core.sshCommand "ssh -i ~/.ssh/id_rsa_gitea -F /dev/null"; git ...

But I always forget those settings, and they don't work well with automation (I need to remember to set it for every repo, manually).

I prefer to fix the ~/.ssh/config file, so it works as expected. How can I do that?

UPDATE:
The verbose ssh log includes this:

debug1: Reading configuration data ~/.ssh/config ... debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering public key: ~/.ssh/id_rsa_server RSA SHA256:... explicit agent debug1: Authentications that can continue: publickey debug1: No more authentication methods to try. [email protected]: Permission denied (publickey). 

So as I said above, it's only offering the first key, then fails. It's not offering the correct key (the second one in the config file).