Before you've pushed anything, the only key that has any rights to the gitosis account is the one you passed to gitosis-init. That other keys have access to your regular user account is irrelevant. Compare the key in ~gitosis/.ssh/authorized_keys and ssh-add -L.
Gitosis works like this: on each push, a post-commitupdate hook runs and gitosis regenerates authorized_keys for its own account. If it didn't pick up your key, it most likely wasn't in the format gitosis expects. Since the hook is post-commitupdate the commitupdate will have been accepted even if it was invalid (gitosis could be more strict by having both a pre-commitan update and a post-commitupdate hook). Make a trivial change and push again from the host that has the initial key to look at gitosis error's messages. Adding the key to authorized_keys manually involves duplicating the correct line, replacing the key type, key data and username.