If you want password authentication via SSH and authorization via Gitolite, you can easily do this.
Basically what you do is, you create an account for each user and put them in a group, or you apply the following Match to all users and make exceptions for ones that should be able to actually log in. For my example I assume you stuff all git users into the group git-users, the hostname is git, Gitolite is operating as user git and Gitolite is installed at /opt/gitolite. IF anything is not true, just adapt the instructions
In /etc/ssh/sshd_config you would have
Match Group git-users X11Forwarding no AllowTcpForwarding no AllowAgentForwarding no PermitTTY no ForceCommand sudo -u git -H /opt/gitolite/src/gitolite-shell $USER
to give control to Gitolite if a user connects and to tell it which user it is.
And that this sudo command works without password you do visudo and there in the sudoers file add
Defaults!/opt/gitolite/src/gitolite-shell env_keep+=SSH_CONNECTION Defaults!/opt/gitolite/src/gitolite-shell env_keep+=SSH_ORIGINAL_COMMAND
to forward SSH_CONNECTION and SSH_ORIGINAL_COMMAND environment variables which Gitolite needs to operate properly and
%git-users git = (git) NOPASSWD: /opt/gitolite/src/gitolite-shell
which says in order that every user in the group git-users is allowed to execute on the host named git as user git without specifying a password the command /opt/gitolite/src/gitolite-shell.