Your best option is likely to revoke access to that certificate and set up individual server users. You can also use a firewall to limit access from specific IPs.
 I have a small tutorial on SSH/SFTP setup for Amazon Linux on my website you may find useful. They key parts are.
 Firewall / Security Groups
 On AWS you can use Security Groups or Network Access Control Lists to specify what IPs can connect on what protocols. You can set this up so only your corporate IP can connect on SSH. This is very quick and easy to do. See instructions here.
 I suggest using security groups rather than NACLs, just because I find it easier. You could use either or both.
 Revoke Existing Key Pair
 To revoke the key pair (documentation, under "Adding or Replacing a Key Pair for Your Instance")
  - Retrieve the public key from your new key pair. For more information, see Retrieving the Public Key for Your Key Pair on Linux or Retrieving the Public Key for Your Key Pair on Windows. 
- Connect to your instance using your existing private key file. 
- Using a text editor of your choice, open the .ssh/authorized_keys file on the instance. Paste the public key information from your new key pair underneath the existing public key information. Save the file. 
- Disconnect from your instance, and test that you can connect to your instance using the new private key file. 
- (Optional) If you're replacing an existing key pair, connect to your instance and delete the public key information for the original key pair from the .ssh/authorized_keys file. 
Create the new users
 sudo su sudo useradd fred passwd fred 
 Create the keys
 su fred ssh-keygen -f rsa mkdir .ssh touch .ssh/authorized_keys chmod go-w ~/ chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys cat fred.pub >> /home/fred/.ssh/authorized_keys 
 Allow user login
 vi /etc/ssh/sshd_config PasswordAuthentication no AllowUsers ec2-user fred