Unable to ssh to Google Compute Engine

Question

I used to log in to the instance through GCE web UI by clicking "Open in web browser".

Recently I try to log in the instance using the same way, but the window just keep showing "connecting" and didn't do anything.

I try ssh from google cloud shell. And what I get is:

USERNAME@cloudshell:~ (voltaic-phalanx-786)$ gcloud compute ssh --zone "asia-east1-c" "newforum" --project "voltaic-phalanx-786" --ssh-flag="-vvv" OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d 10 Sep 2019 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug2: resolve_canonicalize: hostname X.X.X.X is address debug2: ssh_connect_direct debug1: Connecting to X.X.X.X [X.X.X.X] port 22. debug1: Connection established. debug1: identity file /home/USERNAME/.ssh/google_compute_engine type 0 debug1: identity file /home/USERNAME/.ssh/google_compute_engine-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to X.X.X.X:22 as 'USERNAME' debug1: using hostkeyalias: compute.xxx debug3: hostkeys_foreach: reading file "/home/USERNAME/.ssh/google_compute_known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/USERNAME/.ssh/google_compute_known_hosts:1 debug3: load_hostkeys: loaded 1 keys from compute.xxx debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-grou p14-sha256,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected] ,[email protected],[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,[email protected],zlib debug2: compression stoc: none,[email protected],zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-grou p14-sha256,diffie-hellman-group14-sha1 debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,[email protected] debug2: compression stoc: none,[email protected] debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none [email protected]: Permission denied (publickey). ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. 

I try restart the instance. The last few lines of serial port output is:

Apr 6 14:14:45 newforum systemd[1]: Starting Google Compute Engine Startup Scripts... Apr 6 14:14:46 newforum GCEMetadataScripts[1575]: 2021/04/06 14:14:46 GCEMetadataScripts: Starting startup scripts (version 20201217.02-0ubuntu1~18.04.0). Apr 6 14:14:46 newforum GCEMetadataScripts[1575]: 2021/04/06 14:14:46 GCEMetadataScripts: Found startup-script in metadata. Apr 6 14:14:46 newforum GCEMetadataScripts[1575]: 2021/04/06 14:14:46 GCEMetadataScripts: startup-script: Skipping adding existing rule Apr 6 14:14:46 newforApr 6 14:14:46 newforum systemd[1]: Started Google Compute Engine Startup Scripts. Apr 6 14:14:46 newforum systemd[1]: Startup finished in 6.585s (kernel) + 17.727s (userspace) = 24.313s. Ubuntu 18.04.5 LTS newforum ttyS0 newforum login: Apr 6 14:15:14 newforum snapd[1048]: stateengine.go:150: state ensure error: Get https://api.snapcraft.io/api/v1/snaps/sections: dial tcp: lookup api.snapcraft.io on [::1]:53: read udp [::1]:60506->[::1]:53: read: connection refused Apr 6 14:15:16 newforum snapd[1048]: daemon.go:589: gracefully waiting for running hooks Apr 6 14:15:16 newforum snapd[1048]: daemon.go:591: done waiting for running hooks Apr 6 14:15:16 newforum snapd[1048]: daemon stop requested to wait for socket activation Apr 6 14:29:37 newforum systemd[1]: Starting Cleanup of Temporary Directories... Apr 6 14:29:37 newforum systemd[1]: Started Cleanup of Temporary Directories. 

And I still can't log into the instance. How can I do?

Answer 1

First of all, your post might be deleted since sensitive information was shared and it might have been flagged already. To minimize risks to your project I recommend you to speed up the deletion process (If you just edit the post the information is kept in the edit history).

That being said, please check the following:

Reading through your logs I noticed the error Permission denied (publickey) which is commonly attributed to OS login activated at instance or project level.

Follow this guide to disable OS Login if enabled, as explained in the warning for this step:

Caution: Enabling OS Login on instances disables metadata-based SSH key configurations on those instances. Disabling OS Login restores SSH keys that you have configured in project or instance metadata.

You can also try to enable Serial Port access to this instance and troubleshoot the SSH connection from here, the steps are as follow:

1. Enabling access for a VM instance

• In the Google Cloud Console, go to the VM instances page.
• Go to the VM instances page
• Click the instance you want to enable access for.
• Click Edit.
• Under the Remote access section, toggle the Enable connecting to serial ports checkbox.
• Save your changes.

2. Adding a startup script to create a local user.

• In the Google Cloud Console, go to the VM instances page.

• Go to the VM instances page

• Click the instance for which you want to add a startup script. The instance details page displays.

• From the instance details page, complete the following steps: Click the Edit button at the top of the page.

• Under Custom metadata, click Add item.

• Add your startup script using one of the following keys (you should replace variables on caps like USERNAME and PASSWORD):

Key: startup-script Value: #! /bin/bash sudo useradd USERNAME; echo -e "PASSWORD\nPASSWORD" | passwd USERNAME | echo 'USERNAME ALL=(ALL:ALL) ALL' >> /etc/sudoers 

• With this startup script you will create a local user on linux OS with sudo privileges.

• Stop/Start the VM

3. Connecting to a serial console

• In the Google Cloud Console, go to the VM instances page.
• Go to the VM instances page
• Click the instance you want to connect to.
• Under Remote access, click Connect to the serial console to connect on the default port (port 1).
• Login using the USERNAME and PASSWORD you provided on the script.
• Once logged in just type “bash” command to start the bash terminal like when accessed via SSH.
• Restart SSH service:

$ sudo service sshd status $ sudo service sshd restart 

• After restarting, try to connect via SSH again.

As a recovery method, if you are able to create a new instance and connect to it with no problem, you can create a Snapshot from the disk of the affected instance and create a new instance from it or you can detach the affected disk and attach it to a new instance to transfer the information.

It would also be helpful if you collect and share your logs:

1. Go to Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM -> at the VM instance details find section Logs and click on Serial port 1 (console)

2. Reboot your VM instance (if possible).

3. Check full boot log for any errors or/and warnings.

4. If you found errors/warning related to disk space You can try to resize it according to the documentation Resizing a zonal persistent disk or follow this article Recovering an inaccessible instance or a full boot disk.

5. If you require more help, share the full log by using pastebin.com.