Can't log in via SSH to any accounts using /bin/bash shell on Synology NAS

Question

I'm trying to install bash as the default shell on a ARM Linux running on an embedded device (Synology DS212+ NAS). But there's something really wrong, and I can't figure out what it is.

Symptoms:

1) Root has /bin/bash as default shell, and can log in normally via SSH:

$ grep root /etc/passwd root:x:0:0:root:/root:/bin/bash $ ssh root@NAS root@NAS's password: Last login: Sun Dec 16 14:06:56 2012 from desktop # 

2) joeuser has /bin/bash as default shell, and receives "Permission denied" when trying to log in via SSH:

$ grep joeuser /etc/passwd joeuser:x:1029:100:Joe User:/home/joeuser:/bin/bash $ ssh joeuser@localhost joeuser@NAS's password: Last login: Sun Dec 16 14:07:22 2012 from desktop Permission denied, please try again. Connection to localhost closed. 

3) changing joeuser's shell back to /bin/sh:

$ grep joeuser /etc/passwd joeuser:x:1029:100:Joe User:/home/joeuser:/bin/sh $ ssh joeuser@localhost Last login: Sun Dec 16 15:50:52 2012 from localhost $ 

To make things even more strange, I can log in as joeuser using /bin/bash using the serial console (!). Also a su - joeuser as root works fine, so the bash binary itself is working fine.

In an act of despair, I changed joeuser's uid to 0 on /etc/passwd, but also didn't work, so it doesn't seem to be something permission related.

Seems that bash is doing some extra checking that sshd didn't like, and blocking the connections for non-root users. Maybe some sort of sanity checking - or terminal emulation - that is triggering the SIGCHLD, but only when called via ssh.

I already went through every single item on sshd_config, and also put SSHD in debug mode, but didn't find anything strange. Here's my /etc/ssh/sshd_config:

LogLevel DEBUG LoginGraceTime 2m PermitRootLogin yes RSAAuthentication yes PubkeyAuthentication yes AuthorizedKeysFile %h/.ssh/authorized_keys ChallengeResponseAuthentication no UsePAM yes AllowTcpForwarding no ChrootDirectory none Subsystem sftp internal-sftp -f DAEMON -u 000 

And here's the output from /usr/syno/sbin/sshd -d, showing the failed attempt of joeuser trying to log in, with /bin/bash as the shell:

debug1: Config token is loglevel debug1: Config token is logingracetime debug1: Config token is permitrootlogin debug1: Config token is rsaauthentication debug1: Config token is pubkeyauthentication debug1: Config token is authorizedkeysfile debug1: Config token is challengeresponseauthentication debug1: Config token is usepam debug1: Config token is allowtcpforwarding debug1: Config token is chrootdirectory debug1: Config token is subsystem debug1: HPN Buffer Size: 87380 debug1: sshd version OpenSSH_5.8p1-hpn13v11 debug1: read PEM private key done: type RSA debug1: private host key: #0 type 1 RSA debug1: read PEM private key done: type DSA debug1: private host key: #1 type 2 DSA debug1: read PEM private key done: type ECDSA debug1: private host key: #2 type 3 ECDSA debug1: rexec_argv[0]='/usr/syno/sbin/sshd' debug1: rexec_argv[1]='-d' Set /proc/self/oom_adj from 0 to -17 debug1: Bind to port 22 on ::. debug1: Server TCP RWIN socket size: 87380 debug1: HPN Buffer Size: 87380 Server listening on :: port 22. debug1: Bind to port 22 on 0.0.0.0. debug1: Server TCP RWIN socket size: 87380 debug1: HPN Buffer Size: 87380 Server listening on 0.0.0.0 port 22. debug1: Server will not fork when running in debugging mode. debug1: rexec start in 6 out 6 newsock 6 pipe -1 sock 9 debug1: inetd sockets after dupping: 4, 4 Connection from 127.0.0.1 port 52212 debug1: HPN Disabled: 0, HPN Buffer Size: 87380 debug1: Client protocol version 2.0; client software version OpenSSH_5.8p1-hpn13v11 SSH: Server;Ltype: Version;Remote: 127.0.0.1-52212;Protocol: 2.0;Client: OpenSSH_5.8p1-hpn13v11 debug1: match: OpenSSH_5.8p1-hpn13v11 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.8p1-hpn13v11 debug1: permanently_set_uid: 1024/100 debug1: MYFLAG IS 1 debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: AUTH STATE IS 0 debug1: REQUESTED ENC.NAME is 'aes128-ctr' debug1: kex: client->server aes128-ctr hmac-md5 none SSH: Server;Ltype: Kex;Remote: 127.0.0.1-52212;Enc: aes128-ctr;MAC: hmac-md5;Comp: none debug1: REQUESTED ENC.NAME is 'aes128-ctr' debug1: kex: server->client aes128-ctr hmac-md5 none debug1: expecting SSH2_MSG_KEX_ECDH_INIT debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: KEX done debug1: userauth-request for user joeuser service ssh-connection method none SSH: Server;Ltype: Authname;Remote: 127.0.0.1-52212;Name: joeuser debug1: attempt 0 failures 0 debug1: Config token is loglevel debug1: Config token is logingracetime debug1: Config token is permitrootlogin debug1: Config token is rsaauthentication debug1: Config token is pubkeyauthentication debug1: Config token is authorizedkeysfile debug1: Config token is challengeresponseauthentication debug1: Config token is usepam debug1: Config token is allowtcpforwarding debug1: Config token is chrootdirectory debug1: Config token is subsystem debug1: PAM: initializing for "joeuser" debug1: PAM: setting PAM_RHOST to "localhost" debug1: PAM: setting PAM_TTY to "ssh" debug1: userauth-request for user joeuser service ssh-connection method password debug1: attempt 1 failures 0 debug1: do_pam_account: called Accepted password for joeuser from 127.0.0.1 port 52212 ssh2 debug1: monitor_child_preauth: joeuser has been authenticated by privileged process debug1: PAM: establishing credentials User child is on pid 9129 debug1: Entering interactive session for SSH2. debug1: server_init_dispatch_20 debug1: server_input_channel_open: ctype session rchan 0 win 65536 max 16384 debug1: input_session_request debug1: channel 0: new [server-session] debug1: session_new: session 0 debug1: session_open: channel 0 debug1: session_open: session 0: link with channel 0 debug1: server_input_channel_open: confirm session debug1: server_input_global_request: rtype [email protected] want_reply 0 debug1: server_input_channel_req: channel 0 request pty-req reply 1 debug1: session_by_channel: session 0 channel 0 debug1: session_input_channel_req: session 0 req pty-req debug1: Allocating pty. debug1: session_new: session 0 debug1: session_pty_req: session 0 alloc /dev/pts/1 debug1: server_input_channel_req: channel 0 request shell reply 1 debug1: session_by_channel: session 0 channel 0 debug1: session_input_channel_req: session 0 req shell debug1: Setting controlling tty using TIOCSCTTY. debug1: Received SIGCHLD. debug1: session_by_pid: pid 9130 debug1: session_exit_message: session 0 channel 0 pid 9130 debug1: session_exit_message: release channel 0 debug1: session_by_tty: session 0 tty /dev/pts/1 debug1: session_pty_cleanup: session 0 release /dev/pts/1 Received disconnect from 127.0.0.1: 11: disconnected by user debug1: do_cleanup debug1: do_cleanup debug1: PAM: cleanup debug1: PAM: closing session debug1: PAM: deleting credentials 

Here you have the full output of sshd -dd, together with ssh -vv.

Bash:

# bash --version GNU bash, version 3.2.49(1)-release (arm-none-linux-gnueabi) Copyright (C) 2007 Free Software Foundation, Inc. 

The bash binary was cross compiled from source. I also tried using a pre-compiled binary from the Optware distribution, but had the exact same problem. I checked for missing shared libraries using objdump -x, but they're all there.

Any ideas what could be causing this "Permission denied, please try again."? I'm almost diving in the bash source code to investigate, but trying to avoid hours chasing something that may be silly.

EDIT: adding more information about bash and the system

$ ls -la /bin/bash -rwxr-xr-x 1 root root 724676 Dec 15 23:57 /bin/bash $ file /bin/bash /bin/bash: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.14, stripped $ uname -a Linux NAS 2.6.32.12 #2661 Mon Nov 12 23:10:15 CST 2012 armv5tel GNU/Linux synology_88f6282_212+ $ grep bash /etc/shells /bin/bash /bin/bash2 

Answer 1

For future reference: after way too many hours researching and debugging this issue, I finally discovered the root cause.

The OpenSSH version used by Synology is a highly customized version, that does not behave like the original code. It has lots of hacks and ad-hoc customizations - e.g., additional checking before accepting a login to see if the SSH service is enabled within the web interface, or stripping special chars (;, |, ') from rsync commands, or... wait for it... avoiding regular users to use a shell different than /bin/sh or /bin/ash. Yeah, hard coded within the binary.

Here's the piece of code from OpenSSH 5.8p1, as distributed by Synology on their source code (DSM4.1 - branch 2636), file session.c:

void do_child(Session *s, const char *command) { ... #ifdef MY_ABC_HERE char szValue[8]; int RunSSH = 0; SSH_CMD SSHCmd = REQ_UNKNOWN; if (1 == GetKeyValue("/etc/synoinfo.conf", "runssh", szValue, sizeof(szValue))) { if (strcasecmp(szValue, "yes") == 0) { RunSSH = 1; } } if (IsSFTPReq(command)){ SSHCmd = REQ_SFTP; } else if (IsRsyncReq(command)){ SSHCmd = REQ_RSYNC; } else if (IsTimebkpRequest(command)){ SSHCmd = REQ_TIMEBKP; } else if (RunSSH && IsAllowShell(pw)){ SSHCmd = REQ_SHELL; } else { goto Err; } if (REQ_RSYNC == SSHCmd) { pw = SYNOChgValForRsync(pw); } if (!SSHCanLogin(SSHCmd, pw)) { goto Err; } goto Pass; Err: fprintf(stderr, "Permission denied, please try again.\n"); exit(1); Pass: #endif /* MY_ABC_HERE */ ... } 

As you can imagine, the IsAllowShell(pw) was the culprit:

static int IsAllowShell(const struct passwd *pw) { struct passwd *pUnPrivilege = NULL; char *szUserName = NULL; if (!pw || !pw->pw_name) { return 0; } szUserName = pw->pw_name; if(!strcmp(szUserName, "root") || !strcmp(szUserName, "admin")){ return 1; } if (NULL != (pUnPrivilege = getpwnam(szUserName))){ if (!strcmp(pUnPrivilege->pw_shell, "/bin/sh") || !strcmp(pUnPrivilege->pw_shell, "/bin/ash")) { return 1; } } return 0; } 

No wonder why I was experiencing such an odd behavior. Only shells /bin/sh and /bin/ash would be accepted for users different than root or admin. And this regardless of the uid (I had tested also making joeuser uid=0, and it didn't work. Now it's obvious why).

Once identified the cause, the fix was easy: just remove the call to IsAllowShell(). It took me a while to get the right configuration to cross-compile openssh and all its dependencies, but it worked well in the end.

If anyone is interested in doing the same (or trying to cross-compile other kernel modules or binaries for Synology), here's my version of Makefile. It was tested with OpenSSH-5.8p1 source, and works well with models running Marvell Kirkwood mv6281/mv6282 CPU (like DS212+). I used a host running Ubuntu 12.10 x64.

Bottom line: bad practice, terrible code, and a great example of what not to do. I understand sometimes OEMs need to develop special customizations, but they should think twice before digging too deep. Not only this results in unmaintainable code for them, but also creates all sorts of unforeseen issues down the road. Thankfully GPL exist to keep them honest - and open.

Answer 2

To circumvent the problem, and since I installed bash via ipkg and I can't be sure /opt will always be available (mounted correctly), I simply put the following in my .profile

[ -x /opt/bin/bash ] && exec /opt/bin/bash 

while /etc/passwd contains /bin/ash as shell.

Answer 3

Let's see. It's isolated to a single shell, plus you're looking at sshd debug output, so it's not world writable permission problems with ~joeuser/.ssh. That's the one that gets most people.

Have you tried creating an additional normal user (i.e. not joeuser) to make sure it experiences the same problem? That would isolate it to the user's configuration vs. system-wide configuration.

If it's a system-wide issue, the next thing I'd take a look at are shared configuration files like /etc/profile that get sourced by everyone. There might be a conditional block that isn't firing if the username is root. (not effective userid, since you already tested for that)

Check dmesg for segmentation fault reports if you haven't already, just in case there's something even weirder going on.

Answer 4

try /etc/ssh/sshd_config
search for AllowUsers

if it's there try adding joeuser there, just username

also it may be blocked in pam... I do not recall which file is it...

Answer 5

Their modified version of openssh looks for a /bin/sh shell ?

Easy solution then:

ln -fs /bin/bash /bin/sh

Answer 6

In case anyone stumbles across this because they made the same mistake I did:

yes: $ sudo usermod -s /bin/bash your_username

no: $ sudo usermod -s bash your_username

The 2nd will result in a permission denied when ssh'ing in.

Answer 7

Try reinstalling Bash and see if that helps.