12

I have read numerous solutions to this problem, but none seem to apply to what I'm seeing. Most focus on directory permissions, but those appear to be correct in this case. TL;DR: Two Centos7 servers with the same home directory; one's sshd is not allowing publickey authentication even though it is enabled.

I have two centos7 servers, let's call them centos-a and centos-b. Home directories are mounted via NFS, so the .ssh directories are identical between both (confirmation of this below). I can ssh from centos-a to centos-a, but not to centos-b. I can ssh from centos-b to centos-a and to centos-b.

ssh-ability centos-a centos-b
centos-a YES NO
centos-b YES YES 
[myuser@centos-a ~]$ ls -la ~/.ssh total 16 drwx------. 1 myuser domain users 0 Jul 6 11:45 . drwx------. 1 myuser domain users 0 Jul 7 13:44 .. -rw-------. 1 myuser domain users 1212 Jul 6 12:02 authorized_keys -rw-------. 1 myuser domain users 1675 Jul 6 11:45 id_rsa -rw-r--r--. 1 myuser domain users 402 Jul 6 11:45 id_rsa.pub -rw-r--r--. 1 myuser domain users 1119 Jul 6 17:49 known_hosts [myuser@centos-a ~]$ md5sum ~/.ssh/* 65b4fdf2d59cee3ae45b8480454453ec /home/myuser/.ssh/authorized_keys fa3e9fc5a8ff08787ff2ba8f979da24e /home/myuser/.ssh/id_rsa dca36ab3ec342423c5eca588f2ad5678 /home/myuser/.ssh/id_rsa.pub f67bc94bc7a30b9876e3027b24f893d8 /home/myuser/.ssh/known_hosts [myuser@centos-a ~]$ ssh centos-a hostname centos-a [myuser@centos-a ~]$ ssh centos-b hostname myuser@centos-b's password: 
[myuser@centos-b ~]$ ls -la ~/.ssh total 16 drwx------. 1 myser domain users 0 Jul 6 11:45 . drwx------. 1 myser domain users 0 Jul 7 13:44 .. -rw-------. 1 myser domain users 1212 Jul 6 12:02 authorized_keys -rw-------. 1 myser domain users 1675 Jul 6 11:45 id_rsa -rw-r--r--. 1 myser domain users 402 Jul 6 11:45 id_rsa.pub -rw-r--r--. 1 myser domain users 1119 Jul 6 17:49 known_hosts [myuser@centos-b ~]$ md5sum ~/.ssh/* 65b4fdf2d59cee3ae45b8480454453ec /home/myuser/.ssh/authorized_keys fa3e9fc5a8ff08787ff2ba8f979da24e /home/myuser/.ssh/id_rsa dca36ab3ec342423c5eca588f2ad5678 /home/myuser/.ssh/id_rsa.pub f67bc94bc7a30b9876e3027b24f893d8 /home/myuser/.ssh/known_hosts [myuser@centos-b ~]$ ssh centos-b hostname centos-b [myuser@centos-b ~]$ ssh centos-a hostname centos-a

As shown above, the permissions on the .ssh directory appears to be correct (and, regardless, is identical between both machines).

ssh -vvv on the failing ssh shows:

OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017 ... debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4 debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000 ... debug1: Host 'centos-b' is known and matches the ECDSA host key. ... debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup gssapi-keyex debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_is_enabled gssapi-keyex debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug2: we did not send a packet, disable method debug3: authmethod_lookup gssapi-with-mic debug3: remaining preferred: publickey,keyboard-interactive,password debug3: authmethod_is_enabled gssapi-with-mic debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1211402155) debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1211402155) debug2: we did not send a packet, disable method debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/myuser/.ssh/id_rsa debug3: send_pubkey_test debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug1: Trying private key: /home/myuser/.ssh/id_dsa debug3: no such identity: /home/myuser/.ssh/id_dsa: No such file or directory debug1: Trying private key: /home/myuser/.ssh/id_ecdsa debug3: no such identity: /home/myuser/.ssh/id_ecdsa: No such file or directory debug1: Trying private key: /home/myuser/.ssh/id_ed25519 debug3: no such identity: /home/myuser/.ssh/id_ed25519: No such file or directory debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password myuser@centos-b's password:

Contrast this to what I see going from centos-b to centos-a, which works:

... debug1: Unspecified GSS failure. Minor code may provide more information Server not found in Kerberos database debug3: send packet: type 50 debug2: we sent a gssapi-with-mic packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug2: we did not send a packet, disable method debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/myuser/.ssh/id_rsa debug3: send_pubkey_test debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 60 debug1: Server accepts key: pkalg rsa-sha2-512 blen 279 debug2: input_userauth_pk_ok: fp SHA256:nAO5pVOzqUQzEUSEBN37WKp6ADs9Sk4rfTRGmk0FHEY debug3: sign_and_send_pubkey: RSA SHA256:nAO5pVOzqUQzEUSEBN37WKp6ADs9Sk4rfTRGmk0FHEY debug3: send packet: type 50 debug3: receive packet: type 52 debug1: Authentication succeeded (publickey).

I've enabled sshd log messages in /etc/ssh/sshd_config and restarted the service

# Logging SyslogFacility AUTH SyslogFacility AUTHPRIV LogLevel INFO

But there are no additional useful messages in either /var/log/secure or /var/log/messages.

Interestingly the ssh from centos-b to centos-b is using gssapi authentication. If I force it to use publickey it fails:

[myuser@centos-b ~]$ ssh -vvv -o PreferredAuthentications=publickey centos-b hostname ... debug1: Offering RSA public key: /home/myuser/.ssh/id_rsa debug3: send_pubkey_test debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 51 ... Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

and I see in /var/log/messages:

Jul 7 13:52:10 centos-b sshd[23266]: Connection closed by 192.168.1.100 port 48064 [preauth]

pubkey is enabled:

[root@centos-b ssh]# sshd -T | grep -i pub pubkeyauthentication yes pubkeyacceptedkeytypes [email protected],ecdsa-sha...

The sshd_config is a stock Centos7 sshd_config, and is identical between centos-a and centos-b (verified by piping the following command through md5sum on both machines

[root@centos-b ssh]# grep -v -e '^#' -e '^$' /etc/ssh/sshd_config HostKey /etc/ssh/ssh_host_rsa_key HostKey /etc/ssh/ssh_host_ecdsa_key HostKey /etc/ssh/ssh_host_ed25519_key SyslogFacility AUTHPRIV AuthorizedKeysFile .ssh/authorized_keys PasswordAuthentication yes ChallengeResponseAuthentication no GSSAPIAuthentication yes GSSAPICleanupCredentials no UsePAM yes X11Forwarding yes AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE AcceptEnv XMODIFIERS Subsystem sftp /usr/libexec/openssh/sftp-server

Any suggestions for what I'm missing?

Improve this question
2
  • 3
    debug3: receive packet: type 51 is a generic SSH2_MSG_USERAUTH_FAILURE per ssh2.h. What does the sshd_config look like on the servers? Commented Jul 8, 2022 at 12:54
  • The sshd_config is the stock one installed with Centos7; I'll update the question with the contents Commented Jul 8, 2022 at 20:28

3 Answers 3

Reset to default
6

The issue will likely have to do with the SELinux contexts on the ID's .ssh directory (and maybe more). look to the contexts (in .ssh) to have a type of ssh_home_t.

This is similar to the SSH user file's required permissions (no rwx for group/other). I've not checked but it may be required on both ends (both source and target user's .ssh dir needs that same contexts).

This is an example of the "correct" contexts:

[account@hostname .ssh]# ls -alZ drwx------. account account unconfined_u:object_r:ssh_home_t:s0 . drwx------. account account unconfined_u:object_r:user_home_dir_t:s0 .. -rw-------. account account unconfined_u:object_r:ssh_home_t:s0 authorized_keys -rw-------. account account unconfined_u:object_r:ssh_home_t:s0 id_rsa -rw-------. account account unconfined_u:object_r:ssh_home_t:s0 id_rsa.pub -rw-r--r--. account account unconfined_u:object_r:ssh_home_t:s0 known_hosts

In order to fix the context, you can use

sudo chcon -t ssh_home_t /home/username/.ssh/authorized_keys
Improve this answer
2
  • Using an NFS mount for the user's home directory and .ssh sub-directory, I had to enable the SELinux use_nfs_home_dirs permission using setsebool -P use_nfs_home_dirs 1 Commented Jun 26, 2024 at 18:02
  • To fix bad permissions, I used chcon -t ssh_home_t /home/user/.ssh/authorized_keys Commented Jul 25, 2024 at 1:24
3

Some additional googling revealed to me that the issue was with SELinux being enabled on the new systems I've configured. In my case setting this to permissive solves my issue.

# getenforce Enforcing # setenforce 0 # getenforce Permissive

That might not be the right solution for people who require SELinux in their environment. The permanent change involves updating /etc/selinux/config.

Improve this answer
1
  • 1
    yo... this is REALLY bad taste. You accepted your own answer, but it is VERY INSECURE and does not address the actual problem. The answer with more upvotes is more correct. Commented Jul 25, 2024 at 1:22
3

Well I just spent 5 hours chasing my tail on this with a newly built Debian 12 server; doing the same things:

  • Checking permissions on the .ssh files/folders
  • Comparing /etc/ssh/sshd_config files with working hosts
  • Comparing sshd -T outputs with working hosts
  • Comparing outputs of ssh -vvv root@host 2>ssh.log
  • LOTS of web research

Turns out, and I am still not quite certain why this happened for this particular host - I do this on dozens of vms a month, but when I executed scp-copy-id root@host, it installed an alternate public key I use for other purposes rather than what I thought was the default ~/.ssh/id_rsa.pub.

So if you have arrived here and have not yet done so, compare the ~/.ssh/authorized_keys on the target with the ~/.ssh/id_rsa.pub on the client.

If these are not the same, then you need to remove the incorrect entry from authorized_keys (or just delete the file if there is only one) on the target host and then get explicit:

ssh-copy-id -i ~/.ssh/id_rsa.pub root@host

I have so amended my build procedures and now I will have several large martinis ...

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.