Running xubuntu 16.04, with xfce, I'm trying to use ssh keys with passphrases. I would like to add my passphrased key to my ssh-agent, but I don't know why I can't add it. I don't have gnome keyring enabled or anything alike in my startup. ssh-add privatekey, adds the key but when I try to ssh again it just prints the error two times. Some fixes say to disable the gnome keyring on startup but I've already had it disabled. This all occured when I replaced the ssh keys for my raspberrypi.
> OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016 debug1: Reading configuration data /home/potato/.ssh/config debug1: /home/potato/.ssh/config line 1: Applying options for paj debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug1: Connecting to 111.229.105 [111.229.105] port 22253. debug1: Connection established. debug1: identity file /home/potato/.ssh/hplaptop_to_pi type 1 debug1: key_load_public: No such file or directory debug1: identity file /home/potato/.ssh/hplaptop_to_pi-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Raspbian-5+deb8u3 debug1: match: OpenSSH_6.7p1 Raspbian-5+deb8u3 pat OpenSSH* compat 0x04000000 debug1: Authenticating to 111.229.105:22253 as 'pi' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: [email protected] debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ecdsa-sha2-nistp256 SHA256:zrjeaaHD8TjzsdsdssssA2fXnG3gxp2U debug1: Host '[111.229.105]:22253' is known and matches the ECDSA host key. debug1: Found key in /home/potato/.ssh/known_hosts:2 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/potato/.ssh/hplaptop_to_pi debug1: Server accepts key: pkalg ssh-rsa blen 535 sign_and_send_pubkey: signing failed: agent refused operation debug1: Offering RSA public key: potato@potato-HP-tomato debug1: Authentications that can continue: publickey debug1: Offering RSA public key: potato@hplaptop debug1: Authentications that can continue: publickey debug1: Offering RSA public key: potato@hplaptop debug1: Server accepts key: pkalg ssh-rsa blen 535 sign_and_send_pubkey: signing failed: agent refused operation debug1: Offering RSA public key: rsa-key-20141222 debug1: Authentications that can continue: publickey debug1: Offering RSA public key: potato@potatolaptop debug1: Authentications that can continue: publickey debug1: No more authentication methods to try. Permission denied (publickey). 
ssh-add -L? What is inecho $SSH_AUTH_SOCK?ssh-add -Loutputs the public(private?) keys, a few of them.echo #SSH_AUTH_SOCKoutputs/run/user/1000/keyring/sshssh-agent.