ssh keys not working



  • The other day I upgraded my nodebb vm from Ubuntu 14.04 to 16.04 to match what we're running here.

    Today I made a fix for the erroneous error message for the youtube plugin and then I went to push to github:

    $ git push --follow-tags origin master
    Permission denied (publickey).
    fatal: Could not read from remote repository.
    

    :wtf:

    Let's look at .ssh:

    ~/.ssh$ ls -la ~
    ...
    drwx------   2 boomzilla boomzilla   4096 Jul 17 14:48 .ssh
    ...
    
    ~/.ssh$ ls -l
    total 16
    -rw------- 1 boomzilla boomzilla  398 Jul 17 13:05 authorized_keys
    -rw------- 1 boomzilla boomzilla 1675 Jul 17 13:05 boomzilla_id_rsa
    -rw-r--r-- 1 boomzilla boomzilla  398 Jul 17 14:48 boomzilla_id_rsa.pub
    -rw-r--r-- 1 boomzilla boomzilla 2214 Jul 18 17:09 known_hosts
    

    OK...that all looks good.

    $ ssh-add -l
    The agent has no identities.
    

    Huh? What am I missing? This was working prior to the upgrade. (INB4: Idiot).



  • Go into your repo and git config core.sshCommand "ssh -i $HOME/.ssh/boomzilla_id_rsa"



  • @ben_lubar Nope. Also it doesn't recognize the key when I try to ssh into the server here (although it does work when I specify the key on the command line).

    Hmmm...I wonder if this is in any way related to the fact that post-upgrade the VM starts out at a text console instead of starting KDE like it did before? Doesn't seem like it should.



  • I used ssh-add to explicitly add the key and now the key works in that shell.

    boomzilla Why does it not Just Work anymore in all places?



  • @boomzilla
    Your key files do not have the standard names. It looks for ~/.ssh/id_rsa. You may also be able to tell it to use this key by default by creating an appropriate ~/.ssh/config - see man ssh_config



  • @PleegWat Hmmm....I copied them to id_rsa. But I didn't have to do that before. Did something change with ssh? I can't see any configuration changes.


  • The Cold Doesn't Bother Us Anyway

    @boomzilla said in ssh keys not working:

    @PleegWat Hmmm....I copied them to id_rsa. But I didn't have to do that before. Did something change with ssh? I can't see any configuration changes.

    in theory you shouldn't have needed to change anything, as ssh should have offered all the public keys it found, but your system level ssh configuration may have changed to disable that functionality as many distributions consider that behavior to be insecure (for reasons i've never been able to get explained properly)

    you can probably reenable the functionality if you fiddle with your system ssh_config settings


  • Discourse touched me in a no-no place

    @boomzilla said in ssh keys not working:

    Huh? What am I missing? This was working prior to the upgrade.

    Does it work when you explicitly put the identities in)? If not, you're being caught by something more obscure like a strengthening of the minimum-required-security-config that is making it reject your key during loading. That sort of thing is really annoying.

    The -v option to ssh itself can help a lot with debugging this sort of thing. (Also, I use https URLs with github, not ssh ones, but that's because it works better with my IDE.)



  • @boomzilla said in ssh keys not working:

    Hmmm...I wonder if this is in any way related to the fact that post-upgrade the VM starts out at a text console instead of starting KDE like it did before?

    You upgraded Ubuntu version and expected things to still work ?
    :wtf:



  • @TimeBandit said in ssh keys not working:

    You upgradedused Ubuntu version and expected things to still work ?

    FTFY :trolleybus:



  • @accalia said in ssh keys not working:

    you can probably reenable the functionality if you fiddle with your system ssh_config settings

    I've never changed them before and an inspection of what I have on a different 14.04 vs the 16.04 showed no changes.

    @dkf said in ssh keys not working:

    Does it work when you explicitly put the identities in)

    Yes.

    @dkf said in ssh keys not working:

    Also, I use https URLs with github, not ssh ones, but that's because it works better with my IDE.

    It wasn't a git thing. I got the same thing using plain old ssh.

    OK, comparing the output of ssh -v...

    On a 14.04 machine:

    debug1: Host 'what.thedailywtf.com' is known and matches the ED25519 host key.
    debug1: Found key in /home/xxx/.ssh/known_hosts:9
    debug1: ssh_ed25519_verify: signature correct
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: SSH2_MSG_NEWKEYS received
    debug1: SSH2_MSG_SERVICE_REQUEST sent
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey
    debug1: Next authentication method: publickey
    debug1: Offering RSA public key: boomzilla
    debug1: Server accepts key: pkalg ssh-rsa blen 279
    debug1: Authentication succeeded (publickey).
    

    The respective lines in my 16.04:

    debug1: Host 'what.thedailywtf.com' is known and matches the ED25519 host key.
    debug1: Found key in /home/boomzilla/.ssh/known_hosts:7
    debug1: rekey after 4294967296 blocks
    debug1: SSH2_MSG_NEWKEYS sen
    

    Got rid of known_hosts to see if that would make a difference...

    Warning: Permanently added 'what.thedailywtf.com,158.69.225.103' (ED25519) to the list of known hosts.
    debug1: rekey after 4294967296 blocks
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: rekey after 4294967296 blocks
    

    Nope.


  • The Cold Doesn't Bother Us Anyway

    @boomzilla said in ssh keys not working:

    @accalia said in ssh keys not working:

    you can probably reenable the functionality if you fiddle with your system ssh_config settings

    I've never changed them before and an inspection of what I have on a different 14.04 vs the 16.04 showed no changes.

    @dkf said in ssh keys not working:

    Does it work when you explicitly put the identities in)

    Yes.

    @dkf said in ssh keys not working:

    Also, I use https URLs with github, not ssh ones, but that's because it works better with my IDE.

    It wasn't a git thing. I got the same thing using plain old ssh.

    OK, comparing the output of ssh -v...

    On a 14.04 machine:

    debug1: Host 'what.thedailywtf.com' is known and matches the ED25519 host key.
    debug1: Found key in /home/xxx/.ssh/known_hosts:9
    debug1: ssh_ed25519_verify: signature correct
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: SSH2_MSG_NEWKEYS received
    debug1: SSH2_MSG_SERVICE_REQUEST sent
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey
    debug1: Next authentication method: publickey
    debug1: Offering RSA public key: boomzilla
    debug1: Server accepts key: pkalg ssh-rsa blen 279
    debug1: Authentication succeeded (publickey).
    

    The respective lines in my 16.04:

    debug1: Host 'what.thedailywtf.com' is known and matches the ED25519 host key.
    debug1: Found key in /home/boomzilla/.ssh/known_hosts:7
    debug1: rekey after 4294967296 blocks
    debug1: SSH2_MSG_NEWKEYS sen
    

    Got rid of known_hosts to see if that would make a difference...

    Warning: Permanently added 'what.thedailywtf.com,158.69.225.103' (ED25519) to the list of known hosts.
    debug1: rekey after 4294967296 blocks
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: rekey after 4294967296 blocks
    

    Nope.

    huh.... that's weird as hell.......


  • Discourse touched me in a no-no place

    @boomzilla said in ssh keys not working:

    I used ssh-add to explicitly add the key and now the key works in that shell.

    My (partial) ~/.bashrc on most systems. Will load (and request passwords, if required, on the first shell since boot) all private keys that have a public key present in ~/.ssh

    CERTFILES=
    for public in ~/.ssh/*.pub; do
        private=${public%.*}
        if [ -e $private ]; then
            CERTFILES="${CERTFILES} $private"
        fi
    done
    KEYCHAIN=`which keychain`
    if [ ! "" = "$SSH_AGENT_PID" ]; then
    echo "Keychain: SSH_AGENT_PID is set, so running keychain to load keys."
        $KEYCHAIN $CERTFILES && source ~/.keychain/$HOSTNAME-sh
    else
        echo "Keychain: Found no SSH_AUTH_SOCK, so running keychain to start ssh-agent & load keys."
        $KEYCHAIN $CERTFILES && source ~/.keychain/$HOSTNAME-sh
    fi
    

    And if you want those keys to work on a remote system without having that system actually hold the private keys, ~/.ssh/config:

    host *
         ForwardAgent yes
    

    That goes at the end. Have it set to no, and explicitly set to yes on specific hosts as required if you think it's too open.



  • @PJH So, I just confirmed (again) that my configs are the same. Pretty sure it's all default. These are the only lines that aren't comments:

    Host *
        SendEnv LANG LC_*
        HashKnownHosts yes
        GSSAPIAuthentication yes
        GSSAPIDelegateCredentials no
    

    Nothing in either .bashrc doing anything ssh related. I mean...I know how to make it work now, I'm just confused as to why it stopped working. I'm assuming something changed between versions:

    • OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
    • OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016

  • Discourse touched me in a no-no place

    @boomzilla said in ssh keys not working:

    I'm assuming something changed between versions:

    <looks>

    Nope. both MAJOR version numbers look the same... Even the supporting library PATCH versions look similar.. :control_knobs:

    More seriously, I suspect some CVE mitigation between them has resulted in some assumptions being either being rescinded or made invalid resulting in your particular setup being affected.



  • @PJH said in ssh keys not working:

    Nope. both MAJOR version numbers look the same.

    On openssl, yes, not on openssh (6 vs 7). Though I have no idea which one is responsible for determining which keys to use (I would guess ssh, but that's just an ass pull).


  • Discourse touched me in a no-no place

    @boomzilla said in ssh keys not working:

    I would guess ssh, but that's just an ass pull

    My gut reaction too, given the name of where the keys and config are actually stored...


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.