More bash WTF-ery



  • I need to split something like CSV into multiple variables.

    Let's start with the basic read pattern...

    read a b <<<'a b' ; echo "$a|$b"
    # > a|b
    

    Ok... how about a way to get the dynamic input?

    read a b <<<$( echo 'a b' ) ; echo "$a|$b"
    # > a|b
    

    Nice... Now, I need to split on something other than space...

    IFS=, read a b <<<'a,b' ; echo "$a|$b"
    # > a|b
    

    Still good... And now for the final touch, to put it all together...

    IFS=, read a b <<<$( echo 'a,b' ) ; echo "$a|$b"
    # > a b|
    

    WTF Bash!? Y U hate me!? Wasn't shellshock enough!?

    To make it clear...

    echo "a='$a'"
    # > a='a b'
    
    echo "b='$b'"
    # > b=''
    

    If any of you bash gurus can spend a minute, mind telling me what the hell is going on here?


  • :belt_onion:

    looks like the IFS=, part doesn't operate in the correct scope when you do the $( echo 'a,b' ) as opposed to regular 'a,b' ?

    because it tried to split on " " again, and there isn't one.



  • My thinking too. But doesn't that sets the environment for the read part? Or did it all go into a subshell somehow?

    This didn't work either

    read a b <<<$( IFS=, echo 'a,b' ) ; echo "$a|$b"
    # > a,b|
    

  • :belt_onion:

    no idea there, i'd guess subshell-hell. (i actually read yours as "subs" "hell")

    can set IFS=, on bash.rc and see if fix?

    i'm not an expert when it comes to environmental problems (of the shell or nature variety)


  • :belt_onion:

    oh. i dont know that this is an environmental problem actually, but this fixes it.

    IFS=, read a b <<<"$(echo "a,b")"; echo "$a|$b"

    needs more quotes ;o



  • Yay! Nice one, @darkmatter.

    I'd give you a cookie, but I eat them all myself, so have a like instead.



  • @darkmatter said:

    IFS=, read a b <<<"$(echo "a,b")"; echo "$a|$b"

    Incidentally, this also works (but uses pipes instead of the <<< syntax)

    IFS=, echo "a,b" | read a b ; echo "$a|$b"


  • :belt_onion:

    I'm partial to this format:
    [code]IFS=, read a b <<<"echo "a,b""; echo "$a|$b"[/code]

    :green_heart: for backticks



  • Ahh, good old bash quote inception.



  • Related to this, here's more bash insanity.

    I'm actually trying to parse the output from SSH and do something with that stupid message warning you that the RSA key on the other end has changed. You know, this thing:

    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that a host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    ...
    Please contact your system administrator.
    Add correct host key in /home/cartman/.ssh/known_hosts to get rid of this message.
    Offending RSA key in /home/cartman/.ssh/known_hosts:14
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
    

    So, in order to extract the known_hosts line and location, I'm doing something like this:

    local vals=$( echo "$ssh_output" | sed -nr 's/Offending RSA key in ([^:]+):([0-9]+)/\1,\2/p' )
    IFS=, read path line <<<"$(echo "$vals")"
    echo "Need to remove line $line from $path"
    

    The idea being, I'll get the output like this:

    Need to remove line 14 from /home/cartman/.ssh/known_hosts
    

    Instead, what I get is this:

     from /home/ivan/.ssh/known_hosts
    

    So I've been wrecking my brain the last hour at work, trying various things and just couldn't pin it down. The echo output kept getting chewed in unpredictable ways, almost like there was a memory leak or something. So I said, screw it. Went home and played my games and tried to forget about this stupid thing.

    It kept bugging me.

    So now, it's 1:30 in the morning. I've been at it for the last hour or so. I finally nailed the sucker.

    The money-shot command:

    echo "START:$line:END" | cat -v
    # > START:14^M:END
    

    ^M is apparently what you get when you half-parse windows style output on a *nix machine (the \n part, I presume). Which of course does FUCK ALL to explain

    • why did ssh produce this mangled line ending
    • why did sed add this garbage to my NUMBERS FUCKING ONLY capture group
    • why did echo choke on this one character and broke apart into a PILE OF SHIT it is

    Like the shellshock and the IFS thing weren't enough, I needed this TRIFECTA OF LINUX BULLSHIT piled on top of it.

    FUCK YOU LINUX.

    I'm going to sleep.



  • @cartman82 said:

    why did sed add this garbage to my NUMBERS FUCKING ONLY capture group

    It didn't; It simply failed to remove it from the line. Nothing in the RE matched it; therefore, it was not affected by the substitution.

    @cartman82 said:

    why did echo choke on this one character and broke apart into a PILE OF SHIT it is

    I think the output you got is probably
    Need to remove line 14\r from /home/cartman/.ssh/known_hosts
    You couldn't see the "Need to remove..." because it was overwritten by the " from /home..." part.



  • Which is why this isn't in the sidebar. I know it's my mistake, but they made it easy for me.

    Oh, except the ssh printing its output using \n instead of \r. Fuck that shit.



  • If you're trying to automatically remove the offending line, there's an easier trick: You can configure the affected host to store its keys in /dev/null, without asking whether to store a new key.

    I did this for our QA boxes when they were getting reimaged once a week or so.


  • Discourse touched me in a no-no place

    @cartman82 said:

    So, in order to extract the known_hosts line and location, I'm doing something like this:

    What's wrong with ssh-keygen -R <hostname>?



  • Isn't it also a bit WTF to respond to the "remote host identification has changed" warning by trying to automatically remove the stored key? Security features are there for a reason, and if you're dealing with this often enough that you have to write a script to do it, something is horribly wrong.


  • Discourse touched me in a no-no place

    @marinus said:

    Security features are there for a reason, and if you're dealing with this often enough that you have to write a script to do it, something is horribly wrong.

    In this instance the something is the remote host changing its keys that frequently...



  • As mentioned, I've ran into hosts being re-imaged often and changing their key during that. Scripting it is still a bad idea though - do it in configuration instead.

    I don't have the config I used to hand (it was at work), but check 'man ssh_config'. You need to make a host-specific section for the affected hosts, setting the authorized_keys file to /dev/null, and disabling confirmation when the key is not known.

    That file is also useful to set different default user names for different remote hosts.


  • Discourse touched me in a no-no place

    @PleegWat said:

    but check 'man ssh_config'.

    I know. Work box:

    [pjh@lenovo ~]$ wc -l ~/.ssh/config
    128 /home/pjh/.ssh/config
    [pjh@lenovo ~]$ 
    

    Home:

    [pjh@sofa ~]$ wc -l ~/.ssh/config
    425 /home/pjh/.ssh/config
    [pjh@sofa ~]$ 
    

    Having looked at the home one, that needs pruning - lot of out of date stuff in there...



  • @PleegWat said:

    If you're trying to automatically remove the offending line, there's an easier trick: You can configure the affected host to store its keys in /dev/null, without asking whether to store a new key.

    I did this for our QA boxes when they were getting reimaged once a week or so.

    Or just delete the known_hosts file and start over. I still like to have some protection.

    @PJH said:

    What's wrong with ssh-keygen -R <hostname>?

    Sounds good.

    However, in my specific case, at the point I need to do this, I have the ssh error output but no longer have the original hostname. I can yank it through the call chain, but IMO it's not a big deal to just delete this line myself.

    @marinus said:

    Isn't it also a bit WTF to respond to the "remote host identification has changed" warning by trying to automatically remove the stored key?

    Not automatically. This will be a user triggered action.

    @marinus said:

    Security features are there for a reason, and if you're dealing with this often enough that you have to write a script to do it, something is horribly wrong.

    Sure, for an old fashioned kind of server you inherited from granpa, and are carefully tuning and optimizing and love like a pet. We are more in the fashionable "servers as cattle" mindset, where we just deal with shitton of crappy VM-s all over the place. This kind of thing is more commonplace then you think.


  • Discourse touched me in a no-no place

    @darkmatter said:

    IFS=, read a b <<<"$(echo "a,b")"; echo "$a|$b"

    This also works:

    cat <<<$( echo 'a,b' ) | { IFS=, read a b ; echo "$a|$b" ; }
    

    But in general, this is about the point where I go “fuck it” and use a real programming language with a decidable and predictable grammar.



  • The problem is you don't want the read to be in a subshell, because the variables it sets won't be visible outside it.

    I've never seen <<<$(...) used before. I'd probably set up a named pipe using <(). Also remember you can use exec to set up extra file descriptors.

    read a b < <(echo "a,b")

    Untested, I don't have a bash shell to hand.


  • Discourse touched me in a no-no place

    The problem is you need to set IFS to comma for the read, which itself sets variables, but you really don't want to set it wider than you have to. Or you could use the -d option (and only one variable per read) which reduces the madness slightly, but you still need to mess around with subshells to get the value actually going into where they need to go.

    The real solution appears to be to (ab)use set and positional parameters (with the help of shift to undo some of the abuse):

    { IFS=, ; set x $( echo 'Q,W,E' ) ; shift; }; echo "$1|$2|$3"
    

    I truly hate doing complicated scripting in Shell.



  • Doesn't shift move argv?


  • sockdevs

    yes. it'sd massively useful and also incredibly dangerous.

    usually if i find myself reaching for shift i instead reach for python or perl



  • I use it in a bunch of my private helper scripts. I'll have 1 or 2 positional arguments I handle specially, shift them out, then pass the remainder to a different command (usually find)


  • Discourse touched me in a no-no place

    @ben_lubar said:

    Doesn't shift move argv?

    Of course. That's why I put the x in there as well, that and the fact that set is evil and does something different when you don't give it any arguments; you have to allow for an empty value in the general case.

    (The mad tricks you learn from reading configure scripts…)


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.