More bash WTF-ery
-
I need to split something like CSV into multiple variables.
Let's start with the basic
readpattern...read a b <<<'a b' ; echo "$a|$b" # > a|bOk... how about a way to get the dynamic input?
read a b <<<$( echo 'a b' ) ; echo "$a|$b" # > a|bNice... Now, I need to split on something other than space...
IFS=, read a b <<<'a,b' ; echo "$a|$b" # > a|bStill good... And now for the final touch, to put it all together...
IFS=, read a b <<<$( echo 'a,b' ) ; echo "$a|$b" # > a b|WTF Bash!? Y U hate me!? Wasn't shellshock enough!?
To make it clear...
echo "a='$a'" # > a='a b' echo "b='$b'" # > b=''If any of you bash gurus can spend a minute, mind telling me what the hell is going on here?
-
looks like the IFS=, part doesn't operate in the correct scope when you do the $( echo 'a,b' ) as opposed to regular 'a,b' ?
because it tried to split on " " again, and there isn't one.
-
My thinking too. But doesn't that sets the environment for the
readpart? Or did it all go into a subshell somehow?This didn't work either
read a b <<<$( IFS=, echo 'a,b' ) ; echo "$a|$b" # > a,b|
-
no idea there, i'd guess subshell-hell. (i actually read yours as "subs" "hell")
can set IFS=, on bash.rc and see if fix?
i'm not an expert when it comes to environmental problems (of the shell or nature variety)
-
oh. i dont know that this is an environmental problem actually, but this fixes it.
IFS=, read a b <<<"$(echo "a,b")"; echo "$a|$b"
needs more quotes ;o
-
Yay! Nice one, @darkmatter.
I'd give you a cookie, but I eat them all myself, so have a like instead.
-
IFS=, read a b <<<"$(echo "a,b")"; echo "$a|$b"
Incidentally, this also works (but uses pipes instead of the
<<<syntax)IFS=, echo "a,b" | read a b ; echo "$a|$b"
-
I'm partial to this format:
[code]IFS=, read a b <<<"echo "a,b""; echo "$a|$b"[/code]
for backticks
-
Ahh, good old bash quote inception.
-
Related to this, here's more bash insanity.
I'm actually trying to parse the output from SSH and do something with that stupid message warning you that the RSA key on the other end has changed. You know, this thing:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is ... Please contact your system administrator. Add correct host key in /home/cartman/.ssh/known_hosts to get rid of this message. Offending RSA key in /home/cartman/.ssh/known_hosts:14 Password authentication is disabled to avoid man-in-the-middle attacks. Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).So, in order to extract the known_hosts line and location, I'm doing something like this:
local vals=$( echo "$ssh_output" | sed -nr 's/Offending RSA key in ([^:]+):([0-9]+)/\1,\2/p' ) IFS=, read path line <<<"$(echo "$vals")" echo "Need to remove line $line from $path"The idea being, I'll get the output like this:
Need to remove line 14 from /home/cartman/.ssh/known_hostsInstead, what I get is this:
from /home/ivan/.ssh/known_hostsSo I've been wrecking my brain the last hour at work, trying various things and just couldn't pin it down. The echo output kept getting chewed in unpredictable ways, almost like there was a memory leak or something. So I said, screw it. Went home and played my games and tried to forget about this stupid thing.
It kept bugging me.
So now, it's 1:30 in the morning. I've been at it for the last hour or so. I finally nailed the sucker.
The money-shot command:
echo "START:$line:END" | cat -v # > START:14^M:END^Mis apparently what you get when you half-parse windows style output on a *nix machine (the\npart, I presume). Which of course does FUCK ALL to explain- why did
sshproduce this mangled line ending - why did
sedadd this garbage to my NUMBERS FUCKING ONLY capture group - why did
echochoke on this one character and broke apart into a PILE OF SHIT it is
Like the shellshock and the IFS thing weren't enough, I needed this TRIFECTA OF LINUX BULLSHIT piled on top of it.
FUCK YOU LINUX.
I'm going to sleep.
- why did
-
why did sed add this garbage to my NUMBERS FUCKING ONLY capture group
It didn't; It simply failed to remove it from the line. Nothing in the RE matched it; therefore, it was not affected by the substitution.
why did echo choke on this one character and broke apart into a PILE OF SHIT it is
I think the output you got is probably
Need to remove line 14\r from /home/cartman/.ssh/known_hosts
You couldn't see the "Need to remove..." because it was overwritten by the " from /home..." part.
-
Which is why this isn't in the sidebar. I know it's my mistake, but they made it easy for me.
Oh, except the ssh printing its output using
\ninstead of\r. Fuck that shit.
-
If you're trying to automatically remove the offending line, there's an easier trick: You can configure the affected host to store its keys in /dev/null, without asking whether to store a new key.
I did this for our QA boxes when they were getting reimaged once a week or so.
-
So, in order to extract the known_hosts line and location, I'm doing something like this:
What's wrong with
ssh-keygen -R <hostname>?
-
Isn't it also a bit WTF to respond to the "remote host identification has changed" warning by trying to automatically remove the stored key? Security features are there for a reason, and if you're dealing with this often enough that you have to write a script to do it, something is horribly wrong.
-
Security features are there for a reason, and if you're dealing with this often enough that you have to write a script to do it, something is horribly wrong.
In this instance the something is the remote host changing its keys that frequently...
-
As mentioned, I've ran into hosts being re-imaged often and changing their key during that. Scripting it is still a bad idea though - do it in configuration instead.
I don't have the config I used to hand (it was at work), but check 'man ssh_config'. You need to make a host-specific section for the affected hosts, setting the authorized_keys file to /dev/null, and disabling confirmation when the key is not known.
That file is also useful to set different default user names for different remote hosts.
-
but check 'man ssh_config'.
I know. Work box:
[pjh@lenovo ~]$ wc -l ~/.ssh/config 128 /home/pjh/.ssh/config [pjh@lenovo ~]$Home:
[pjh@sofa ~]$ wc -l ~/.ssh/config 425 /home/pjh/.ssh/config [pjh@sofa ~]$Having looked at the home one, that needs pruning - lot of out of date stuff in there...
-
If you're trying to automatically remove the offending line, there's an easier trick: You can configure the affected host to store its keys in /dev/null, without asking whether to store a new key.
I did this for our QA boxes when they were getting reimaged once a week or so.
Or just delete the known_hosts file and start over. I still like to have some protection.
What's wrong with ssh-keygen -R <hostname>?
Sounds good.
However, in my specific case, at the point I need to do this, I have the ssh error output but no longer have the original hostname. I can yank it through the call chain, but IMO it's not a big deal to just delete this line myself.
Isn't it also a bit WTF to respond to the "remote host identification has changed" warning by trying to automatically remove the stored key?
Not automatically. This will be a user triggered action.
Security features are there for a reason, and if you're dealing with this often enough that you have to write a script to do it, something is horribly wrong.
Sure, for an old fashioned kind of server you inherited from granpa, and are carefully tuning and optimizing and love like a pet. We are more in the fashionable "servers as cattle" mindset, where we just deal with shitton of crappy VM-s all over the place. This kind of thing is more commonplace then you think.
-
IFS=, read a b <<<"$(echo "a,b")"; echo "$a|$b"
This also works:
cat <<<$( echo 'a,b' ) | { IFS=, read a b ; echo "$a|$b" ; }But in general, this is about the point where I go “fuck it” and use a real programming language with a decidable and predictable grammar.
-
The problem is you don't want the read to be in a subshell, because the variables it sets won't be visible outside it.
I've never seen <<<$(...) used before. I'd probably set up a named pipe using <(). Also remember you can use exec to set up extra file descriptors.
read a b < <(echo "a,b")
Untested, I don't have a bash shell to hand.
-
The problem is you need to set
IFSto comma for theread, which itself sets variables, but you really don't want to set it wider than you have to. Or you could use the-doption (and only one variable perread) which reduces the madness slightly, but you still need to mess around with subshells to get the value actually going into where they need to go.The real solution appears to be to (ab)use
setand positional parameters (with the help ofshiftto undo some of the abuse):{ IFS=, ; set x $( echo 'Q,W,E' ) ; shift; }; echo "$1|$2|$3"I truly hate doing complicated scripting in Shell.
-
Doesn't
shiftmove argv?
-
yes. it'sd massively useful and also incredibly dangerous.
usually if i find myself reaching for shift i instead reach for python or perl
-
I use it in a bunch of my private helper scripts. I'll have 1 or 2 positional arguments I handle specially, shift them out, then pass the remainder to a different command (usually find)
-
Doesn't
shiftmove argv?Of course. That's why I put the
xin there as well, that and the fact thatsetis evil and does something different when you don't give it any arguments; you have to allow for an empty value in the general case.(The mad tricks you learn from reading
configurescripts…)
