Bash script help (because I suck at it)


  • BINNED

    Ok, so I'm terrible with bash scripting — I never know which tool to choose for output processing, and I only know the basics of sed even then, so I need some advice.

    Sample output:

    Test-Queue1 has 0 calls (max 3) in 'ringall' strategy (2s holdtime, 0s talktime), W:0, C:1, A:37, SL:0.0% within 0s
       No Members
       No Callers
     
    Test-Queue2 has 0 calls (max 2) in 'ringall' strategy (0s holdtime, 0s talktime), W:0, C:0, A:0, SL:0.0% within 0s
       Members:
          SIP/933 (ringinuse disabled) (dynamic) (Not in use) has taken 1 calls (last was 151789 secs ago)
          SIP/931 (ringinuse disabled) (dynamic) (Not in use) has taken no calls yet
          SIP/930 (ringinuse disabled) (dynamic) (Not in use) has taken no calls yet
       No Callers
    

    What I need:

    • name of the queue — in this case it's Test-Queue1 and Test-Queue2. I need this stored into a variable
    • name of the member — for purposes of this it's pretty much (SIP/\d+)\s. I need this stored in a variable as well
    • x in (last was x secs ago) — I need this compared to a value

    Basically I need to execute a command if x is larger than a predefined value and the command will contain the other two values so I need them for concatenation purposes.

    General outline would probably be enough I guess, I'm fine with looking up docs where needed, I just suck at choosing the right tool for the job...


  • Banned

    @Onyx have you thought of making Python script instead?


  • BINNED

    @Gąska yeah, well, I considered something other than bash, though I'd probably go PHP over Python because I can be sure that's installed (yes, I know, it's Python, what are the odds of it not being installed, still, paranoid). It seemed kinda silly to pull out and batch run an extra interpreter every minute for something like this.


  • Banned

    @Onyx rule of thumb: if you need to put even the least bit of logic in your script, DON'T USE SHELL SCRIPTS.


  • Discourse touched me in a no-no place

    @Gąska said in Bash script help (because I suck at it):

    @Onyx rule of thumb: if you need to put even the least bit of logic in your script, DON'T USE SHELL SCRIPTS.

    This.

    While I've written some pretty hefty bash scripts in my time (what passes for my CI server uses a 648 line one for example,) the input they require is generally minimal - you're basically after something that can parse input and keep internal state which bash is particularly bad at IME.

    Find another language is my advice.



  • Is this not the raison d'être for Perl?

    Just sayin, is all.



  • @Gąska Which is funny because half of the logic in any Linux system is written in shell scripts.



  • @Onyx Pipe your sample output into this script's standard input:

    #!/bin/bash
    mintime=3600
    
    sed -ne '
    /^[^ ]/{
            s/ .*//
            h
    }
    /last was /{
            s/^ *//
            s/ .*last was /\t/
            s/ .*$//
            G
            s/\n/\t/
            p
    }' |
    
    while read member time queue
    do
            if ((time > mintime))
            then
                    echo "$time > $mintime for member $member in queue $queue"
            fi
    done
    
    

  • Discourse touched me in a no-no place

    @Onyx said in Bash script help (because I suck at it):

    bash scripting

    @flabdablet said in Bash script help (because I suck at it):

    ['bash' script]

    That's like one of those 'tetris in 140 bytes' things isn't it? With most of the hard work hidden behind a curtain somewhere?

    👿

    (That said I thought sed might have been a cromulent answer - I just haven't (ab)used it sufficiently yet to have given this one a go.)


  • Winner of the 2016 Presidential Election

    @anonymous234 said in Bash script help (because I suck at it):

    Which is funny because half of the logic in any Linux system is written in shell scripts.

    Thankfully, systemd already killed a lot of shell scripts.


  • Banned

    @anonymous234 said in Bash script help (because I suck at it):

    @Gąska Which is funny because half of the logic in any Linux system is written in shell scripts.

    It would be funny if no one had to use Linux.



  • @flabdablet I always forget sed is much MUCH more powerful than the simple filtering or search/replace regex it's usually used for.


  • Discourse touched me in a no-no place

    @PJH said in Bash script help (because I suck at it):

    (That said I thought sed might have been a cromulent answer - I just haven't (ab)used it sufficiently yet to have given this one a go.)

    I think it's possible, but painful as it doesn't really have great choice of internal variables. This is where awk is better. Or any of a number of other programming languages.



  • @dkf said in Bash script help (because I suck at it):

    This is where awk is better.

    Considered awk, but for this particular job it seemed to me that sed was just as easy and rather less wordy.



  • @dkf said in Bash script help (because I suck at it):

    This is where awk is better. Or any of a number of other programming languages.

    Awk, perl, python — which I'd choose would depend on circumstances. If I just needed to get it done fast, I'd use perl because I know it the best by far. If I wasn't in such a hurry, I might chose python, because it's a simple little script that would give me a chance to practice writing/thinking pythonically. Awk would probably be fine, too, but I don't know it and see no reason to learn it when perl or python will do everything it will do and more.


  • Discourse touched me in a no-no place

    @flabdablet said in Bash script help (because I suck at it):

    rather less wordy

    Well yes, almost everything is wordier than sed. Not sure if that counts as a recommendation. ;)

    OTOH, I never really got the hang of how to handle multiple lines in sed. I know it can do it, but I just didn't bother to learn as at that point I could always reach for a different tool that wasn't quite so miserably terse. I'll still use sed for some things, but not usually where the problem is as complex as the one described here.


  • Considered Harmful

    @HardwareGeek said in Bash script help (because I suck at it):

    little script that would give me a chance to practice writing/thinking pythonically

    Would you explain this for those of us that don't speak snake? I know some languages that have really changed the way I think about problems, but I don't know enough about Python to appreciate pythonic thinking. What I saw just looked like a standard mostly procedural language (C-like in structure, if not in syntax).


  • Discourse touched me in a no-no place

    @error While I don't write Python, here's how I'd be thinking:

    1. Read all the input data into memory at once. Because unless it is massive, that's easier. 100MB is not massive.
    2. Split into records. Looks like they're separated by visually-blank lines.
    3. Parse each record. Can be done on a line-by-line basis now.
    4. Output in whatever's convenient.

    I would also be entirely happy with saying “yes, mofos, you need to have a programming tool from this millennium present; suck it up”.


  • Considered Harmful

    I use node as my go-to task interpreter now. It's weird how JavaScript has come from being this pariah toy language to being a dominant force. I don't see eye-to-eye with Douglas Crockford about a lot of things, but he's probably to thank for his evangelism of the language.



  • @error
    I'm probably not the best to explain it, as I don't really python, yet. You're right, it is pretty much a standard procedural language. The main difference I see, so far, from languages that I'm more accustomed to is that, in many cases, operations that would require explicit loops to iterate over a collection can be condensed into a single expression with the iterator as part of the expression. And everything, even simple numbers, is an object. To me, pythonic thinking consists mostly of sprinkling the syntactic sugar, rather than the more verbose loops I'd use in other languages.

    There's almost certainly more to it than that, but that's what I've picked up in my forays into python (which, so far, have consisted more of learning the API of an application that uses python as its scripting language than of learning the intricacies of the language itself).


  • Considered Harmful

    @HardwareGeek said in Bash script help (because I suck at it):

    operations that would require explicit loops to iterate over a collection can be condensed into a single expression with the iterator as part of the expression.

    Do you mean like

    and
    ?


    Filed under: OneBox FTL



  • @error said in Bash script help (because I suck at it):

    Would you explain this for those of us that don't speak snake? I know some languages that have really changed the way I think about problems, but I don't know enough about Python to appreciate pythonic thinking. What I saw just looked like a standard mostly procedural language (C-like in structure, if not in syntax).

    It's closer to C++ than C, since it does use classes a lot. Other than a few bits of syntactic sugar, though, it isn't really much different than the rest of the C++/Java family.



  • @error said in Bash script help (because I suck at it):

            List comprehension
    

    (Great quote formatting there, NodeBB. :wtf: ) Yeah, exactly. Apparently that feature exists in other languages, too, but none that I'm familiar with.

    @error said in Bash script help (because I suck at it):

            Closure (computer programming)
    

    That too, I guess, but I haven't delved that deeply into python, yet.


  • Banned

    @HardwareGeek said in Bash script help (because I suck at it):

    Apparently that feature exists in other languages, too, but none that I'm familiar with.

    Java 8 in nutshell.


  • BINNED

    Right, well, I whipped something together in PHP while waiting, then it needed modifications and since I was in a hurry I thought "what the hell I understand this better than sed wizardry, I'll study that later"... And it bit me in the ass, of course, fucking PHP...

    I'll pull out the PHP WTF tomorrow, didn't have time to post from work. And I'll study the see solution, seems useful regardless if I use it right now or not, so thanks @flabdablet



  • @Onyx said in Bash script help (because I suck at it):

    sed wizardry

    Most of it is just s/this/that/ commands, which I'm sure you already understand. The only slightly wizardly bits are the use of the -n option (so that sed produces no output at all until it hits an explicit p command), the use of regex addresses along with curly braces to limit certain edits to certain lines, the use of the h command (which copies the pattern space to the hold space) and the G command (which appends a newline, then the hold space, to the pattern space).

    All it's doing is holding the first word of any line that doesn't start with a space (these are your queue names); removing everything but the member name and seconds count from any line containing a seconds count, and then whacking the most recently seen queue name on the end of the result before spitting it out. Lines that contain neither queue names nor seconds counts don't trigger either of the curly-braced clauses, so they're just ignored.

    Convince yourself it's working (or not) by deleting the pipe and everything after it, so you can see exactly what's getting fed into the while read loop.



  • @Onyx sed is the wrong tool. Use awk as others have said.



  • @JazzyJosh Working code or GTFO :-)



  • @flabdablet I haven't written an awk script in 4 years.

    The advantage, @Onyx is that awk is very good at column processing, and the data you need are already in columns.



  • @JazzyJosh meh.

    #!/bin/bash
    mintime=3600
    
    awk '
    /^[^ ]/{
            queue = $1
    }
    /last was /{
            sub(/^ */, "")
            sub(/ .*last was /, "\t")
            sub(/ .*$/, "\t" queue)
            print
    }' |
    
    while read member time queue
    do
            if ((time > mintime))
            then
                    echo "$time > $mintime for member $member in queue $queue"
            fi
    done
    

    @Onyx: Once you understand that awk is essentially a fancier sed, and like sed processes input a line at a time, the only non-obvious thing about the awk script above is the way awk breaks input lines into fields. Fields are a special kind of variable in awk, and they have names that start with $; other variables have plain words for names. When awk reads a line, it goes in field $0 (which is also the default target for the sub and print functions, much like $_ in perl). Fields $1, $2, $3... get implicitly filled in by splitting $0 on whitespace (which is why the awk version of this logic doesn't need to do line editing to isolate the queue name).



  • @Onyx said in Bash script help (because I suck at it):

    It seemed kinda silly to pull out and batch run an extra interpreter every minute for something like this.

    Unfortunately, you're not gaining anything here by sticking to Bash. Bash is going to have to fork to run your script anyway. If this is your only objection, you might as well use Python or even Perl instead of bash.



  • @flabdablet :effort: mostly.



  • @Onyx said in Bash script help (because I suck at it):

    It seemed kinda silly to pull out and batch run an extra interpreter every minute for something like this.

    That's where sed and awk are good, because they're insanely cheap to run (sed more so).


  • Winner of the 2016 Presidential Election

    @JazzyJosh I just read an awk tutorial. I was surprised by how sane the language is. Now I regret not actually learning it earlier. (So far, I've only used it to print a specific output column and remembered that specific command.)



  • @JazzyJosh said in Bash script help (because I suck at it):

    the data you need are already in columns

    Unless you're absolutely sure that the bit that reads (Not in use) will always have three words in it, I don't think that's a safe assumption.


  • BINNED

    @flabdablet said in Bash script help (because I suck at it):

    Unless you're absolutely sure that the bit that reads (Not in use) will always have three words in it

    It won't. Off the top of my head, it can also be In use or Ringing, probably more but I almost never use that specific output for anything so I'm not sure (and Asterisk is many things, but "consistent" ain't one of them, so).

    The modification, BTW, was capturing that as well, the if should be, in pseudo code, if time < constant and state != "In use" (time updates after a call ends, and if it's longer than constant it might log the agent out when not necessary).



  • @asdf said in Bash script help (because I suck at it):

    @JazzyJosh I just read an awk tutorial. I was surprised by how sane the language is. Now I regret not actually learning it earlier. (So far, I've only used it to print a specific output column and remembered that specific command.)

    One of my "if I even get a few million bucks" product idea is a graphical awk application. There's a huge market of people doing CSV stuff in Excel despite the fact that Excel is fucking awful at that work, because none of the open source-y people can figure out how to make a usable UI.



  • @blakeyrat said in Bash script help (because I suck at it):

    One of my "if I even get a few million bucks" product idea is a graphical awk application. There's a huge market of people doing CSV stuff in Excel despite the fact that Excel is fucking awful at that work, because none of the open source-y people can figure out how to make a usable UI.

    Why do you need a few million bucks for that? It's a good idea. I know your time is valuable and you probably have things you want to do more than extra work, but maybe you should just get it done in a few work-weeks and start selling it. You know your tools well enough.



  • @Captain It would take more than a few work weeks to just begin designing the UI.

    Sure if I wanted to make it really shitty and awful, I could probably shit-out something very quickly.



  • @Onyx Same logic, pure bash, no additional interpreter:

    #!/bin/bash
    mintime=3600
    while IFS= read line
    do
            set -- $line
            case "$line" in
            *last\ was\ *)
                    member=$1
                    line=${line##*last was }
                    time=${line%% *}
                    if ((time > mintime))
                    then
                            echo "$time > $mintime for member $member in queue $queue"
                    fi
                    ;;
            \ *)
                    ;;
            *)
                    queue=$1
                    ;;
            esac
    done
    

    Wizardly features used:

    IFS= before read makes the read builtin operate with its IFS (input field separator) environment variable cleared out, which makes it treat the entire input line (including leading and trailing spaces) as a single word and assign the whole thing to line. Doing the assignment as a command prefix in this way makes its effect local to that command line only.

    set -- $line :wtf:?

    If you do help set in a bash terminal, you will see that set -- assigns "any remaining parameters" to the positional arguments. As used here, $line expands to the whole line just read by read; because it's not quoted on the set command line, it gets word-split on whitespace, and those words are treated as separate parameters and assigned to $1, $2, $3... in turn. Note that as well as being word split it will be processed for globs, so if there's a chance that your input might contain * or ? or [] this will need to be suppressed with a few extra lines.

    ${var%%pattern} means the contents of var with the greediest possible match for pattern removed from the end. ${var##pattern} likewise but removed from the start.

    If you're going to embed this in a longer script and you have the text you want it to parse available as output from some other process, use whatever | while IFS= read line to pipe it into the while loop. If it's at rest in a file, use done </path/to/data.


  • BINNED

    @Captain said in Bash script help (because I suck at it):

    maybe you should just get it done in a few work-weeks and start selling it.

    I started building a tool for work once that would generate a regular expression for you, all the user had to do was mess with a few checkboxes and dropdowns. I abandoned it due to lack of time to polish it properly, but yeah, I think this is doable for a reasonable set of requirements.

    The end result doesn't have to be regex, of course, it could generate rules however you choose. Hell, that would be even easier, regex decomposition is trickier than just having something in JSON/XML/Whatever.

    Maybe I should dig that thing out, finishing it might be worthwhile.



  • @Onyx I think you guys are really underestimating how long quality software works.


  • Notification Spam Recipient

    @blakeyrat said in Bash script help (because I suck at it):

    quality software

    E_FILE_NOT_FOUND 🚎


  • Winner of the 2016 Presidential Election

    @blakeyrat Sounds interesting, but I suspect you have to be extremely careful to not end up with an either extremely enterprisey or extremely useless program. I also suspect that people have tried something similar before and failed.

    I don't think it's reasonable to circumvent the programming part altogether and let people "program" in the GUI. What I think might be useful is a manager-friendly IDE for an awk-like DSL, which visualizes the effects your program would have on an example file. Add a few language bindings to integrate that system into other application and you might have a useful product.



  • @asdf I was thinking more like a pipeline/filtering application, where you pull in the file, and build filters by dragging and dropping "smaller" filters into a flowcharty thing.



  • @asdf said in Bash script help (because I suck at it):

    Sounds interesting, but I suspect you have to be extremely careful to not end up with an either extremely enterprisey or extremely useless program.

    Tableau is successful and they tackle that problem (kind of) plus more. (They actually focus their product more on analysis and charting; two other things Excel is commonly used for but is actually pretty terrible at.)

    But Tableau required millions of dollars and years of effort to create, it wasn't some guy "doing a couple work weeks" as a side-job.



  • @blakeyrat But you said it yourself -- Tableau's scope is much larger than the awk front-end for CSV processing you described.



  • @Captain Yes? I don't see how that changes anything I said here.

    I don't even think I could pick which windowing library and programming language/environment to use in a couple work-weeks.



  • @Onyx said in Bash script help (because I suck at it):

    I started building a tool for work once that would generate a regular expression for you, all the user had to do was mess with a few checkboxes and dropdowns. I abandoned it due to lack of time to polish it properly, but yeah, I think this is doable for a reasonable set of requirements.


  • Banned

    @blakeyrat said in Bash script help (because I suck at it):

    @Captain It would take more than a few work weeks to just begin designing the UI.

    You know that side project don't have to follow your company's processes?


Log in to reply