The Horror of the Mover



  • I could have sworn I'd already posted this, but I went back and looked and couldn't find it ... I transcribed and sanitized on-the-fly, so any typo's are likely my fault. Not only is this ugly, but there was a (not-so?) subtle bug that went unseen for a number of years which was fixed last year-ish (and I didn't include the fix.) Can you find it?

    #!/bin/csh -f
    # Purpose: Copies and renames new input files to the appropriate
    #          input directories for APP. Compress the file then
    #          move compressed file to archive.

    set DATADIR = /production/data/input/SOURCE

    set APPDIR = /production/APP/data/input/
    set APPDIR1 = /production/APP/data/input1/
    set APPDIR2 = /production/APP/data/input2/

    set ARCHDIR = /archive/SOURCE/

    set LOGDIR = /production/logs/
    set LOGFILE = ${LOGDIR}/mover.log

    set DATE = `date`
    set MONTH = `date +%m`
    set YEAR = `date +%Y`
    set HOUR = `date +%H`

    cd ${DATADIR}

    unalias mv
    unalias rm

    while 1

      ls *.dat >&! /dev/null

      ## if($status != 0) then
      ##   exit
      ## endif

      if ($status == 0) then
        set MONTH = `date +%m`
        set HOUR  = `date +%H`
        set YEAR  = `date +%Y`

        foreach file (*.SOURCE)
          date '+%y/%m/%d %H:%M was the time of the last data pull' >! ${DATADIR}/mover.touch

          # make sure permissions are accessible
          # echo setting permissions on input files
          chmod 777 ${file}

          # copy the input files to the correct directories for APP
          echo "Copying File to APP Directories detail. ${file} " `date` >>! ${LOGFILE}
     
          switch ( $HOUR )
            case "01":
              cp ${file} ${APPDIR}
              mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
              echo "copying file for input detail"
              breaksw
            case "02":
              cp ${file} ${APPDIR1}
              mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
              echo "copying file for input detail 1"
              breaksw
            case "03":
              cp ${file} ${APPDIR2}
              mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
              echo "copying file for input detail 2"
              breaksw
            case "04":
              cp ${file} ${APPDIR}
              mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
              echo "copying file for input detail"
              breaksw
            case "05":
              cp ${file} ${APPDIR1}
              mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
              echo "copying file for input detail 1"
              breaksw
            case "06":
              cp ${file} ${APPDIR2}
              mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
              echo "copying file for input detail 2"
              breaksw
            case "07":
              cp ${file} ${APPDIR}
              mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
              echo "copying file for input detail"
              breaksw
            case "08":
              cp ${file} ${APPDIR1}
              mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
              echo "copying file for input detail 1"
              breaksw
            case "09":
              cp ${file} ${APPDIR2}
              mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
              echo "copying file for input detail 2"
              breaksw
            case "10":
              cp ${file} ${APPDIR}
              mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
              echo "copying file for input detail"
              breaksw
            case "11":
              cp ${file} ${APPDIR1}
              mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
              echo "copying file for input detail 1"
              breaksw
            case "12":
              cp ${file} ${APPDIR2}
              mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
              echo "copying file for input detail 2"
              breaksw
            case "13":
              cp ${file} ${APPDIR}
              mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
              echo "copying file for input detail"
              breaksw
            case "14":
              cp ${file} ${APPDIR1}
              mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
              echo "copying file for input detail 1"
              breaksw
            case "15":
              cp ${file} ${APPDIR2}
              mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
              echo "copying file for input detail 2"
              breaksw
            case "16":
              cp ${file} ${APPDIR}
              mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
              echo "copying file for input detail"
              breaksw
            case "17":
              cp ${file} ${APPDIR1}
              mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
              echo "copying file for input detail 1"
              breaksw
            case "18":
              cp ${file} ${APPDIR2}
              mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
              echo "copying file for input detail 2"
              breaksw
            case "19":
              cp ${file} ${APPDIR}
              mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
              echo "copying file for input detail"
              breaksw
            case "20":
              cp ${file} ${APPDIR1}
              mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
              echo "copying file for input detail 1"
              breaksw
            case "21":
              cp ${file} ${APPDIR2}
              mv ${APPDIR2}/${file} ${APPDIR2}/${file}.dat
              echo "copying file for input detail 2"
              breaksw
            case "22":
              cp ${file} ${APPDIR}
              mv ${APPDIR}/${file} ${APPDIR}/${file}.dat
              echo "copying file for input detail"
              breaksw
            case "23":
              cp ${file} ${APPDIR1}
              mv ${APPDIR1}/${file} ${APPDIR1}/${file}.dat
              echo "copying file for input detail 1"
              breaksw
     default:
       echo "Problem"
          endsw

          ## mv ${APPDIR}/${file} ${APPDIR}/${file}.dat

          # compress the files and move them to the archive directory
          echo "Compressing and moving input file" >>! ${LOGFILE}

          compress ${file}

          # we have to put the files in the directories based on month and year
          if !( -d ${ARCHDIR}${YEAR} ) then
            mkdir ${ARCHDIR}${YEAR}
          endif

          switch ( $MONTH )
            case "01":
              if !( -d {$ARCHDIR}${YEAR}/JAN ) then
                mkdir {$ARCHDIR}${YEAR}/JAN
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/JAN;breaksw
            case "02":
              if !( -d {$ARCHDIR}${YEAR}/FEB ) then
                mkdir {$ARCHDIR}${YEAR}/FEB
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/FEB;breaksw
            case "03":
              if !( -d {$ARCHDIR}${YEAR}/MAR ) then
                mkdir {$ARCHDIR}${YEAR}/MAR
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/MAR;breaksw
            case "04":
              if !( -d {$ARCHDIR}${YEAR}/APR ) then
                mkdir {$ARCHDIR}${YEAR}/APR
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/APR;breaksw
            case "05":
              if !( -d {$ARCHDIR}${YEAR}/MAY ) then
                mkdir {$ARCHDIR}${YEAR}/MAY
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/MAY;breaksW
            case "06":
              if !( -d {$ARCHDIR}${YEAR}/JUN ) then
                mkdir {$ARCHDIR}${YEAR}/JUN
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/JUN;breaksw
            case "07":
              if !( -d {$ARCHDIR}${YEAR}/JUL ) then
                mkdir {$ARCHDIR}${YEAR}/JUL
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/JUL;breaksw
            case "08":
              if !( -d {$ARCHDIR}${YEAR}/AUG ) then
                mkdir {$ARCHDIR}${YEAR}/AUG
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/AUG;breaksw
            case "09":
              if !( -d {$ARCHDIR}${YEAR}/SEP ) then
                mkdir {$ARCHDIR}${YEAR}/SEP
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/SEP;breaksw
            case "10":
              if !( -d {$ARCHDIR}${YEAR}/OCT ) then
                mkdir {$ARCHDIR}${YEAR}/OCT
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/OCT;breaksw
            case "11":
              if !( -d {$ARCHDIR}${YEAR}/NOV ) then
                mkdir {$ARCHDIR}${YEAR}/NOV
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/NOV;breaksw
            case "12":
              if !( -d {$ARCHDIR}${YEAR}/DEC ) then
                mkdir {$ARCHDIR}${YEAR}/DEC
              endif
              mv ${file}.Z {$ARCHDIR}${YEAR}/DEC;breaksw
            default:
              echo "There was an error generating the month and the files have been placed"
              echo "  in the " ${ARCHDIR} " directory "
              mv ${file}.Z ${ARCHDIR}
          endsw

        else
     
          if !( -d ${ARCHDIR}${YEAR}/INCOMPLETE ) then
            mkdir ${ARCHDIR}${YEAR}/INCOMPLETE
          endif
          mv ${file}.Z ${ARCHDIR}${YEAR}/INCOMPLETE;breaksw

        endif

        end

      endif

      sleep 50

    end



  • Looks ugly, but it also looks like a bash script and I've never seen a bash script that wasn't ugly.



  • This is way out of my specialty, but the first thing that popped out for me was the hour switch block.  It doesn't include midnight, right?



  • @mott555 said:

    Looks ugly, but it also looks like a bash script and I've never seen a bash script that wasn't ugly.
     

    What gave it away as a bash script?  Was it the line reading "#!/bin/csh" at the top?

     



  • but there was a (not-so?) subtle bug that went unseen for a number of years which was fixed last year-ish (and I didn't include the fix.) Can you find it?

    Was it "You forgot to write the log file at midnight" ? Or was it "You wrote a script in csh" ?



  • @zelmak said:

    I could have sworn I'd already posted this, but I went back and looked and couldn't find it ... I transcribed and sanitized on-the-fly, so any typo's are likely my fault. Not only is this ugly, but there was a (not-so?) subtle bug that went unseen for a number of years which was fixed last year-ish (and I didn't include the fix.) Can you find it?

    It's got a Wilson Pickett bug!

     



  • Is it that you're copying each file to the month? Looks like you copy file1.Z to JAN and then you'd copy file2.Z to JAN as well.

    but that's a quick look-see, it's lunch time and my vendor's just brought me a sandwich for doing alarm testing.



  • @Rootbeer said:

    @mott555 said:
    Looks ugly, but it also looks like a bash script and I've never seen a bash script that wasn't ugly.
    What gave it away as a bash script?  Was it the line reading "#!/bin/csh" at the top?
    Wrong meaning of the phrase "looks like", genius. He literally meant it looks like what it is, not that he's guessing that it's a bash script.

     



  • @Zylon said:

    @Rootbeer said:

    @mott555 said:
    Looks ugly, but it also looks like a bash script and I've never seen a bash script that wasn't ugly.
    What gave it away as a bash script?  Was it the line reading "#!/bin/csh" at the top?
    Wrong meaning of the phrase "looks like", genius. He literally meant it looks like what it is, not that he's guessing that it's a bash script.

     

    Except that it's NOT a bash script.

    Or am I just missing some really obvious irony in these comments?



  • Hell if I know. I don't do *nix.



  • @Zylon said:

    @Rootbeer said:

    @mott555 said:
    Looks ugly, but it also looks like a bash script and I've never seen a bash script that wasn't ugly.
    What gave it away as a bash script?  Was it the line reading "#!/bin/csh" at the top?
    Wrong meaning of the phrase "looks like", genius. He literally meant it looks like what it is, not that he's guessing that it's a bash script.

    If that's what he meant then he worded it confusingly.  That second clause, joined by "but", looks like it's going to be some kind of explanation for or justification of why it "looks ugly".  If it's not mott's assumption that it was a bash script, why would it be relevant that all bash scripts are ugly?  Why would anyone write "It looks ugly, but it also looks ugly, because it looks like something that is always ugly" on purpose?  The "It looks ugly, but it has to because it is an example of something that is always ugly" interpretation seemed more obvious to me.




  • @Zylon said:

    Hell if I know. I don't do *nix.

    Shit was SO csh.

     



  • @Zylon said:

    Hell if I know. I don't do *nix.

    It's a shell script, but not a bash script.

    #!/some_path/sh == bourne SHell. The "lowest common denominator", de facto, always-there *nix shell for a long time.
    #!/some_path/bash == Bourne Again SHell. Seems to be the most popular nowadays. Based on bourne shell, not surprisingly.
    #!/some_path/ksh == Korn SHell. Another bourne shell derivative that was popular in the 90s. #!/some_path/csh == C SHell. The one used in the OP. A shell with somewhat more C-like syntax than the bourne family. Next to plain ol' bourne, was almost a guarantee to be present back in the day.
    #!/some_path/tcsh == Some variation on csh that I know nothing about.

    Fascinating, wot?



  • I notice the switch only accounts for 23 hours out of the day.

    @zelmak said:

    echo "Problem"
    I should say so.



  • Looks like the entirety of the first switch statement can be replaced by an array and a modulus. The second switch statement with just an array for the months.
    Of course, if you want to be *really* sure it wont break anything you'd check that the numbers are in the same range first.



  • @zelmak said:

    #!/bin/rm -f

    FTFY

    @zelmak said:

        set MONTH = `date +%m`
        set HOUR  = `date +%H`
        set YEAR  = `date +%Y`

    This is a race condition, isn't it?


  • One might add that ksh was the standard shell on most Unixes, bash took over because it was the GNU version so became standard in the Linux world, and csh is an evil mess that should never have been spawned (and I still know people who use it every day, poor buggers).

     There's also zsh, whick is the Dvorak keyboard / Ron Paul of shells.

    Yes, I am this boring in real life too.



  • @fatbull said:

    @zelmak said:

    #!/bin/rm -f

    FTFY

    @zelmak said:

        set MONTH = `date +%m`
        set HOUR  = `date +%H`
        set YEAR  = `date +%Y`

    This is a race condition, isn't it?
    So that's why it omits case "00"!  It avoids the race!



  • @Iago said:

    One might add that ksh was the standard shell on most Unixes

    IIRC, sh was the only one that was guaranteed to always be present, at least up until Linux became popular. I think Linux still always has sh, but it seems it's often just a link to bash.

    There's also zsh, whick is the Dvorak keyboard / Ron Paul of shells.

    Yes, I am this boring in real life too.

    I can't believe I forgot zsh, my shell of choice. It's very similar to bash, but seems to have a few additional features



  • @jverd said:

    IIRC, sh was the only one that was guaranteed to always be present, at least up until Linux became popular. I think Linux still always has sh, but it seems it's often just a link to bash.

    It is indeed often a symlink to bash. I think it was Ubuntu which started symlinking it to dash instead (and broke a ton of poorly written scripts which started #!/bin/sh but assumed bash). And in some cases it's a symlink to busybox.



  • @jverd said:

    @Iago said:

    One might add that ksh was the standard shell on most Unixes

    IIRC, sh was the only one that was guaranteed to always be present, at least up until Linux became popular. I think Linux still always has sh, but it seems it's often just a link to bash.

    There's also zsh, whick is the Dvorak keyboard / Ron Paul of shells.

    Yes, I am this boring in real life too.

    I can't believe I forgot zsh, my shell of choice. It's very similar to bash, but seems to have a few additional features


    Actually, it is closer to ksh than to bash, and I love it. I am using maybe 5% of the features, but it's way more practical than bash. :)



  • @bannedfromcoding said:

    Actually, it is closer to ksh than to bash, and I love it. I am using maybe 5% of the features, but it's way more practical than bash. :)

    ^- the reason you'll never see usable software from the *nix crowd.



  • @blakeyrat said:

    @bannedfromcoding said:
    Actually, it is closer to ksh than to bash, and I love it. I am using maybe 5% of the features, but it's way more practical than bash. :)

    ^- the reason you'll never see usable software from the *nix crowd.


    I thought it's kinda obvious, but since a while I don't write software. ;)


  • Garbage Person

    @Iago said:

     There's also zsh, whick is the Dvorak keyboard / Ron Paul of shells.
    I'm not sure whether this comparison is more insulting to Dvorak keyboards, Ron Paul or zsh.



  • @Weng said:

    @Iago said:

     There's also zsh, whick is the Dvorak keyboard / Ron Paul of shells.
    I'm not sure whether this comparison is more insulting to Dvorak keyboards, Ron Paul or zsh.

    It simultaneously insults Dvorak keyboards more than Ron Paul, Ron Paul more than zsh, and zsh more than Dvorak keyboards!

     



  • Seeing as the thread is titled 'the horror of the mover'...



    set LOGDIR = /production/logs/

    set LOGFILE = ${LOGDIR}/mover.log




    How does *nix handle double slashes on a filename?



  • @cmccormick said:

    Seeing as the thread is titled 'the horror of the mover'...

    set LOGDIR = /production/logs/
    set LOGFILE = ${LOGDIR}/mover.log


    How does *nix handle double slashes on a filename?

    No problems at all in the middle of a pathname.  '//' at the front of a path is reserved and may have any special meaning the implementation wants to use it for, but in the middle it's just an empty path component.

     



  •  Leaving out the question of using csh at all, or the inadvisability of using shell scripts for any but the most trivial tasks (or where, for protability it's unavoidable - configure for example), since this script will break with file names containing spaces or some shell metacharacters,  I see the following:

    It's syntactically invalid - there's a switch block after the endsw in the second block. There's an endif when the script is in a foreach on the following line.

    It appears it's supposed to run every 50 seconds, so race conditions when the various date fields change aren't all that unlikely.

    Why would anyone use unalias rm, unalias mv instead of the safe approach of using full paths?

    Why would anyone set all the .SOURCE files to be executable? Or world writeable for that matter?

    If the system it's running on is running on a localtime including daylight savings, it will dump two hours worth of backups in one directory

    One wonders if there really is a need to copy the files into appdir then mv them as opposed to simply copying them to the final file name

     What happens between midnight and 1 AM?

     The script is very noisy with 'copying file for input detail ...' without actually giving useful information, like what file is being copied.

    The same quality of error reporting is shown by outputting 'Problem' when the copy to the app directory

    There's probably more wrong here. 

    But it makes a good case for code reviews, I'd hate to be the author facing my colleagues to explain this one

     

     

     



  • @jes said:

    It's syntactically invalid - there's a switch block after the endsw in the second block.

    No there isn't. 

    @jes said:

    There's an endif when the script is in a foreach on the following line.

    That thing you thought was a spurious switch block is in fact a spurious else..endif block, if you read it again.  The structure of the code looks like this:

    while 1
      if ($status == 0) then
        foreach file (*.SOURCE)
          [ ... stuff ... ]
          [ first switch block ]
          [ ... stuff ... ]
          [ second switch block ]
          [ spurious else..endif block]
        end # foreach end
      endif # if $status end
      sleep 50
    end # while 1 end 

     




  • The bug was, in-fact, the missing '00' hour check. But, even tho the fact that something odd had occurred was being logged for OVER SIX YEARS, no one noticed that we never had any data for that timeframe. When it was discovered (someone was doing some data-trending pulls from the DB and noticed that the 'zero-hour' had no data going ways back) the powers-that-be shrugged, said, put in the last six months worth, and we'll be okay. Yet, we still have data in the system dating back to 2003 so 'all the data is important' ... except when its not.

    The reason for the cp and then subsequent mv was to stave off (yet another?) race-condition ... where the file copy would be incomplete, but the program reading it would start reading it anyway. Since the input program only reads .dat files, the cp can complete, then the mv will make it 'readable' by the input program.

    The whole date thing? Meh.

    But why on earth would you go to such ends as to change a perfectly acceptable (and sortable) numeric month value and change it to the English abbreviation for it?

    And those hugemongous switch statements ... ugh ...

    And dumping all input from an hour into one directory? We have three 'data loaders'; one reads its own directory. So, one directory fills up, the others remain idle. Yay. Why do we have three again?

     


  • Discourse touched me in a no-no place

    @jes said:

    Why would anyone use unalias rm, unalias mv instead of the safe approach of using full paths?
    Because they want to use $PATH to search for them?


    The intention behind unalias is to remove the alias, not to fix which binary is being used. (rm is commonly aliased as 'rm -i' for example.)



  • @PJH said:


    The intention behind unalias is to remove the alias, not to fix which binary is being used. (rm is commonly aliased as 'rm -i' for example.)

    Which only helps if rm is aliased (as opposed to being a shell function or a script or executable ahead of /bin in the path). I can't imagine why the author would be unaliasing other than to make sure he knew exactly what the effect of executing rm would be, and the only way to do that is to just explicitly invoke /bin/rm. (And even that relies on an assumption, albeit a safer one that what you get with a simple unalias.)



  • @blakeyrat said:

    @bannedfromcoding said:
    Actually, it is closer to ksh than to bash, and I love it. I am using maybe 5% of the features, but it's way more practical than bash. :)

    ^- the reason you'll never see usable software from the *nix crowd.

    Taking "usable" in the context of software usability, which I believe is how you generally use it, as opposed to meaning "there are people who manage to use this software to accomplish its intended purpose," I still think you're looking at two effects, not a cause and effect. But I do agree that they're related. </pd>


  • Discourse touched me in a no-no place

    @jverd said:

    @PJH said:

    The intention behind unalias is to remove the alias, not
    to fix which binary is being used. (rm is commonly aliased as 'rm -i' for
    example.)
    Which only helps if rm is aliased
    Which is the intention.


    @jverd said:
    (as opposed to being a shell function or a script or executable ahead of /bin in
    the path). I can't imagine why the author would be unaliasing other than to make
    sure he knew exactly what the effect of executing rm would be, and the only way
    to do that is to just explicitly invoke /bin/rm. (And even that relies on an
    assumption, albeit a safer one that what you get with a simple unalias.)
    -i requires keyboard input. For an unattended script, you don't require someone sitting at the keyboard. unaliasing ensures that -i isn't on the command line.



  • Unaliasing doesn't ensure that isn't a "-i" on the command line.

    Even using the full path doesn't ensure that, there is no way to ensure it. But using the full path removes a lot of other ways the "-i" could get there.


  • Discourse touched me in a no-no place

    @Mcoder said:

    Unaliasing doesn't ensure that isn't a "-i" on the command line.

    Apart from aliasing, how else are you postulating a -i could get in the way?

    Even using the full path doesn't ensure that, there is no way to ensure it. But using the full path removes a lot of other ways the "-i" could get there.
    Seriously, I'm interested as to how, in a script, a stray -i (or any other bollocks an alias could introduce) could be introduced. And why fixing the path to the binary could remove it where an unalias couldn't.

    Assume, humour me, that the system in question isn't under 'attack'.

  • :belt_onion:

    @PJH said:

    Apart from aliasing, how else are you postulating a -i could get in the way?

    I am equally baffled. I think the postulate is "Nothing is for sure in Linux because Linux is hard; let's go shopping."


Log in to reply