sh script to encode IP camera frames


  • Notification Spam Recipient

    So, I have a dumb IP camera that can upload frames on movement detection. This results in a folder of (thousands of) files like so:

    0_1513829546622_c565d83a-e788-4e35-9c2e-9acd54acba4b-image.png

    I'd like to craft a script that can take groups of them and smush them into video, and then delete them.

    For this, I know I have available to use ffmpeg, bash and cron, but tying them all together is leaving me scratching my head.

    Any hints on how to start? I think I need to:

    • Generate groups of files, probably based on hour and minute
    • Splat the list into ffmpeg (somehow) and provide an output file
    • Delete the files
    • Run this hourly by cron

    Any good pointers?



  • ffmpeg -pattern_type glob -i '000DC5D6F204()_1_20171222*_*.jpg' 20171222.mp4 && rm 000DC5D6F204\(\)_1_20171222*_*.jpg
    

  • Notification Spam Recipient

    @ben_lubar Nifty. I think adding -r 3 at the beginning will let it slow the framerate down so the entire day doesn't go by in a minute, yes?

    How do I group it by hour?

    And can I overlay the filename for each frame?



  • @tsaukpaetra said in sh script to encode IP camera frames:

    @ben_lubar Nifty. I think adding -r 3 at the beginning will let it slow the framerate down so the entire day doesn't go by in a minute, yes?

    -r 3 will make it run at 3 FPS, assuming the container format and video codec support it.

    How do I group it by hour?

    Add another two digits to the filename prefix. You can generate those names with date '+[some percent signs and letters I guess]'

    And can I overlay the filename for each frame?


  • Notification Spam Recipient

    Ok, I've come up with:

    #!/bin/sh
    cd /mnt/Security/video/MouseCam/grabs
    export target=000DC5D6F204\(\)_1_`date -v-1H +%Y%m%d%H`
    export dest=`date -v-1H +%Y%m%d-%H`.mp4
    
    echo "Targeting $target"
    echo "Destination file: $dest"
    
    ffmpeg -r 3 -pattern_type glob -i "${target}*_*.jpg" ${dest} && rm ${target}*_*.jpg
    
    echo Done.
    

    Here's to hoping nothing messes up, because I'm sticking it in the crontab and coming back in the morning...



  • @tsaukpaetra I'm not sure whether the unquoted ${target} will cause a syntax error due to the parentheses.



  • @tsaukpaetra Better quote all variable substitutions you don't want to expand special characters from, just in case:

    target="000DC5D6F204()_1_`date -v-1H +%Y%m%d%H`"
    dest="`date -v-1H +%Y%m%d-%H`.mp4"
    ffmpeg -r 3 -pattern_type glob -i "${target}*_*.jpg" "${dest}" && rm "${target}"*_*.jpg
    

    Also, this command is for BSD date(1), right?


  • area_can


  • area_can


  • Notification Spam Recipient

    @aitap said in sh script to encode IP camera frames:

    @tsaukpaetra Better quote all variable substitutions you don't want to expand special characters from, just in case:

    target="000DC5D6F204()_1_`date -v-1H +%Y%m%d%H`"
    dest="`date -v-1H +%Y%m%d-%H`.mp4"
    ffmpeg -r 3 -pattern_type glob -i "${target}*_*.jpg" "${dest}" && rm "${target}"*_*.jpg
    

    Also, this command is for BSD date(1), right?

    Yes.


  • Notification Spam Recipient

    @tsaukpaetra said in sh script to encode IP camera frames:

    nothing messes up

    Nothing messed up, because the damn thing didn't run.

    Apparently, my crontab entry was wrong:

    #Encode the mouse cam on the hour
    0	*	*	*	*	root /bin/sh /mnt/Security/video/MouseCam/grabs/splice.sh
    

    I'll figure that out later, for now Imma add a pre-process ffmpeg command to spudge the filename on the pictures.

    Revised script:

    #!/bin/sh
    
    cd /mnt/Security/video/MouseCam/grabs
    export target=000DC5D6F204\(\)_1_`date -v-1H +%Y%m%d%H`
    export dest=`date -v-1H +%Y%m%d-%H`.mp4
    
    echo "Targeting $target"
    echo "Destination file: $dest"
    
    echo "Stamping files..."
    
    for i in `ls ${target}*_*.jpg`; do 
        export time=$(echo $i | cut -d . -f 1)
        ffmpeg -loglevel panic -hide_banner -nostats -i $i -vf "drawtext=text='$time': fontcolor=white: box=1: boxcolor=black@0.7: boxborderw=2" -y $i
    done
    
    echo "Splicing..."
    
    ffmpeg -loglevel panic -hide_banner -nostats -r 3 -pattern_type glob -i "${target}*_*.jpg" ${dest} && rm ${target}*_*.jpg
    
    echo Done.
    


  • @tsaukpaetra said in sh script to encode IP camera frames:

    0 * * * * root /bin/sh /mnt/Security/video/MouseCam/grabs/splice.sh

    ITYM

    0 * * * * /mnt/Security/video/MouseCam/grabs/splice.sh
    

  • Notification Spam Recipient

    @ben_lubar said in sh script to encode IP camera frames:

    ITYM

    Modeled after all the other entries that seem to work:

    0_1513922849257_70d00594-4dfc-4b85-ad8c-a87797f17ade-image.png

    The logfile claims it ran:

    Dec 21 21:30:00 namp /usr/sbin/cron[81622]: (root) CMD (/bin/sh /usr/local/etc/rc.d/dansguardian start)
    Dec 21 22:00:00 namp /usr/sbin/cron[91195]: (root) CMD (/bin/sh /mnt/Security/video/MouseCam/grabs/splice.sh)
    Dec 21 22:00:00 namp /usr/sbin/cron[91197]: (root) CMD (/root/transmission_remove_finished2.sh >> /var/log/Sickrage_Remove.log)
    Dec 21 22:00:00 namp /usr/sbin/cron[91198]: (root) CMD (/usr/local/bin/webalizer)
    Dec 21 22:00:00 namp /usr/sbin/cron[91199]: (root) CMD (/usr/local/bin/sarg >> /tmp/cron.sarg.log )
    Dec 21 22:00:00 namp /usr/sbin/cron[91200]: (root) CMD (/usr/local/bin/squid-analyzer --debug >> /tmp/cron.squida.log)
    Dec 21 22:00:00 namp /usr/sbin/cron[91201]: (root) CMD (/bin/sh /usr/local/etc/rc.d/dansguardian start)
    Dec 21 22:00:00 namp /usr/sbin/cron[91202]: (root) CMD (newsyslog)
    Dec 21 22:30:00 namp /usr/sbin/cron[94593]: (root) CMD (/bin/sh /usr/local/etc/rc.d/dansguardian start)
    Dec 21 23:00:00 namp /usr/sbin/cron[95977]: (root) CMD (/bin/sh /mnt/Security/video/MouseCam/grabs/splice.sh)
    Dec 21 23:00:00 namp /usr/sbin/cron[95979]: (root) CMD (/root/transmission_remove_finished2.sh >> /var/log/Sickrage_Remove.log)
    Dec 21 23:00:00 namp /usr/sbin/cron[95980]: (root) CMD (/usr/local/bin/webalizer)
    Dec 21 23:00:00 namp /usr/sbin/cron[95981]: (root) CMD (/usr/local/bin/sarg >> /tmp/cron.sarg.log )
    Dec 21 23:00:00 namp /usr/sbin/cron[95982]: (root) CMD (/usr/local/bin/squid-analyzer --debug >> /tmp/cron.squida.log)
    Dec 21 23:00:00 namp /usr/sbin/cron[95983]: (root) CMD (/bin/sh /usr/local/etc/rc.d/dansguardian start)
    Dec 21 23:00:00 namp /usr/sbin/cron[95984]: (root) CMD (newsyslog)
    

    Perhaps I'll add some redirection like others to see if there's an error or something.


  • Notification Spam Recipient

    @tsaukpaetra said in sh script to encode IP camera frames:

    Perhaps I'll add some redirection like others to see if there's an error or something.

    Dec 22 00:01:00 namp /usr/sbin/cron[4480]: (root) CMD (/bin/sh /mnt/Security/video/MouseCam/grabs/splice.sh >> /mnt/Security/video/MouseCam/grabs/splice.log)
    Targeting 000DC5D6F204()_1_2017122123
    Destination file: 20171221-23.mp4
    Stamping files...
    Splicing...
    Done.
    

    It seems to believe it's working, but it's obviously not using the right directory, since nothing is... done?

    I'm at a loss. Only thing I can think of is to include the full path for everything?


  • Banned

    @tsaukpaetra is the file saved anywhere on the disk? If so, maybe try printing current directory.



  • @tsaukpaetra maybe redirect stderr, too (>>logfile 2>&1)? Do a set -x in the beginning so that sh would print all commands that it actually tries to run?



  • @tsaukpaetra The only gotcha I remember about cron is that it does not use your environment (obviously, since it's not run in your session), so if e.g. one of the commands you call is not in its PATH, or something like that, it will fail.



  • @aitap said in sh script to encode IP camera frames:

    @tsaukpaetra maybe redirect stderr, too (>>logfile 2>&1)? Do a set -x in the beginning so that sh would print all commands that it actually tries to run?

    As long as we're using set, how about adding e in there as well, so that if the cd or ffmpeg or rm commands fail, the whole script stops?

    Also, @Tsaukpaetra, doing this is very dangerous:

    for i in `ls ${target}*_*.jpg`; do 
    

    You should always quote variables unless you want them to expand, especially if the place the variable was defined required quoting or backslashes, and `ls [something]` is very rarely if ever something you want to use, because it will throw away the hard work you did quoting your stuff. Just change it to this:

    for i in "$target"*_*.jpg; do
    

    That also saves you a subshell and one call to ls that contains all the names of jpeg files as arguments. In cases where ls would act differently than echo such as a directory matching the glob, this will break, but it will break anyway in the ls version, and it actually breaks less this way.

    Quote the $i in the next two lines while you're at it, for the same reason.

    Also, just in case you didn't know, quoting things in shell scripts is not the same as quoting things in most other programming languages - you can put multiple quoted and unquoted strings together as long as there's no whitespace. My favorite example of this is '"'"' in the middle of a single quoted string to insert a single quote. ('\'' also works but it's not as fun.) Double quotes around a variable will prevent it from being expanded into multiple arguments except in the case of "$@" and "$*", where it expands the way you would expect (instead of making everything messy).

    And as long as I'm giving you shell tips you didn't ask for, you can quote subshells like "`foo`" and you can skip the set command by changing the shebang (#!/bin/sh) to #!/bin/sh -ex. That makes errors in commands terminate the script and sets it to display every command it executes. The shebang (named for # followed by !, "hash bang") can actually contain any command - if you put #!/bin/echo there, your shell will print the path to the command instead of running it, for example.

    It's also worth pointing out that $foo*.txt should never be used, "$foo"*.txt is replaced with a list of files whose names start with the value of $foo and end with .txt, and "$foo*.txt" is replaced with the value of $foo followed by literal *.txt as one argument.



  • @gÄ…ska said in sh script to encode IP camera frames:

    @tsaukpaetra is the file saved anywhere on the disk? If so, maybe try printing current directory.

    On Linux, if a cron command has output, it gets mailed to the user owning the cron entry by default (usually sudo mail will show you). I'm not sure how it works for BSD.



  • I thought that systemd stuff replaced cron? Hopefully with something not nearly as ass-tastic.


  • Java Dev

    @blakeyrat As far as I know, none of the systemd stuff comes with a GUI.



  • @pleegwat better is orthogonal to gui.

    The crap-oh-no daemon should have been shot when the eighties rolled around.



  • @gleemonk Srsly. All the crap in systemd that Linux users have been gnashing their teeth about for the last 4-5 years is just implementing all the features Windows 2000 had. It's not "destroying the soul of Linux!" it's "dragging it kicking and screaming 15 years into the future".


  • Notification Spam Recipient

    @ben_lubar said in sh script to encode IP camera frames:

    @aitap said in sh script to encode IP camera frames:

    @tsaukpaetra maybe redirect stderr, too (>>logfile 2>&1)? Do a set -x in the beginning so that sh would print all commands that it actually tries to run?

    As long as we're using set, how about adding e in there as well, so that if the cd or ffmpeg or rm commands fail, the whole script stops?

    Sure.

    @ben_lubar said in sh script to encode IP camera frames:

    Also, @Tsaukpaetra, doing this is very dangerous:

    for i in `ls ${target}*_*.jpg`; do 
    

    You should always quote variables unless you want them to expand, especially if the place the variable was defined required quoting or backslashes, and `ls [something]` is very rarely if ever something you want to use, because it will throw away the hard work you did quoting your stuff. Just change it to this:

    for i in "$target"*_*.jpg; do
    

    🤷♂ It's not like I'm in a class teaching me these things. So my representive line should actually be:

    for i in /mnt/Security/video/MouseCam/grabs/"$target"*_*.jpg; do 
    

    ???

    That also saves you a subshell and one call to ls that contains all the names of jpeg files as arguments. In cases where ls would act differently than echo such as a directory matching the glob, this will break, but it will break anyway in the ls version, and it actually breaks less this way.

    TIL

    Quote the $i in the next two lines while you're at it, for the same reason.

    Also, just in case you didn't know, quoting things in shell scripts is not the same as quoting things in most other programming languages - you can put multiple quoted and unquoted strings together as long as there's no whitespace. My favorite example of this is '"'"' in the middle of a single quoted string to insert a single quote. ('\'' also works but it's not as fun.) Double quotes around a variable will prevent it from being expanded into multiple arguments except in the case of "$@" and "$*", where it expands the way you would expect (instead of making everything messy).

    And as long as I'm giving you shell tips you didn't ask for, you can quote subshells like "`foo`" and you can skip the set command by changing the shebang (#!/bin/sh) to #!/bin/sh -ex. That makes errors in commands terminate the script and sets it to display every command it executes. The shebang (named for # followed by !, "hash bang") can actually contain any command - if you put #!/bin/echo there, your shell will print the path to the command instead of running it, for example.

    It's also worth pointing out that $foo*.txt should never be used, "$foo"*.txt is replaced with a list of files whose names start with the value of $foo and end with .txt, and "$foo*.txt" is replaced with the value of $foo followed by literal *.txt as one argument.

    I'm going to nod my head pretending I understood all that...

    So, me revised version should look like this????

    #!/bin/sh
    
    set -x
    
    cd /mnt/Security/video/MouseCam/grabs
    export target=000DC5D6F204\(\)_1_`date -v-1H +%Y%m%d%H`
    export dest=`date -v-1H +%Y%m%d-%H`.mp4
    
    echo "Targeting $target"
    echo "Destination file: $dest"
    
    echo "Stamping files..."
    
    for i in /mnt/Security/video/MouseCam/grabs/"$target"*_*.jpg; do 
        export time=$(echo $i | cut -d . -f 1)
        /usr/local/bin/ffmpeg -loglevel panic -hide_banner -nostats -i "/mnt/Security/video/MouseCam/grabs/""$i" -vf "drawtext=text='$time': fontcolor=white: box=1: boxcolor=black@0.7: boxborderw=2" -y "/mnt/Security/video/MouseCam/grabs/""$i"
    done
    
    echo "Splicing..."
    
    /usr/local/bin/ffmpeg -loglevel info -hide_banner -nostats -r 3 -pattern_type glob -i "/mnt/Security/video/MouseCam/grabs/""${target}""*_*.jpg" "/mnt/Security/video/MouseCam/grabs/""${dest}" && rm "/mnt/Security/video/MouseCam/grabs/""${target}"*_*.jpg"
    
    echo Done.
    

    Also:

    root@namp:/mnt/Security/video/MouseCam/grabs # mail
    No mail for root
    

    Apparently cron wasn't set up to send mail on this system. 🤷♂



  • @tsaukpaetra said in sh script to encode IP camera frames:

    export time=$(echo $i | cut -d . -f 1)
    

    You missed quoting $i here, which might cause problems. I'd have to look up what parentheses mean in shell scripts.



  • @tsaukpaetra said in sh script to encode IP camera frames:

    rm "/mnt/Security/video/MouseCam/grabs/""${target}"*_*.jpg"
    

    There's a stray quote at the end of the line, which definitely will cause problems.


  • Notification Spam Recipient

    @ben_lubar said in sh script to encode IP camera frames:

    cause problems.

    Oh god, literally everything is broken now!

    root@namp:/mnt/Security/video/MouseCam/grabs # ./splice.sh
    + cd /mnt/Security/video/MouseCam/grabs
    + date -v-1H +%Y%m%d%H
    + export 'target=000DC5D6F204()_1_2017122211'
    + date -v-1H +%Y%m%d-%H
    + export dest=20171222-11.mp4
    + echo 'Targeting 000DC5D6F204()_1_2017122211'
    Targeting 000DC5D6F204()_1_2017122211
    + echo 'Destination file: 20171222-11.mp4'
    Destination file: 20171222-11.mp4
    + echo 'Stamping files...'
    Stamping files...
    + echo '/mnt/Security/video/MouseCam/grabs/000DC5D6F204()_1_2017122211*_*.jpg'
    + cut -d . -f 1
    + export 'time=/mnt/Security/video/MouseCam/grabs/000DC5D6F204()_1_2017122211*_*'
    + /usr/local/bin/ffmpeg -loglevel panic -hide_banner -nostats -i '/mnt/Security/video/MouseCam/grabs//mnt/Security/video/MouseCam/grabs/000DC5D6F204()_1_2017122211*_*.jpg' -vf 'drawtext=text='\''/mnt/Security/video/MouseCam/grabs/000DC5D6F204()_1_2017122211*_*'\'': fontcolor=white: box=1: boxcolor=black@0.7: boxborderw=2' -y '/mnt/Security/video/MouseCam/grabs//mnt/Security/video/MouseCam/grabs/000DC5D6F204()_1_2017122211*_*.jpg'
    + echo Splicing...
    Splicing...
    ./splice.sh: 21: Syntax error: Unterminated quoted string
    

  • Java Dev

    @tsaukpaetra For that time=, you may want to try basename $i .jpg



  • If you'd given up on Bash and just written a quick and dirty program in C#, you'd be done by now. Long past done by now.


  • Considered Harmful

    @blakeyrat Or Powershell. 🛒


  • Notification Spam Recipient

    @blakeyrat said in sh script to encode IP camera frames:

    If you'd given up on Bash and just written a quick and dirty program in C#, you'd be done by now. Long past done by now.

    I'm definitely not interested in trying to get mono working on this BSD jail. Thanks for the tip though!


  • Notification Spam Recipient

    @pleegwat said in sh script to encode IP camera frames:

    @tsaukpaetra For that time=, you may want to try basename $i .jpg

    I blame SO. Seems like it would work though. Will need to wait for five minutes though to it can try it against this next set.


  • Java Dev

    @tsaukpaetra I'd look at a scripting language with ffmpeg bindings. Perl or python may have one?



  • @tsaukpaetra Python's decent at file operations and probably already installed.

    My real point is just: Bash sucks for doing any real work.


  • Notification Spam Recipient

    @pleegwat said in sh script to encode IP camera frames:

    @tsaukpaetra I'd look at a scripting language with ffmpeg bindings. Perl or python may have one?

    But... I'm practically done with the script? Only thing is getting it to execute properly when (apparently) run without an environment.

    @blakeyrat said in sh script to encode IP camera frames:

    @tsaukpaetra Python's decent at file operations and probably already installed.

    I know even less python than I do sh.

    @blakeyrat said in sh script to encode IP camera frames:

    My real point is just: Bash sucks for doing any real work.

    It's not really doing any work. Unless looping through a set of pattern-matched files (twice, because ffmpeg) is "real work"...

    And, I'm not using bash, I'm using sh as indicated by the shebang...


  • Java Dev

    @tsaukpaetra Yeah, I mean, if you care. In this case it probably doesn't, since you're not really running into the big weaknesses here:

    • You don't get hostile file names
    • You don't really care about error handling
    • Performance is not an objective.

    All three of those, while definitely not impossible in bash, quickly make your code a lot trickier.



  • @pleegwat Especially the first one, which is pretty much impossible.


  • Notification Spam Recipient

    @pleegwat said in sh script to encode IP camera frames:

    Yeah, I mean, if you care.

    If I was to expand this into a more useful thing, I probably would.

    As it stands, this is well on the way to becoming a Mission Critical App That Must Not Be Touched Or Else businesshome process!


  • Notification Spam Recipient

    @tsaukpaetra said in sh script to encode IP camera frames:

    try it against this next set.

    Almost worked!

    Here's version 0.6 RC1:

    #!/bin/sh
    
    #set -x
    
    hago='1'
    [ -n "$1" ] && hago=$1
    echo $hago
    
    cd /mnt/Security/video/MouseCam/grabs
    export target=000DC5D6F204\(\)_1_`date -v-"${hago}"H +%Y%m%d%H`
    export dest=`date -v-"${hago}"H +%Y%m%d-%H`.mp4
    
    echo "Targeting $target"
    echo "Destination file: $dest"
    
    echo "Stamping files..."
    
    for i in `ls /mnt/Security/video/MouseCam/grabs/${target}*_*.jpg`; do 
        export time=`basename $i .jpg`
        /usr/local/bin/ffmpeg -loglevel panic -hide_banner -nostats -i $i -vf "drawtext=text='$time': fontcolor=white: box=1: boxcolor=black@0.7: boxborderw=2" -y $i
    done
    
    echo "Splicing..."
    
    /usr/local/bin/ffmpeg -loglevel info -hide_banner -nostats -r 3 -pattern_type glob -i "/mnt/Security/video/MouseCam/grabs/${target}*_*.jpg" /mnt/Security/video/MouseCam/grabs/${dest} && rm /mnt/Security/video/MouseCam/grabs/${target}*_*.jpg
    
    echo Done.
    

    If I was feeling adventurous I'd probably look to see if I would prevent running the second ffmpeg command if there weren't any pictures for that hour, but that's not really important.



  • It always hurts to agree with blakey, but I was going to suggest the same thing. Python would be far better at handling the file names and other string and array operations, which are a real pain in the ass to do in any shell scripting language. Even if there isn't an ffmpeg library for Python, you can put together the correct command line and then use subprocess to run the ffmpeg command in a shell.


  • Notification Spam Recipient

    @dragnslcr Thanks, but as I mentioned, I'm really not doing anything that really warrants using a whole new language (that, again, I'm not interested in learning a new language for this).

    I'm not doing any string magic more complicated than concatenation, I'm not handling arrays, I'm not doing maths, I'm not making http requests, I'm not dynamically generating objects, I'm not making a reusable module, I'm not doing anything that (in my opinion) deserves loading up a whole bang-wiz environment.

    I'm directory-listing and shoving that list through a loop to stamp a string of text via ffmpeg, and then using ffmpeg to string the images together into a video.

    As demonstrated, this is easily within the realm of a shell script (hell, that's practically what they were made for!), so going through the trouble of invoking another heavy-duty language just for this? I can't justify it.

    It seems you and Blakeyrat are doing the equivalent of telling me:

    Oh, you want to do a mail-merge? Well you're gonna want a database, I recommend postgre, use that to store your fields in a table. Oh, you're using Word? Well, you'll need an interop library (if it doesn't exist, it should be easy enough to make a call to AutoHotKey). Make sure you have a good mailto: handler, Outlook doesn't like you sending emails programmatically.

    No. I need a simple CSV file that I can just send to Word that I can use the built-in Mail-Merge feature. I don't need cruft and extra stuff for this small project that probably (hopefully!) won't be needed in a few weeks once we destroy this rat.


  • Notification Spam Recipient

    Status: Looks fully functional now, and the thing I added so it will be able to go further back in time than an hour seems to be working too.

    Mission success, I guess.


  • Java Dev

    @blakeyrat said in sh script to encode IP camera frames:

    @pleegwat Especially the first one, which is pretty much impossible.

    Well, yes, filenames can get pretty evil in unix. However, that does not matter if you know in advance what filenames you are actually going to get. That's why I mention hostile file names - if you don't know in advance where your file names are coming from, this is a much more serious problem. I wouldn't use shell code like this in production if there was a snowflake's chance in hell that an external actor could influence my file names.



  • @tsaukpaetra said in sh script to encode IP camera frames:

    It seems you and Blakeyrat are doing the equivalent of telling me:

    You know nothing about me.


  • Notification Spam Recipient

    @blakeyrat said in sh script to encode IP camera frames:

    @tsaukpaetra said in sh script to encode IP camera frames:

    It seems you and Blakeyrat are doing the equivalent of telling me:

    You know nothing about me.

    When did I say I know anything about you? What are you trying to say? I can't read your mind!