sh script to encode IP camera frames
-
So, I have a dumb IP camera that can upload frames on movement detection. This results in a folder of (thousands of) files like so:
I'd like to craft a script that can take groups of them and smush them into video, and then delete them.
For this, I know I have available to use ffmpeg, bash and cron, but tying them all together is leaving me scratching my head.
Any hints on how to start? I think I need to:
- Generate groups of files, probably based on hour and minute
- Splat the list into ffmpeg (somehow) and provide an output file
- Delete the files
- Run this hourly by cron
Any good pointers?
-
ffmpeg -pattern_type glob -i '000DC5D6F204()_1_20171222*_*.jpg' 20171222.mp4 && rm 000DC5D6F204\(\)_1_20171222*_*.jpg
-
@ben_lubar Nifty. I think adding
-r 3
at the beginning will let it slow the framerate down so the entire day doesn't go by in a minute, yes?How do I group it by hour?
And can I overlay the filename for each frame?
-
@tsaukpaetra said in sh script to encode IP camera frames:
@ben_lubar Nifty. I think adding
-r 3
at the beginning will let it slow the framerate down so the entire day doesn't go by in a minute, yes?-r 3
will make it run at 3 FPS, assuming the container format and video codec support it.How do I group it by hour?
Add another two digits to the filename prefix. You can generate those names with
date '+[some percent signs and letters I guess]'
And can I overlay the filename for each frame?
-
Ok, I've come up with:
#!/bin/sh cd /mnt/Security/video/MouseCam/grabs export target=000DC5D6F204\(\)_1_`date -v-1H +%Y%m%d%H` export dest=`date -v-1H +%Y%m%d-%H`.mp4 echo "Targeting $target" echo "Destination file: $dest" ffmpeg -r 3 -pattern_type glob -i "${target}*_*.jpg" ${dest} && rm ${target}*_*.jpg echo Done.
Here's to hoping nothing messes up, because I'm sticking it in the crontab and coming back in the morning...
-
@tsaukpaetra I'm not sure whether the unquoted
${target}
will cause a syntax error due to the parentheses.
-
@tsaukpaetra Better quote all variable substitutions you don't want to expand special characters from, just in case:
target="000DC5D6F204()_1_`date -v-1H +%Y%m%d%H`" dest="`date -v-1H +%Y%m%d-%H`.mp4" ffmpeg -r 3 -pattern_type glob -i "${target}*_*.jpg" "${dest}" && rm "${target}"*_*.jpg
Also, this command is for BSD date(1), right?
-
@tsaukpaetra I think encoding with MJPEG might suit your needs: https://video.stackexchange.com/questions/7903/how-to-losslessly-encode-a-jpg-image-sequence-to-a-video-in-ffmpeg
-
-
@aitap said in sh script to encode IP camera frames:
@tsaukpaetra Better quote all variable substitutions you don't want to expand special characters from, just in case:
target="000DC5D6F204()_1_`date -v-1H +%Y%m%d%H`" dest="`date -v-1H +%Y%m%d-%H`.mp4" ffmpeg -r 3 -pattern_type glob -i "${target}*_*.jpg" "${dest}" && rm "${target}"*_*.jpg
Also, this command is for BSD date(1), right?
Yes.
-
@tsaukpaetra said in sh script to encode IP camera frames:
nothing messes up
Nothing messed up, because the damn thing didn't run.
Apparently, my crontab entry was wrong:
#Encode the mouse cam on the hour 0 * * * * root /bin/sh /mnt/Security/video/MouseCam/grabs/splice.sh
I'll figure that out later, for now Imma add a pre-process ffmpeg command to spudge the filename on the pictures.
Revised script:
#!/bin/sh cd /mnt/Security/video/MouseCam/grabs export target=000DC5D6F204\(\)_1_`date -v-1H +%Y%m%d%H` export dest=`date -v-1H +%Y%m%d-%H`.mp4 echo "Targeting $target" echo "Destination file: $dest" echo "Stamping files..." for i in `ls ${target}*_*.jpg`; do export time=$(echo $i | cut -d . -f 1) ffmpeg -loglevel panic -hide_banner -nostats -i $i -vf "drawtext=text='$time': fontcolor=white: box=1: boxcolor=black@0.7: boxborderw=2" -y $i done echo "Splicing..." ffmpeg -loglevel panic -hide_banner -nostats -r 3 -pattern_type glob -i "${target}*_*.jpg" ${dest} && rm ${target}*_*.jpg echo Done.
-
@tsaukpaetra said in sh script to encode IP camera frames:
0 * * * * root /bin/sh /mnt/Security/video/MouseCam/grabs/splice.sh
ITYM
0 * * * * /mnt/Security/video/MouseCam/grabs/splice.sh
-
@ben_lubar said in sh script to encode IP camera frames:
ITYM
Modeled after all the other entries that seem to work:
The logfile claims it ran:
Dec 21 21:30:00 namp /usr/sbin/cron[81622]: (root) CMD (/bin/sh /usr/local/etc/rc.d/dansguardian start) Dec 21 22:00:00 namp /usr/sbin/cron[91195]: (root) CMD (/bin/sh /mnt/Security/video/MouseCam/grabs/splice.sh) Dec 21 22:00:00 namp /usr/sbin/cron[91197]: (root) CMD (/root/transmission_remove_finished2.sh >> /var/log/Sickrage_Remove.log) Dec 21 22:00:00 namp /usr/sbin/cron[91198]: (root) CMD (/usr/local/bin/webalizer) Dec 21 22:00:00 namp /usr/sbin/cron[91199]: (root) CMD (/usr/local/bin/sarg >> /tmp/cron.sarg.log ) Dec 21 22:00:00 namp /usr/sbin/cron[91200]: (root) CMD (/usr/local/bin/squid-analyzer --debug >> /tmp/cron.squida.log) Dec 21 22:00:00 namp /usr/sbin/cron[91201]: (root) CMD (/bin/sh /usr/local/etc/rc.d/dansguardian start) Dec 21 22:00:00 namp /usr/sbin/cron[91202]: (root) CMD (newsyslog) Dec 21 22:30:00 namp /usr/sbin/cron[94593]: (root) CMD (/bin/sh /usr/local/etc/rc.d/dansguardian start) Dec 21 23:00:00 namp /usr/sbin/cron[95977]: (root) CMD (/bin/sh /mnt/Security/video/MouseCam/grabs/splice.sh) Dec 21 23:00:00 namp /usr/sbin/cron[95979]: (root) CMD (/root/transmission_remove_finished2.sh >> /var/log/Sickrage_Remove.log) Dec 21 23:00:00 namp /usr/sbin/cron[95980]: (root) CMD (/usr/local/bin/webalizer) Dec 21 23:00:00 namp /usr/sbin/cron[95981]: (root) CMD (/usr/local/bin/sarg >> /tmp/cron.sarg.log ) Dec 21 23:00:00 namp /usr/sbin/cron[95982]: (root) CMD (/usr/local/bin/squid-analyzer --debug >> /tmp/cron.squida.log) Dec 21 23:00:00 namp /usr/sbin/cron[95983]: (root) CMD (/bin/sh /usr/local/etc/rc.d/dansguardian start) Dec 21 23:00:00 namp /usr/sbin/cron[95984]: (root) CMD (newsyslog)
Perhaps I'll add some redirection like others to see if there's an error or something.
-
@tsaukpaetra said in sh script to encode IP camera frames:
Perhaps I'll add some redirection like others to see if there's an error or something.
Dec 22 00:01:00 namp /usr/sbin/cron[4480]: (root) CMD (/bin/sh /mnt/Security/video/MouseCam/grabs/splice.sh >> /mnt/Security/video/MouseCam/grabs/splice.log) Targeting 000DC5D6F204()_1_2017122123 Destination file: 20171221-23.mp4 Stamping files... Splicing... Done.
It seems to believe it's working, but it's obviously not using the right directory, since nothing is... done?
I'm at a loss. Only thing I can think of is to include the full path for everything?
-
@tsaukpaetra is the file saved anywhere on the disk? If so, maybe try printing current directory.
-
@tsaukpaetra maybe redirect stderr, too (
>>logfile 2>&1
)? Do aset -x
in the beginning so thatsh
would print all commands that it actually tries to run?
-
@tsaukpaetra The only gotcha I remember about cron is that it does not use your environment (obviously, since it's not run in your session), so if e.g. one of the commands you call is not in its PATH, or something like that, it will fail.
-
@aitap said in sh script to encode IP camera frames:
@tsaukpaetra maybe redirect stderr, too (
>>logfile 2>&1
)? Do aset -x
in the beginning so thatsh
would print all commands that it actually tries to run?As long as we're using
set
, how about addinge
in there as well, so that if thecd
orffmpeg
orrm
commands fail, the whole script stops?Also, @Tsaukpaetra, doing this is very dangerous:
for i in `ls ${target}*_*.jpg`; do
You should always quote variables unless you want them to expand, especially if the place the variable was defined required quoting or backslashes, and
`ls [something]`
is very rarely if ever something you want to use, because it will throw away the hard work you did quoting your stuff. Just change it to this:for i in "$target"*_*.jpg; do
That also saves you a subshell and one call to
ls
that contains all the names of jpeg files as arguments. In cases wherels
would act differently thanecho
such as a directory matching the glob, this will break, but it will break anyway in thels
version, and it actually breaks less this way.Quote the
$i
in the next two lines while you're at it, for the same reason.Also, just in case you didn't know, quoting things in shell scripts is not the same as quoting things in most other programming languages - you can put multiple quoted and unquoted strings together as long as there's no whitespace. My favorite example of this is
'"'"'
in the middle of a single quoted string to insert a single quote. ('\''
also works but it's not as fun.) Double quotes around a variable will prevent it from being expanded into multiple arguments except in the case of"$@"
and"$*"
, where it expands the way you would expect (instead of making everything messy).And as long as I'm giving you shell tips you didn't ask for, you can quote subshells like
"`foo`"
and you can skip theset
command by changing the shebang (#!/bin/sh
) to#!/bin/sh -ex
. That makese
rrors in commands terminate the script and sets it to display every command it ex
ecutes. The shebang (named for#
followed by!
, "hash bang") can actually contain any command - if you put#!/bin/echo
there, your shell will print the path to the command instead of running it, for example.It's also worth pointing out that
$foo*.txt
should never be used,"$foo"*.txt
is replaced with a list of files whose names start with the value of$foo
and end with.txt
, and"$foo*.txt"
is replaced with the value of$foo
followed by literal*.txt
as one argument.
-
@gÄ…ska said in sh script to encode IP camera frames:
@tsaukpaetra is the file saved anywhere on the disk? If so, maybe try printing current directory.
On Linux, if a cron command has output, it gets mailed to the user owning the cron entry by default (usually
sudo mail
will show you). I'm not sure how it works for BSD.
-
I thought that systemd stuff replaced cron? Hopefully with something not nearly as ass-tastic.
-
@blakeyrat As far as I know, none of the systemd stuff comes with a GUI.
-
@pleegwat better is orthogonal to gui.
The crap-oh-no daemon should have been shot when the eighties rolled around.
-
@gleemonk Srsly. All the crap in systemd that Linux users have been gnashing their teeth about for the last 4-5 years is just implementing all the features Windows 2000 had. It's not "destroying the soul of Linux!" it's "dragging it kicking and screaming 15 years into the future".
-
@ben_lubar said in sh script to encode IP camera frames:
@aitap said in sh script to encode IP camera frames:
@tsaukpaetra maybe redirect stderr, too (
>>logfile 2>&1
)? Do aset -x
in the beginning so thatsh
would print all commands that it actually tries to run?As long as we're using
set
, how about addinge
in there as well, so that if thecd
orffmpeg
orrm
commands fail, the whole script stops?Sure.
@ben_lubar said in sh script to encode IP camera frames:
Also, @Tsaukpaetra, doing this is very dangerous:
for i in `ls ${target}*_*.jpg`; do
You should always quote variables unless you want them to expand, especially if the place the variable was defined required quoting or backslashes, and
`ls [something]`
is very rarely if ever something you want to use, because it will throw away the hard work you did quoting your stuff. Just change it to this:for i in "$target"*_*.jpg; do
It's not like I'm in a class teaching me these things. So my representive line should actually be:
for i in /mnt/Security/video/MouseCam/grabs/"$target"*_*.jpg; do
???
That also saves you a subshell and one call to
ls
that contains all the names of jpeg files as arguments. In cases wherels
would act differently thanecho
such as a directory matching the glob, this will break, but it will break anyway in thels
version, and it actually breaks less this way.TIL
Quote the
$i
in the next two lines while you're at it, for the same reason.Also, just in case you didn't know, quoting things in shell scripts is not the same as quoting things in most other programming languages - you can put multiple quoted and unquoted strings together as long as there's no whitespace. My favorite example of this is
'"'"'
in the middle of a single quoted string to insert a single quote. ('\''
also works but it's not as fun.) Double quotes around a variable will prevent it from being expanded into multiple arguments except in the case of"$@"
and"$*"
, where it expands the way you would expect (instead of making everything messy).And as long as I'm giving you shell tips you didn't ask for, you can quote subshells like
"`foo`"
and you can skip theset
command by changing the shebang (#!/bin/sh
) to#!/bin/sh -ex
. That makese
rrors in commands terminate the script and sets it to display every command it ex
ecutes. The shebang (named for#
followed by!
, "hash bang") can actually contain any command - if you put#!/bin/echo
there, your shell will print the path to the command instead of running it, for example.It's also worth pointing out that
$foo*.txt
should never be used,"$foo"*.txt
is replaced with a list of files whose names start with the value of$foo
and end with.txt
, and"$foo*.txt"
is replaced with the value of$foo
followed by literal*.txt
as one argument.I'm going to nod my head pretending I understood all that...
So, me revised version should look like this????
#!/bin/sh set -x cd /mnt/Security/video/MouseCam/grabs export target=000DC5D6F204\(\)_1_`date -v-1H +%Y%m%d%H` export dest=`date -v-1H +%Y%m%d-%H`.mp4 echo "Targeting $target" echo "Destination file: $dest" echo "Stamping files..." for i in /mnt/Security/video/MouseCam/grabs/"$target"*_*.jpg; do export time=$(echo $i | cut -d . -f 1) /usr/local/bin/ffmpeg -loglevel panic -hide_banner -nostats -i "/mnt/Security/video/MouseCam/grabs/""$i" -vf "drawtext=text='$time': fontcolor=white: box=1: boxcolor=black@0.7: boxborderw=2" -y "/mnt/Security/video/MouseCam/grabs/""$i" done echo "Splicing..." /usr/local/bin/ffmpeg -loglevel info -hide_banner -nostats -r 3 -pattern_type glob -i "/mnt/Security/video/MouseCam/grabs/""${target}""*_*.jpg" "/mnt/Security/video/MouseCam/grabs/""${dest}" && rm "/mnt/Security/video/MouseCam/grabs/""${target}"*_*.jpg" echo Done.
Also:
root@namp:/mnt/Security/video/MouseCam/grabs # mail No mail for root
Apparently cron wasn't set up to send mail on this system.
-
@tsaukpaetra said in sh script to encode IP camera frames:
export time=$(echo $i | cut -d . -f 1)
You missed quoting
$i
here, which might cause problems. I'd have to look up what parentheses mean in shell scripts.
-
@tsaukpaetra said in sh script to encode IP camera frames:
rm "/mnt/Security/video/MouseCam/grabs/""${target}"*_*.jpg"
There's a stray quote at the end of the line, which definitely will cause problems.
-
@ben_lubar said in sh script to encode IP camera frames:
cause problems.
Oh god, literally everything is broken now!
root@namp:/mnt/Security/video/MouseCam/grabs # ./splice.sh + cd /mnt/Security/video/MouseCam/grabs + date -v-1H +%Y%m%d%H + export 'target=000DC5D6F204()_1_2017122211' + date -v-1H +%Y%m%d-%H + export dest=20171222-11.mp4 + echo 'Targeting 000DC5D6F204()_1_2017122211' Targeting 000DC5D6F204()_1_2017122211 + echo 'Destination file: 20171222-11.mp4' Destination file: 20171222-11.mp4 + echo 'Stamping files...' Stamping files... + echo '/mnt/Security/video/MouseCam/grabs/000DC5D6F204()_1_2017122211*_*.jpg' + cut -d . -f 1 + export 'time=/mnt/Security/video/MouseCam/grabs/000DC5D6F204()_1_2017122211*_*' + /usr/local/bin/ffmpeg -loglevel panic -hide_banner -nostats -i '/mnt/Security/video/MouseCam/grabs//mnt/Security/video/MouseCam/grabs/000DC5D6F204()_1_2017122211*_*.jpg' -vf 'drawtext=text='\''/mnt/Security/video/MouseCam/grabs/000DC5D6F204()_1_2017122211*_*'\'': fontcolor=white: box=1: boxcolor=black@0.7: boxborderw=2' -y '/mnt/Security/video/MouseCam/grabs//mnt/Security/video/MouseCam/grabs/000DC5D6F204()_1_2017122211*_*.jpg' + echo Splicing... Splicing... ./splice.sh: 21: Syntax error: Unterminated quoted string
-
@tsaukpaetra For that
time=
, you may want to trybasename $i .jpg
-
If you'd given up on Bash and just written a quick and dirty program in C#, you'd be done by now. Long past done by now.
-
@blakeyrat Or Powershell.
-
@blakeyrat said in sh script to encode IP camera frames:
If you'd given up on Bash and just written a quick and dirty program in C#, you'd be done by now. Long past done by now.
I'm definitely not interested in trying to get mono working on this BSD jail. Thanks for the tip though!
-
@pleegwat said in sh script to encode IP camera frames:
@tsaukpaetra For that
time=
, you may want to trybasename $i .jpg
I blame SO. Seems like it would work though. Will need to wait for five minutes though to it can try it against this next set.
-
@tsaukpaetra I'd look at a scripting language with ffmpeg bindings. Perl or python may have one?
-
@tsaukpaetra Python's decent at file operations and probably already installed.
My real point is just: Bash sucks for doing any real work.
-
@pleegwat said in sh script to encode IP camera frames:
@tsaukpaetra I'd look at a scripting language with ffmpeg bindings. Perl or python may have one?
But... I'm practically done with the script? Only thing is getting it to execute properly when (apparently) run without an environment.
@blakeyrat said in sh script to encode IP camera frames:
@tsaukpaetra Python's decent at file operations and probably already installed.
I know even less python than I do sh.
@blakeyrat said in sh script to encode IP camera frames:
My real point is just: Bash sucks for doing any real work.
It's not really doing any work. Unless looping through a set of pattern-matched files (twice, because ffmpeg) is "real work"...
And, I'm not using
bash
, I'm usingsh
as indicated by the shebang...
-
@tsaukpaetra Yeah, I mean, if you care. In this case it probably doesn't, since you're not really running into the big weaknesses here:
- You don't get hostile file names
- You don't really care about error handling
- Performance is not an objective.
All three of those, while definitely not impossible in bash, quickly make your code a lot trickier.
-
@pleegwat Especially the first one, which is pretty much impossible.
-
@pleegwat said in sh script to encode IP camera frames:
Yeah, I mean, if you care.
If I was to expand this into a more useful thing, I probably would.
As it stands, this is well on the way to becoming a Mission Critical App That Must Not Be Touched Or Else
businesshome process!
-
@tsaukpaetra said in sh script to encode IP camera frames:
try it against this next set.
Almost worked!
Here's version 0.6 RC1:
#!/bin/sh #set -x hago='1' [ -n "$1" ] && hago=$1 echo $hago cd /mnt/Security/video/MouseCam/grabs export target=000DC5D6F204\(\)_1_`date -v-"${hago}"H +%Y%m%d%H` export dest=`date -v-"${hago}"H +%Y%m%d-%H`.mp4 echo "Targeting $target" echo "Destination file: $dest" echo "Stamping files..." for i in `ls /mnt/Security/video/MouseCam/grabs/${target}*_*.jpg`; do export time=`basename $i .jpg` /usr/local/bin/ffmpeg -loglevel panic -hide_banner -nostats -i $i -vf "drawtext=text='$time': fontcolor=white: box=1: boxcolor=black@0.7: boxborderw=2" -y $i done echo "Splicing..." /usr/local/bin/ffmpeg -loglevel info -hide_banner -nostats -r 3 -pattern_type glob -i "/mnt/Security/video/MouseCam/grabs/${target}*_*.jpg" /mnt/Security/video/MouseCam/grabs/${dest} && rm /mnt/Security/video/MouseCam/grabs/${target}*_*.jpg echo Done.
If I was feeling adventurous I'd probably look to see if I would prevent running the second ffmpeg command if there weren't any pictures for that hour, but that's not really important.
-
It always hurts to agree with blakey, but I was going to suggest the same thing. Python would be far better at handling the file names and other string and array operations, which are a real pain in the ass to do in any shell scripting language. Even if there isn't an ffmpeg library for Python, you can put together the correct command line and then use
subprocess
to run the ffmpeg command in a shell.
-
@dragnslcr Thanks, but as I mentioned, I'm really not doing anything that really warrants using a whole new language (that, again, I'm not interested in learning a new language for this).
I'm not doing any string magic more complicated than concatenation, I'm not handling arrays, I'm not doing maths, I'm not making http requests, I'm not dynamically generating objects, I'm not making a reusable module, I'm not doing anything that (in my opinion) deserves loading up a whole bang-wiz environment.
I'm directory-listing and shoving that list through a loop to stamp a string of text via ffmpeg, and then using ffmpeg to string the images together into a video.
As demonstrated, this is easily within the realm of a shell script (hell, that's practically what they were made for!), so going through the trouble of invoking another heavy-duty language just for this? I can't justify it.
It seems you and Blakeyrat are doing the equivalent of telling me:
Oh, you want to do a mail-merge? Well you're gonna want a database, I recommend postgre, use that to store your fields in a table. Oh, you're using Word? Well, you'll need an interop library (if it doesn't exist, it should be easy enough to make a call to AutoHotKey). Make sure you have a good
mailto:
handler, Outlook doesn't like you sending emails programmatically.No. I need a simple CSV file that I can just send to Word that I can use the built-in Mail-Merge feature. I don't need cruft and extra stuff for this small project that probably (hopefully!) won't be needed in a few weeks once we destroy this rat.
-
Status: Looks fully functional now, and the thing I added so it will be able to go further back in time than an hour seems to be working too.
Mission success, I guess.
-
@blakeyrat said in sh script to encode IP camera frames:
@pleegwat Especially the first one, which is pretty much impossible.
Well, yes, filenames can get pretty evil in unix. However, that does not matter if you know in advance what filenames you are actually going to get. That's why I mention hostile file names - if you don't know in advance where your file names are coming from, this is a much more serious problem. I wouldn't use shell code like this in production if there was a snowflake's chance in hell that an external actor could influence my file names.
-
@tsaukpaetra said in sh script to encode IP camera frames:
It seems you and Blakeyrat are doing the equivalent of telling me:
You know nothing about me.
-
@blakeyrat said in sh script to encode IP camera frames:
@tsaukpaetra said in sh script to encode IP camera frames:
It seems you and Blakeyrat are doing the equivalent of telling me:
You know nothing about me.
When did I say I know anything about you? What are you trying to say? I can't read your mind!