Script/program for installing software - linux

OffByOne

I have some git repositories that contains maintenance scripts, their dependencies and configuration files. Does anyone know what I can use to install these scripts on the target server?

Constraints:

The scripts are not confined to a single top level directory (like /opt/scriptdir). Instead, some live in /usr/local/bin, some files should go to /etc and others might need to go to other places.
This means I would have to put everything in a .tar.gz file and untar it in the root directory of the target servers. I don't feel comfortable doing that.
Some files need "special treatment": just copying them to the live filesystem isn't enough.
I want to make it as easy as possible to do the Right Thing for my colleagues. They know their way around a command line (somewhat), but are not proficient enough to be able to fix things quickly if something goes wrong.
- There is a script that should be called by cron. Installing it means an entry should be added to the appropriate crontab. I'd like to make this part of the automated installation, instead of having to do this manually.
- One of these repositories contains a web app. It has an accompanying Apache config file, which should be installed. After installing that config file, Apache must be signaled to reload its configuration. I'd like this to be automated.
I'd like to have 1 command to take care of everything.
(Optional, but I'd really like to have this) Since the scripts live in under version control, I have a complete history.
Before copying a file to the live filesystem, I'd like to check if the existing file is identical to a version of that file in the git repo. If it's not, it means that someone changed the file on the live system without storing it in the repository. In that case, the existing file should not be overwritten, because I'd like to check the diff between what is on the live filesystem and what is in the repo and manually merge the two versions.

Solutions I thought of:

Write a Makefile in each repository to take care of copying the files to the correct location and running the pre- and post-installation steps where necessary.
Possible, but not really reusable (I'd have to write a new Makefile for each repo) and I'd have make syntax on me :/
Write something myself. At least my it's-not-really-make would have a sane(r) syntax ;)

Does anyone know of a decent solution?

Gurth

A Bash script maybe?

JBert

Do you have a team of sysadmins, and if so, do they already use something like Ansible or Salt?

Would thus need to be repeated often (e.g. to upgrade)?

What distributions are you targetting?

OffByOne

@Gurth said in Script/program for installing software - linux:

A Bash script maybe?

If I'm going to write something myself, it would be in Perl, but same difference :)

I was wondering if what I'm looking for exists already or if I have to write it myself.

Gurth

@OffByOne I suggested Bash mainly because it looks like most of your requirements are things that a user could do manually using standard terminal commands (copying files, adding a cron job, etc.) so the obvious solution is to just put those commands into a script.

OffByOne

@JBert said in Script/program for installing software - linux:

Do you have a team of sysadmins, and if so, do they already use something like Ansible or Salt?

Heh, no. I'm the only person in the company who's really fluent with linux. That's why ideally there would be a single command to execute and it would Do The Right Thing.

Would thus need to be repeated often (e.g. to upgrade)?

Depends on how often we discover bugs in the vendor provided software :)

The situation is this: we have some servers that are bought from an external company. These servers provide services we need for our business (sorry to be vague, I don't want to go into specifics in a public place). We have shell access (root) to these servers.
We have found some bugs recently in the software of the external company and can work around those with some shell scripts. The external company is fixing the bugs, but until we get those official fixes, we use those temporary workarounds to keep our customers happy.

Also: I'm writing a webservice, so my colleagues can use a webpage to run specific processes without having to learn arcane command line parameters. The webservice takes care of those.
If I have a way to install the fixes for vendor bugs, I can use the same mechanism to deploy my webservice (Perl script + dependencies and an Apache config file).

What distributions are you targetting?

Doesn't really matter. Packaging everything in a .deb or .rpm is overkill. Also, I would have to spend time learning how to create a .deb or .rpm and how to hook the pre/post scripts into it.

OffByOne

@Gurth said in Script/program for installing software - linux:

@OffByOne I suggested Bash mainly because it looks like most of your requirements are things that a user could do manually using standard terminal commands (copying files, adding a cron job, etc.) so the obvious solution is to just put those commands into a script.

Your suggestion is appreciated :)

What I really want to know is if something already exists, so I don't have to think about all the edge cases (and I'm sure I'd overlook some of those) if I write something myself.

"File on the live filesystem doesn't match any of the versions of the same file in the repository" is one of those edge cases. I'm sure there are others that I'm not thinking about.

Grunnen

Why not make a custom rpm/deb packge?

OffByOne

@Grunnen said in Script/program for installing software - linux:

Why not make a custom rpm/deb packge?

Because I would have to invest time to learn how to do that, which pitfalls to avoid, ...

Also, if I learn how to make an RPM, I won't be able to use it on Debian derivatives. If I learn how to make a DEB, I won't be able to use that on Redhat derivatives.
It's not portable enough to my liking.

Also also, I'm not even sure those package formats support all things I'm looking for (especially checking if what is currently in the live filesystem exists in the repo history somewhere, to prevent overwriting a file that has been changed on the filesystem itself by someone).

Onyx

@OffByOne said in Script/program for installing software - linux:

If I learn how to make a DEB, I won't be able to use that on Redhat derivatives.
It's not portable enough to my liking.

FWIW, a .deb is just a tar archive with a copy of directory structure you need on the target (so, for example, if you need to put an executable named foo into /usr/local/bin your .deb will literally have that directory structure present inside it). So, not really hard there. No idea about rpm though.

@OffByOne said in Script/program for installing software - linux:

Also also, I'm not even sure those package formats support all things I'm looking for (especially checking if what is currently in the live filesystem exists in the repo history somewhere, to prevent overwriting a file that has been changed on the filesystem itself by someone).

Well, they have to, the only question is if there's a limitation on granularity and if that granularity even matters. As in, when you upgrade Apache it doesn't touch your existing apache2.conf, but it creates it on initial install. From purely observational standpoint, it would seem that overwriting anything that's not an executable is something you have to do explicitly, rather than having to keep an eye on it. Also, aptitude/apt-get can show you diffs during install and offer to replace the file, leave it be and such, and I'm pretty sure those are the defaults, not explicit scripts.

accalia

@Onyx said in Script/program for installing software - linux:

FWIW, a .deb is just a tar archive with a copy of directory structure you need on the target

there are some other features, like pre install checks, dependencies, conflicts, post install commands you can add to them, but that's just putting special files in special places in the tarfile. the thing will work without those bits.

.deb does have one gotcha though. if /bin/foo is owned by package foo and you try to install a package bar that also has /bin/foo the install will fail because a target file exists that's owned by another package.... unless you force the install anyway, then you have a situation where you can't uninstall either package because the file they want to remove is owned by another package.... and upgrades to either package become more interesting too.

Onyx

@accalia yeah, the extra stuff is nice, and if @OffByOne wants to try and use packages that's what my suggestion for adding cronjobs and such would be, but my primary intent was to demystify the whole packaging thing a bit ;)

boomzilla

@OffByOne said in Script/program for installing software - linux:

Doesn't really matter. Packaging everything in a .deb or .rpm is overkill.

But it sounds like exactly what you're asking for. You'd need to write the scripts that build the packages and then you could build them with a single command.

Dragnslcr

I'll add in my vote for using packages. A configuration management system like Salt or Puppet would probably work, too.

OffByOne

@accalia said in Script/program for installing software - linux:

.deb does have one gotcha though. if /bin/foo is owned by package foo and you try to install a package bar that also has /bin/foo the install will fail because a target file exists that's owned by another package....

That doesn't matter. I'll use it to deploy in-house developed scripts and webservices. There should be no conflicts with files belonging to system packages.

@Onyx said in Script/program for installing software - linux:

@accalia yeah, the extra stuff is nice, and if @OffByOne wants to try and use packages that's what my suggestion for adding cronjobs and such would be, but my primary intent was to demystify the whole packaging thing a bit ;)

Yeah, thanks. :) I assumed packaging stuff would be a lot more involved. I have looked at RPM some time ago and it looked quite involved.
Maybe packaging stuff as .deb and distributing it that way is what I'm going to do.

@boomzilla said in Script/program for installing software - linux:

@OffByOne said in Script/program for installing software - linux:

Doesn't really matter. Packaging everything in a .deb or .rpm is overkill.

But it sounds like exactly what you're asking for. You'd need to write the scripts that build the packages and then you could build them with a single command.

I guess (and a quick DuckDuckGo for "makefile create deb" confirms) that it won't be too hard to create .deb files in a fairly automated manner.

@Dragnslcr said in Script/program for installing software - linux:

I'll add in my vote for using packages. A configuration management system like Salt or Puppet would probably work, too.

Packages it is then. Too bad some of the servers I'll have to deploy to run on Redhat-derived distro's. Installing new software on those servers is frowned upon, even system software :/

Onyx

@OffByOne I wonder, is there an equivalent of (or maybe it's already capable of it?) alien that can convert deb to rpm?

I used it to convert rpms to debs quite successfully in the past.

asdf

@OffByOne Why not build an actual package for your distribution for each repository? That's exactly what a package is: A set of files plus post- and pre-installation scripts.

Edit: For some reason, NodeBB refused to show me the other replies until after I posted. Take a look at https://www.debian.org/doc/manuals/packaging-tutorial/packaging-tutorial.en.pdf if you want to build a Debian package. Skip the documentation about the whole infrastructure/tools for patches, since you're not packaging an upstream project, but your own software. Trust me, it's pretty easy, especially with all the nice packaging tools debian provides.

OffByOne

@Onyx said in Script/program for installing software - linux:

@OffByOne I wonder, is there an equivalent of (or maybe it's already capable of it?) alien that can convert deb to rpm?

I used it to convert rpms to debs quite successfully in the past.

There probably is, but installing new software on those servers is something I shouldn't do, according to management. The reason has something to do with the vendor owning the software on those servers blaming bugs on us installing extra software on them and calling the setup "unsupported".

Apparently we're gaining credibility with the vendor, since they haven't balked at our scripts working around their bugs :)

Still, installing extra software is kind of a grey area.

@asdf said in Script/program for installing software - linux:

@OffByOne Why not build an actual package for your distribution for each repository? That's exactly what a package is: A set of files plus post- and pre-installation scripts.

Not all servers run the same distro. I'd rather not have to build a .rpm, a .deb, ... just to distribute 3 scripts and add them to crontab.

The goal is to make installation of those scripts kinda foolproof (hence the check if files on the live system have been modified, before overwriting them) and make it easy for my colleagues who know a command line, but are not very expert at it.

I'll check out the PDF you linked to in your edit, thanks :)

Gąska

@OffByOne said in Script/program for installing software - linux:

Also, if I learn how to make an RPM, I won't be able to use it on Debian derivatives. If I learn how to make a DEB, I won't be able to use that on Redhat derivatives.
It's not portable enough to my liking.

Do you have different distributions on the servers you want to use it on?

dse

install your .tar.gz to a folder, then rsync from there to the root. rsync can merge much better.

asdf

@dse said in Script/program for installing software - linux:

install your .tar.gz to a folder, then rsync from there to the root. rsync can merge much better.

Bad idea; files you delete in the repo will never be deleted in the root FS.

OffByOne

Thanks all for your insights and tips. I've checked out http://tldp.org/HOWTO/html_single/Debian-Binary-Package-Building-HOWTO/ and it looks easy enough to use for my purposes.

I'll have to think about how to implement my want-to-have feature to verify that the files on the live filesystem are identical to some version of those files in the git repo, but creating deb packages to deploy my scripts to work around vendor bugs and small webapps looks entirely doable :)

dse

@asdf said in Script/program for installing software - linux:

@dse said in Script/program for installing software - linux:

install your .tar.gz to a folder, then rsync from there to the root. rsync can merge much better.

Bad idea; files you delete in the repo will never be deleted in the root FS.

--delete[-before/after]

rsync is too powerful to claim something it cannot do, and what 1 rsync cannot do, 2 rsyncs can :--)

OffByOne

@Gąska said in Script/program for installing software - linux:

Do you have different distributions on the servers you want to use it on?

Yes. Some are Ubuntu Server, some are CentOS. Some are something else (but with either dpkg or yum available). I'd like to have a solution that is distribution-independent.

@dse said in Script/program for installing software - linux:

install your .tar.gz to a folder, then rsync from there to the root. rsync can merge much better.

Then I wouldn't be able to check if someone edited a script on the live filesystem before overwriting it with my latest version. This is really important: sometimes the vendor logs in on those systems and adds/edits scripts as a workaround for bugs we found before they release an official update where the bug is fixed.
It doesn't help that our (direct) vendor uses software developed by another vendor. One of the bugs we found and reported to our vendor has been reported to their vendor, but it remains to be seen when and even if that bug will be fixed.
In the meanwhile we can keep our customers happy by having a workaround implemented.

Two of those servers are load balanced, so the same changes must be made in two places. That's why I'd like something that's automated as much as possible, so accidentally skipping a deployment step or making a typo in a config file doesn't cause two divergent setups.

dse

@OffByOne said in Script/program for installing software - linux:

@Gąska said in Script/program for installing software - linux:

Do you have different distributions on the servers you want to use it on?

Yes. Some are Ubuntu Server, some are CentOS. Some are something else (but with either dpkg or yum available). I'd like to have a solution that is distribution-independent.

@dse said in Script/program for installing software - linux:

install your .tar.gz to a folder, then rsync from there to the root. rsync can merge much better.

Then I wouldn't be able to check if someone edited a script on the live filesystem before overwriting it with my latest version. This is really important: sometimes the vendor logs in on those systems and adds/edits scripts as a workaround for bugs we found before they release an official update where the bug is fixed.
It doesn't help that our (direct) vendor uses software developed by another vendor. One of the bugs we found and reported to our vendor has been reported to their vendor, but it remains to be seen when and even if that bug will be fixed.
In the meanwhile we can keep our customers happy by having a workaround implemented.

Two of those servers are load balanced, so the same changes must be made in two places. That's why I'd like something that's automated as much as possible, so accidentally skipping a deployment step or making a typo in a config file doesn't cause two divergent setups.

You seriously should make an rpm. Then version your package, and clean-up, install new packages based on that.
Also, install your configs to /somefixedplace (/etc ?) then ask users to change/overide theirs in theirs in ~/.app or somewhere other than /somefixedplace, then you wont have to track who changed what. Seriously, if you are distributing for Linux, just follow good/common practices for Linux

OffByOne

@dse said in Script/program for installing software - linux:

You seriously should make an rpm. Then version your package, and clean-up, install new packages based on that.
Also, install your configs to /somefixedplace (/etc ?) then ask users to change/overide theirs in theirs in ~/.app or somewhere other than /somefixedplace, then you wont have to track who changed what. Seriously, if you are distributing for Linux, just follow good/common practices for Linux

I'm not distributing for linux in general, I'm looking for a way to easily distribute some scripts and their configs to vendor-provided machines which happen to run linux.
I don't choose the distro, I don't maintain the software installed on it (which is compiled and installed in /opt anyway), I do need a way of deploying maintenance scripts that's easy enough for my not-so-linux-savvy colleagues to be able to execute without fear of breaking something.

There are no users running my software, so config in ~/.apprc doesn't apply in my situation.

I want to keep a VCS repo (I use git) of the files I or someone else changes (both for documenting what changed and why, as for easy rollback if a change proves to break something).
I'd like a way to go from a git repo with scripts to easily deploy the scripts in it that does some sanity checks (like "did someone change a script on the server, because then I don't want to blindly overwrite it") and it should be easy to do for my colleagues.
For some scripts just copying them to /usr/local/bin isn't enough, some extra magic needs to happen like adding an entry to crontab or restarting a service. I'd like that to be automated too.

Right now there is no version control whatsoever on the scripts running on the servers. My colleagues don't know about VCS (maybe some of them read about them once, but none of them saw a VCS up close).
There is no in-house development btw, we resell boxes and support our customers when those resold boxes don't do what they are supposed to do. I'm the only one with a strong development background here and I'd like to introduce methodologies used in development, like source control and automated tests. That's exactly why I'm hired here btw.
We have infrastructure (bought from a vendor, not developed in-house) where bugs pop up sometimes and then we want to deploy a workaround ASAP, while waiting for an upstream fix from the vendor.

I would really like to introduce a sane way of managing this, that's why I'm trying to get VCS integrated in the workflow. My colleagues are sympathetic to the idea and see the value, but it will be a very hard sell if they have to type (or more likely, copy/paste) 20 CLI commands.
Instead I'd like to give them instructions like "run git pull on your linux VM to get the latest version of the scripts from the git server, then run mycommand install <servername> to deploy the scripts in a safe and automated way to <servername>. That's it.".
What exactly mycommand is or could be, is what I'm trying to figure out with this topic.

Yamikuronue

@OffByOne said in Script/program for installing software - linux:

I would really like to introduce a sane way of managing this, that's why I'm trying to get VCS integrated in the workflow.

Grab Jenkins and have it manage the deployment?

FrostCat

@OffByOne said in Script/program for installing software - linux:

This means I would have to put everything in a .tar.gz file and untar it in the root directory of the target servers. I don't feel comfortable doing that.

Haven't read anything beyond this yet, but surely you could use several smaller tarballs.

FrostCat

@OffByOne said in Script/program for installing software - linux:

Apache must be signaled to reload its configuration. I'd like this to be automated.

Oops, hit Submit too soon, and CBA to scroll to the end of thread. I don't know about Apache but I've seen Unix apps that can be signalled from the command line. You'd just kill -SIGwhatever it, and it knows to re-read its configuration.

cartman82

Make a bash install script. Give your users a one-liner that curl-s it straight from the git and pipes it into an sh. Done and done.

It's the OSS way.

OffByOne

@FrostCat said in Script/program for installing software - linux:

@OffByOne said in Script/program for installing software - linux:

Apache must be signaled to reload its configuration. I'd like this to be automated.

I don't know about Apache but I've seen Unix apps that can be signalled from the command line. You'd just kill -SIGwhatever it, and it knows to re-read its configuration.

Apache can be persuaded to reload its configuration by executing apachectl reload. It was just an example of extra stuff that needs to happen after copying the files.

My point is that I want this to happen automatically by the deployment command (be it as a script in a .deb package, a line in a Makefile, a script I write myself, ...).

I want to be able to tell my coworkers "pull from the git repo and execute this command "<...>" and everything that needs to happen to deploy the files in the repo (copying them to the appropriate locations on the target server, but also restarting daemons, appending a line to a crontab, maybe create a user or sending an email?) will happen. There are failsafes built-in, so if someone has changed a file directly on the server, the "<...>" command will warn you about that instead of just overwriting those changes with whatever the latest version of that file in the repo is".

I'm looking for what my options are for this "<...>" command.

.deb might work (I could use an environment variable to pass the target server or instruct them to copy the .deb to the target server, because it works on the filesystem it's on instead of remotely).
It wouldn't be a nice solution for machines that run a Redhat-derived distro though, because I'd have to install alien or something similar to convert the .deb to .rpm first.

A Makefile might work, but I'd have to think about it, because it's not entirely the right tool for the job. The make command is pretty universally available though, so in that respect it's distro agnostic and that's a plus.

I could just write something myself and add "special" directives (restart daemon, add line to crontab, ...) as needed. This solution would also be able to check before copying each file if the version on the target server matches an older version in the repo and abort if that's not the case, so it won't overwrite (potentially important!) changes made by someone directly on the target server.
It could even do its work over an SSH connection to the target server and eliminate the "copy the .deb/Makefile/repo contents to the server first" step.

I assumed I'm not the first person who wants this, but DuckDuckGo didn't turn up anything resembling what I'm looking for.

OffByOne

@cartman82 said in Script/program for installing software - linux:

Make a bash install script. Give your users a one-liner that curl-s it straight from the git and pipes it into an sh. Done and done.

It's the OSS way.

That's the "write something myself" option. I might just have to do that if existing solutions aren't as flexible as what I'm looking for.

asdf

@OffByOne I think you may have ed there.

powerlord

@OffByOne said in Script/program for installing software - linux:

Apache can be persuaded to reload its configuration by executing apachectl reload. It was just an example of extra stuff that needs to happen after copying the files.

Last I checked, apachectl reload is just sending the apache process a SIGHUP, which for some reason got co-opted by a lot of UNIXy programs as the "reload your config file" signal.

groo

there is that puppet and chef thingies, maybe that would be useful here

flabdablet

@OffByOne said in Script/program for installing software - linux:

Packaging everything in a .deb or .rpm is overkill

Then you'd best do your pounding with the old shoe. The glass bottle could shatter on you.

ben_lubar

@cartman82 said in Script/program for installing software - linux:

Make a bash install script. Give your users a one-liner that curl-s it straight from the git and pipes it into an sh. Done and done.

It's the OSS way.

No, you've gotta pipe it to sudo bash. Otherwise it's not secure. Or something.

cartman82

@ben_lubar said in Script/program for installing software - linux:

No, you've gotta pipe it to sudo bash. Otherwise it's not secure. Or something.

Only if you're a looser who isn't doing everything using the root account.

OffByOne

@asdf said in Script/program for installing software - linux:

@OffByOne I think you may have ed there.

I got the tongue-in-cheek undertone :) Using a bash script (and piping it from a 'curl <URL>') seems to be the OSS way lately though...

@ben_lubar said in Script/program for installing software - linux:

No, you've gotta pipe it to sudo bash. Otherwise it's not secure. Or something.

Didn't you know sudo stands for "SecUre DO"?

OffByOne

@Yamikuronue said in Script/program for installing software - linux:

Grab Jenkins and have it manage the deployment?

I'll look into it, thanks!

wft

@OffByOne Could you just package it? You know, RPM and deb package managers are excellent at handling the stuff you write about.

cark

@OffByOne said in Script/program for installing software - linux:

I've checked out http://tldp.org/HOWTO/html_single/Debian-Binary-Package-Building-HOWTO/

Keep in mind that that page has not been updated in the last 10 years and many steps have likely completely changed.

@OffByOne said in Script/program for installing software - linux:

add line to crontab

Don't literally add a new line to your crontab. Use /etc/cron.d

JBert

@groo said in Script/program for installing software - linux:

there is that puppet and chef thingies, maybe that would be useful here

I've been thinking about tools like that (including what you'd set up with Jenkins) but they assume

the machines are at all times connected to a master server (don't know this is true, could be a problem if @OffByOne sells boxes and have the customer plug it into a private network)
the version from the master server "is king" - there's no versioning on the the remote system (it sounds like the vendor also runs hotfixes on each machine separately?) as these tools assume nobody bypasses the system

I prefer Ansible because it doesn't assume point 1, but point 2 is common among configuration tools. They almost invariably recommend to regenerate files from a template, but then @OffByOne would need to stay in sync with the vendor to continuously update all templates (multiple if there could be differences between distributions).

Grunnen

@OffByOne said in Script/program for installing software - linux:

There are failsafes built-in, so if someone has changed a file directly on the server, the "<...>" command will warn you about that instead of just overwriting those changes with whatever the latest version of that file in the repo is".
I'm looking for what my options are for this "<...>" command.
.deb might work (I could use an environment variable to pass the target server or instruct them to copy the .deb to the target server, because it works on the filesystem it's on instead of remotely).

I'm not sure about rpm myself, but this was the reason why I suggested to use a package. At least for .deb packages, there are three ways by which files can be deployed and this can be chosen on a file-by-file basis.

Normal files: old versions are always overwritten, package cannot be installed if conflicting file already exists, files are removed on package removal
Configuration files: will not overwrite existing files on first installation; will look during upgrades whether the file was modified by hand and then ask what to do; will not remove the file except when explicitly asked to remove configuration files
Manual operations in pre-install, post-install etc. scripts

I don't really have experience with configuration management software, but apparently JBert has written something about that in the post above.

OffByOne

@cark said in Script/program for installing software - linux:

Don't literally add a new line to your crontab. Use /etc/cron.d

Ok, I'll use /etc/cron.d/. The point is that a file must be created there with the appropriate permissions as part of the deployment.

@JBert said in Script/program for installing software - linux:

I've been thinking about tools like that (including what you'd set up with Jenkins) but they assume

the machines are at all times connected to a master server (don't know this is true, could be a problem if @OffByOne sells boxes and have the customer plug it into a private network)

What do you mean by "connected"? In my use case the servers that I'd like to manage this way are on the same subnet, but there is no daemon running on either the servers or the machines where the deployments will be launched from.

the version from the master server "is king" - there's no versioning on the the remote system (it sounds like the vendor also runs hotfixes on each machine separately?) as these tools assume nobody bypasses the system

Thanks for that formulation. What I meant was exactly that: the vendor sometimes applies hotfixes on the machines without going through my version controlled system (because that system is not in place yet). Maybe I could persuade the vendor to use my system when it is implemented.
Either way, I'd really like to prevent the changes made by the vendor to be overwritten without explicit consent by either me or one of my coworkers. It would be nice to offer a way to merge the changes made by the vendor and the changes made by our team. Since our changes will be in a git repository, I should be able to implement that.

I prefer Ansible because it doesn't assume point 1, but point 2 is common among configuration tools. They almost invariably recommend to regenerate files from a template, but then @OffByOne would need to stay in sync with the vendor to continuously update all templates (multiple if there could be differences between distributions).

I hope I can persuade the vendor to use our version controlled system, but I'm not counting on it. I don't mind doing a little extra work merging vendor hotfixes directly on the live filesystem, but I really wouldn't want to silently overwrite them.