Modern-day deployment


  • I survived the hour long Uno hand

    So I know how my company does deploys. But I know also that we're often in a bit of a backwater, clinging to old technologies (like Coldfusion). So I want to know how the rest of ya'll solve these problems, maybe I can learn something:

    In my own personal projects, I am using Github and Travis CI for continuous testing/integration. I want to get to continuous deploys on two particular projects. The first is a node-based project; I still have to figure out how I plan to run the node server alongside my existing apache server, but when I do, I'll want continuous deploys from master into my "production" environment. The app is pretty straightforward, except for an SQLite database I obviously won't want to overwrite every time. How do people deploy these days? I know for Java/Tomcat apps, you want to have the CI system produce a WAR and then move just that up, but obviously a node app doesn't package neatly into a single file... or do I use a packaging system of some kind to do just that?

    The second project I'd like to be deploying continuously is a perl program. The dependencies may change between versions, and some of the files in the repo are not in the final location where they need to be in order to run, so I was working on an installer for it, but it's driven off a large config file, and I don't know what to do about that. I started writing an interactive script that generates a configuration file, but obviously when deploying from CI I just want to use the existing config file instead. I also don't have a daemon script for it yet, so at the moment I deploy by FTPing into the server, uploading the changed files, killing the existing process, and starting it again.

    What would you guys do that isn't terrible?


  • FoxDev

    @Yamikuronue said:

    obviously a node app doesn't package neatly into a single file... or do I use a packaging system of some kind to do just that?

    npm? ;)


  • I survived the hour long Uno hand

    Isn't npm more for libraries tho? Nobody wants my toy app.


  • FoxDev

    You can point npm to a private repo: example


  • I survived the hour long Uno hand

    ok. How do I get my app into the private repo tho? My CI server is in the cloud, as is my prod server. Wouldn't I need my app to be on the box so I can add it to the repo so I can put it on the box?



  • @Yamikuronue said:

    In my own personal projects, I am using Github and Travis CI for continuous testing/integration. I want to get to continuous deploys on two particular projects. The first is a node-based project; I still have to figure out how I plan to run the node server alongside my existing apache server, but when I do, I'll want continuous deploys from master into my "production" environment. The app is pretty straightforward, except for an SQLite database I obviously won't want to overwrite every time. How do people deploy these days? I know for Java/Tomcat apps, you want to have the CI system produce a WAR and then move just that up, but obviously a node app doesn't package neatly into a single file... or do I use a packaging system of some kind to do just that?

    I have some experience there, but you'll probably be disappointed, if you're used to some kind of fancy CI system.

    I set up a git on the server. It uses its own read-only "deploy" key, so it can pull from the git server, but can't push. I use master as the "live" branch and feature branches to push development forward. I'm contemplating moving to named tags for deployment, still not sure about that.

    I set up something like a service script, so that I can start / stop my node app. In my case, it's my own script due to other concerns. But you can use normal service script, or systemd, depending on Linux version.

    I have a script on my own machine that can ssh into the server and

    1. Pull from git. Then do: git reset --hard origin/master (in case of conflicts)
    2. Execute npm install
    3. Execute grunt/gulp/whatever you use for frontend compilation
    4. Restart all services

    If there's a db migration, I do that by hand (basically execute an SQL file against the server). I have some scripting support there, but I'm too chickenshit to use it.

    That's pretty much it. Since there's no compilation, you can just plop files from git and start them up. Make sure your database and config files are in .gitignore and you're good.


  • FoxDev

    @Yamikuronue said:

    ok. How do I get my app into the private repo tho? My CI server is in the cloud, as is my prod server. Wouldn't I need my app to be on the box so I can add it to the repo so I can put it on the box?

    Having never created an npm package, I wouldn't know for sure. But it seems to me that it should be possible to set up the private repo on a suitable server, so you can deploy to whatever machines you need to.

    But then, @cartman82's solution will probably work better.



  • @RaceProUK said:

    Having never created an npm package, I wouldn't know for sure. But it seems to me that it should be possible to set up the private repo on a suitable server, so you can deploy to whatever machines you need to.

    I'll have to look into the npm private deploy at some point. If you have an ecosystem of private modules you want to reuse, that's the only sane way to do it (git modules being a distant second).

    As for me, I have one big monolitic project that just has all the custom node stuff it needs :-(
    The few times I made offshoot projects, it was copy-paste time, no module reuse :-(
    The few reusable modules I did make, I open-sourced and published to npm, so I can reuse them that way.

    So yeah, private npm. On the TODO list.


  • I survived the hour long Uno hand

    @cartman82 said:

    I set up a git on the server.

    This method is basically how my company does things, but with Subversion. So I'm well familiar with it :)

    Is that the best way we've got?



  • @Yamikuronue said:

    Is that the best way we've got?

    The best I know, I'm afraid.

    I'll be following this thread along in case someone comes up with a (significantly) better way.


  • ♿ (Parody)

    Isn't this one of the things that @apapadimoulis' day job is supposed to do?

    :shill:


    Filed Under: Fake emoji markup is the new tag abuse


  • I survived the hour long Uno hand

    If it does it the same as Jenkins or Bamboo handle deployments, you still need a strategy in mind for how you plan to deploy, it just runs it automatically for you. If I had a process I could make Travis do it.


  • ♿ (Parody)

    I've never used it, but his pitch seems to be that it does more than those tools.

    Wasn't he giving out mugs for installing it? Did anyone try it? (I didn't.)



  • Well I just have a tbz and a deployment script, but I think the modern way to package stuff is in a docker image?



  • @Yamikuronue said:

    @cartman82 said:
    I set up a git on the server.

    This method is basically how my company does things, but with Subversion. So I'm well familiar with it

    Is that the best way we've got?

    The version control metadata on the server come in handy when you need to check what is deployed and whether is is unmodified. Plus I would suspect it is less work. You need to set up a trigger on the server either way, the post-install script is going to be the same as well and installing git itself is not a problem. And I don't see any advantage creating package would add.


  • I survived the hour long Uno hand

    I was literally asking, because I have no idea what else is out there for javascript. Maybe there's some cool new shit that automatically deploys, keeps things up to date and running, and also does my taxes and washes my car.

    A good example: I had no idea nodemon was even needed until someone showed it to me and now I hate developing without it.


  • ♿ (Parody)

    @boomzilla said:

    I've never used it, but his pitch seems to be that it does more than those tools.

    Here's a video that I made if you're curious!

    https://www.youtube.com/watch?v=j8m3jOldDjs

    @Yamikuronue said:

    I was literally asking, because I have no idea what else is out there for javascript. Maybe there's some cool new shit that automatically deploys, keeps things up to date and running, and also does my taxes and washes my car.

    JavaScript is just a bunch of files on disk, and those can be packed at build time into a zip file (a "build artifact"), and then deployed to the target path on a server.That's it.

    However... there is most definitely some new JavaScript-based shit (as in, feces) with a stupid, geek-culture themed name that does a world of complicated rubbish that kinda works about 80% of the time, and then fails in a manner that's impossible to debug, diagnose, or rectify. I'd argue that, since you're clearly a masochist for using JavaScript to write software, you should just write your own deployment tool that leverages npm, gulp, github, rpm, unicorn, apache, jquery, and symbolic links.


  • Discourse touched me in a no-no place

    @boomzilla said:

    Did anyone try [BuildMaster]?

    Not me - it's primarily Windows as far as I can tell. While the rest of our company may run under Windows, the stuff we actually sell is developed on systems that are most certainly Not Windows.


  • I survived the hour long Uno hand

    The guys around here are sure that everyone's using git to push to heroku.


  • FoxDev

    @apapadimoulis said:

    However... there is most definitely some new JavaScript-based shit (as in, feces) with a stupid, geek-culture themed name that does a world of complicated rubbish that kinda works about 80% of the time, and then fails in a manner that's impossible to debug, diagnose, or rectify. I'd argue that, since you're clearly a masochist for using JavaScript to write software, you should just write your own deployment tool that leverages npm, gulp, github, rpm, unicorn, apache, jquery, and symbolic links.

    -cough- ouch!

    that hurt!

    I was going to stay out of this discussion but i guess it's time to throw my hat into the ring.

    I'd recommend this approach:

    1. have travis-ci generate build artifacts (.tar.gz is a good format for deploying to linux servers or .zip for cross platform)
    2. have travis-ci use webhooks to trigger a deploy on the deploy target on build success.
    3. use the built in system daemon runner (upstart or systemd or some sort of windows service wrapper depending on your system)
    4. Deploy
    5. Stop the service/daemon
    6. deploy the travis generated build artifact,this should be a simple unzip of the build artifact
    7. perform any necessary smoke tests (using gulp/grunt/mocha/shell script/powershell)
    8. If smoke tests pass start the service/daemon
    9. Use a webhook/email to notify the relevant parties that the build completed (and the status)

    Although i really should spend some time getting to know buildmaster and other CI/CD tools and find out if there's a better way.


  • Discourse touched me in a no-no place

    @accalia said:

    (.tar.gz is a good format for deploying to linux servers or .zip for cross platform)

    Everybody knows WinZip has supported .tar.gz/.tgz for approximately forever, right?


  • FoxDev

    not everyone has winzip installed. but since XP windows has understood .zip (unless using zip64 extensions) natively. ;-)


  • Discourse touched me in a no-no place

    @accalia said:

    not everyone has winzip installed. but since XP windows has understood .zip (unless using zip64 extensions) natively.

    No, but if you're talking businesses, it's not terribly expensive.

    XP cannot create, with the builtin, a passworded zip. I don't think it can read them, either, and I'm pretty sure it can't handle tarballs.

    So if you're willing to spend a little money on WinZip you can use tarballs cross-platform, is all I meant. (7-zip claims to support tars and gzip, so it should work too, free, and more or less unrestricted in its license.)


  • FoxDev

    7-zip is also free


  • Discourse touched me in a no-no place

    @RaceProUK said:

    7-zip is also free

    Yes, I said that. 😛


  • FoxDev

    @FrostCat said:

    No, but if you're talking businesses, it's not terribly expensive.

    that would assume sane business.

    better go for lowest common denominator.

    we're talking about build artifacts here so it won't be passworded. If it's for windows zip is good, and the space savings on tar.gz are not that compelling to raise the denominator.



  • @FrostCat said:

    XP cannot create, with the builtin, a passworded zip. I don't think it can read them, either, and I'm pretty sure it can't handle tarballs.

    You have it backwards, XP could create and read password protected zip files. Newer Windows OSs don't have that ability. That's a good thing though, because the encryption that feature could handle was shit (zipcrypto if memory serves).


  • ♿ (Parody)

    @accalia said:

    -cough- ouch!

    that hurt!

    I know, I know... it's like kicking a retarted dead horse when it's down, but I still can't avoid JavaScript quips(or Linux, for that matter)...

    @accalia said:

    I'd recommend this approach [...]

    The flow is definitely on track; the key is creating immutable build artifacts, and then deploying the artifacts to servers in various environments as part of a dev-test-qa-prod or something wofklow. What something like BuildMater will get you is better visibility into what's actually happening, team collaboration, all that stuff --- if it's just you and the command line, may be a bit much overhead.


  • FoxDev

    @apapadimoulis said:

    I still can't avoid JavaScript quips(or Linux, for that matter)...

    No harm, no fowl, at least so long as i can counter quip. ;-)

    @apapadimoulis said:

    ; the key is creating immutable build artifacts, and then deploying the artifacts to servers in various environments as part of a dev-test-qa-prod or something wofklow.
    indeed. what i often see people miss is that the build artifact absolutely shall not change between environments.

    any bugs found back to square one you go!

    @apapadimoulis said:

    What something like BuildMater will get you is better visibility into what's actually happening, team collaboration, all that stuff -
    definitely something i want to look into then.

    @apapadimoulis said:

    if it's just you and the command line, may be a bit much overhead.
    for my personal projects maybe, but for work? nah i think that's about right.


  • Discourse touched me in a no-no place

    @apapadimoulis said:

    BuildMater


  • Discourse touched me in a no-no place

    @apapadimoulis said:

    boomzilla:
    I've never used [BuildMaster], but his pitch seems to be that it does more than those tools.

    Here's a video that I made if you're curious!

    If it were anyone else this might be cause for a One of Us Badge.


  • FoxDev

    honestly i'd be tempted to award it anyway.

    we don't see Alex often, so it seems that the argument could be armwrestled to fit.


  • Discourse touched me in a no-no place

    @accalia said:

    honestly i'd be tempted to award it anyway.

    What the heck, I haven't ed anyone in a while.


  • ♿ (Parody)

    @accalia said:

    we don't see Alex often, so it seems that the argument could be armwrestled to fit.

    But...but....he has the second earliest post evar!

    https://what.thedailywtf.com/t/test-thread/9323/2?u=boomzilla


  • FoxDev

    i didn't say it would be easy to arm wrestle it to fit. ;-)


  • Java Dev

    We mostly see Alex when he's @mentioned, and that's usually when suggesting people buildmaster.



  • Why not simple use TFS (and/or VSO)....click, click, click (and a few more) and it is good to go.



  • @Yamikuronue said:

    I still have to figure out how I plan to run the node server alongside my existing apache server

    This is all you need:

    <VirtualHost *:80>
      ...
      ProxyPass /jongo/ http://192.168.1.5:8080/nodeapp/
      ProxyPassReverse /nodeapp/ http://192.168.1.5:8080/nodeapp/
    </VirtualHost>
    

Log in to reply