Branches, Builds, and Deployment

Magus

Everyone seems to have a branching policy and a preferred build style. I'm interested in what people use/like. Because I don't like what I use.

For branching, we use some illegitimate offspring of gitflow apparently, though since we moved down to just one team on this project, we don't even typically have feature branches.
Our CI build deploys immediately, with the latest version of all the things, though we can manually make specific deployment sets. We never do.
Promotion between environments is purely artifact-based (using Octopus for one project, and VSTS's built in one for the other
Our back- and front-end code are in their two respective repositories on both projects, despite one of them being a swarm of service fabric actors.
Every week, we alternate between promoting the current build from our QA environment to UAT and promoting UAT to Production. This means that sometimes a component has only been in QA for like a day before it moves on.

I hate this. A lot. I want every bit of code to remain in QA for an entire sprint before promotion, with no exceptions. I get that sometimes you have three people who have to work on the same module and it may all end up in one sprint. I don't think that's good planning, though. Ideally:

master is inviolable, and your pull request must build to be promoted. If deployment fails, your pull request should automatically be reverted. We do the first part.
Apps should be exceedingly modular. If you have an actor that hangs on to gigs of data, or 2000 lines of code, you need to go rethink your life. Even a monolithic webapp can be done as tiny, runtime-loaded assemblies that can be hotswapped. Go small or go home.
When you do the previous thing, you can have each tiny thing be deployable. Coordinating releases gets harder, but you at least have a prayer of getting serious testing time.
Make them all separate repositories. The prospect of having less merges, that are simpler, is worth designing your code in a modular way in and of itself.
Your repository's history should not be changing. Ever. So you don't need to hang on to artifacts, and neither should you. Build code. Handling multiple environments has been supported that way for a really long time.

Obviously this has problems. But I'd much rather deal with them than the kind of horrors we actually encounter.

Unperverted Vixen

This is what we use. I don't know if it's "preferred" or not. I'd be interested to hear opposition.

Source control is TFVC. Two branches. Call them Development and Master.
- This is an ASP.NET web app, so stuff like feature branches aren't viable at the moment; we don't have the ability to stand up a dedicated environment for each feature branch on demand.
No official CI builds, but devs are good about deploying to QA as they finish stuff (so it's not uncommon to see 12-plus per day).
Items get turned over for testing in QA after the build.
Once enough items in a release have passed testing in QA, they get merged to master and built to UAT.
- There's no fixed schedule for when stuff gets merged to master/built, how long the testing period lasts, etc.
Promotion from UAT to production is artifact-based. Since promotion from QA to UAT uses different branches, that's not feasible.
This app is very much a monolith. Going small sounds good until you realize that any database schema changes mean you need to rebuild, recompile, and redistribute, which loses all the benefits of splitting it apart. No thank you.

Greybeard

Projects I work on now:

Master branch and any number of feature branches
A push to any feature branch kicks off a build, deployment to an environment specific to feature branches for that repo, and integration tests.
Master branch is only modified through pull requests from feature branches.
A merge to master kicks off a build. The result is deployed to a succession of environments, running integration tests at each stage, until it is in staging.
Promotion from staging to production requires manual triggering of a promotion artifact by operations.

Magus

@unperverted-vixen said in Branches, Builds, and Deployment:

This app is very much a monolith. Going small sounds good until you realize that any database schema changes mean you need to rebuild, recompile, and redistribute, which loses all the benefits of splitting it apart. No thank you.

That depends on where you put your module boundaries.

ben_lubar

If I'm working on a project with other people:

Git
Every change is in a branch
Pull requests

If I'm working solo on a project (or it's very low churn like the Docker container this forum runs on):

Git
Commit directly to master

Greybeard

@unperverted-vixen said in Branches, Builds, and Deployment:

Two branches. Call them Development and Master.

A bunch of projects here have a "develop" branch. The "develop" branch seems a big anti-pattern to me. If the only way to get into master is through develop and everything in develop goes into master at once, then what is all this faffing about with develop buying anyone? Just put it directly into master!

I heard rumor of some teams cherry-picking from develop into master, which is just .

I am carrying out a war on "develop" branches.

Greybeard

@magus said in Branches, Builds, and Deployment:

I want every bit of code to remain in QA for an entire sprint before promotion, with no exceptions.

So no way to even do a production break-fix before the end of the following sprint?

I'm curious: what is it your QA folks are doing that takes them the bulk of a sprint to detect a stop-ship problem?

Magus

@greybeard said in Branches, Builds, and Deployment:

So no way to even do a production break-fix before the end of the following sprint?

If we have something major enough, we'll talk about it, but for the most part it's protection against making things worse by putting in untested code.

@greybeard said in Branches, Builds, and Deployment:

I'm curious: what is it your QA folks are doing that takes them the bulk of a sprint to detect a stop-ship problem?

We work on two projects. One was built by people who didn't believe in QA, and had no tests of any kind. The other had a team in India writing tests, but the tests only worked on one person's machine (though we only have that person's word that they worked). Our team refuses to lack QA, but we're in constant catch-up mode, and management's insistence that we use our India resources means we're constantly in test maintenance as well.

But generally, it's not about work that takes two weeks, it's about having something stable and used for a long enough period that we feel comfortable calling it stable before we move it forward. You know, unlike Discourse, NodeBB, and most other modern apps.

Greybeard

@magus said in Branches, Builds, and Deployment:

If we have something major enough, we'll talk about it

So by "with no exceptions" you mean "with exceptions"?

But generally, it's not about work that takes two weeks, it's about having something stable and used for a long enough period that we feel comfortable calling it stable before we move it forward.

So instead of test coverage you're just hoping someone happens to notice something and complain?

Sure, when a change to a service doesn't break its own tests but does break the tests of its clients one has to do something to get some confidence that breakage would get noticed, diagnosed, and reported.

But why tie that to the length and boundaries of a sprint?

dfdub

@greybeard said in Branches, Builds, and Deployment:

A bunch of projects here have a "develop" branch. The "develop" branch seems a big anti-pattern to me. If the only way to get into master is through develop and everything in develop goes into master at once, then what is all this faffing about with develop buying anyone? Just put it directly into master!

I couldn't agree more. master should contain the latest version of your code that passed the CI build.

The only reason I can see for this is to support people who depend on your latest master in your builds. Which is a huge WTF in itself, but sadly common.

Zecc

This post is deleted!

Zecc

@greybeard said in Branches, Builds, and Deployment:

@unperverted-vixen said in Branches, Builds, and Deployment:

Two branches. Call them Development and Master.

A bunch of projects here have a "develop" branch. The "develop" branch seems a big anti-pattern to me. If the only way to get into master is through develop and everything in develop goes into master at once, then what is all this faffing about with develop buying anyone? Just put it directly into master!

I heard rumor of some teams cherry-picking from develop into master, which is just .

I am carrying out a war on "develop" branches.

OneFlow – a Git branching model and workflow | End of Line Blog

Unperverted Vixen

@greybeard said in Branches, Builds, and Deployment:

A bunch of projects here have a "develop" branch. The "develop" branch seems a big anti-pattern to me. If the only way to get into master is through develop and everything in develop goes into master at once, then what is all this faffing about with develop buying anyone? Just put it directly into master!

In our case, not everything in development does go to master at once.

I heard rumor of some teams cherry-picking from develop into master, which is just .

That's what we do. Since we can't deploy multiple feature branches, there's the one development branch (with changesets separated out into "features" by using the changeset comment to link it to our ticket tracking system). When a ticket passes testing, we pull the changesets related to the ticket and merge those to master.

I suppose in theory we could use feature branches for development, merge to development for QA testing, then when stuff is approved merge from the feature branch to master. But I just don't see a benefit to that approach, besides burning lots of developer hard drive space.

Magus

@greybeard said in Branches, Builds, and Deployment:

So by "with no exceptions" you mean "with exceptions"?

We haven't had a situation yet where an exception was warranted, and it would take something truly catastrophic for us to do otherwise.

@greybeard said in Branches, Builds, and Deployment:

So instead of test coverage you're just hoping someone happens to notice something and complain?

We're trying to get that test coverage. It's just slow going because we were thrown multiple bad states.

@greybeard said in Branches, Builds, and Deployment:

Sure, when a change to a service doesn't break its own tests but does break the tests of its clients one has to do something to get some confidence that breakage would get noticed, diagnosed, and reported.
But why tie that to the length and boundaries of a sprint?

Because that's the point of sprints? More accurately, though, we just want to have a steady cadence, so alternating between promotions every week is advantageous. We don't want to make something have to wait any longer, and promoting both products at the same time would make us have to deal with problems in both at the same time.

boomzilla

We use svn, so a lot of our process is a hold over from before svn figured out branches and merging.

Current sprint is in the trunk
At the end of the sprint we branch it in order to make fixes. It first goes to UAT and then promoted to production.
We occasionally have feature branches for experimental or very large things.

PleegWat

You won't have heard of our source control system.

We use both feature branches and release branches. Feature branches must pass automated tests and review before merging into master. Release branches get made shortly before release. Bugfixes have their own feature branches, go to master first then (if required and approved) get backported to the release branch using a separate intermediate feature branch on the release branch.

Greybeard

@zecc said in Branches, Builds, and Deployment:

OneFlow – a Git branching model and workflow | End of Line Blog

It's off-brand anti-patterns!

dfdub

@pleegwat said in Branches, Builds, and Deployment:

You won't have heard of our source control system.

I sense a story?

djls45

@ben_lubar said in Branches, Builds, and Deployment:

If I'm working solo on a project (or it's very low churn like the Docker container this forum runs on):

Git

Commit directly to master

Ah, so production IS testing. :D

PleegWat

@dfdub It's an ancient in-house behemoth that never got publicly released. Provides both versioned files and build artifacts via symlinks to NFS shares. All metadata is stored in a major relational database cluster.

Designed to allow reasonable development processes on products that take over 24 hours for a full recompile.

sockpuppet7

@greybeard said in Branches, Builds, and Deployment:

A bunch of projects here have a "develop" branch. The "develop" branch seems a big anti-pattern to me. If the only way to get into master is through develop and everything in develop goes into master at once, then what is all this faffing about with develop buying anyone? Just put it directly into master!

It's buying you that this way the master branch is a place where you'll apply hotfixes if needed. It's not the only way for doing that, but I don't see how other methods would be significantly better than that.

dcon

@djls45 said in Branches, Builds, and Deployment:

Ah, so production IS testing.

Well, duh. And our users are unpaid QA drones.

sprint, release beta1
sprint, release beta2
sprint, GA
repeat!

:customer:: What's in the GA?
: Doesn't matter! Schedule is met!

Kian

I should clarify we don't have a service but rather an on premise product, so the final result of or build system is an installer. We don't have envirinments to deploy to or promote between. So things are a bit simpler for us.

Anyway, we use gitflow, kinda, and the organization pays GitHub for private repos.

Develop holds the latest version of the code. To get code into develop, devs create private branches for each change. Each commit to their branches will trigger builds and unit tests. When the work is done they do a pull request, and we review the change. Once it is approved it is merged.

The merge will trigger the build and unit tests again. Additionally, develop runs nightly regression tests and full builds (targeting every one of our supported OSs). We automate the unit test and nightly builds with Jenkins, but we can of course force it to happen on the dev branches if we want to. It's just that the full suite takes over 10 hours.

When we are ready to release a new version, assuming we didn't find any showstoppers in develop, we merge to master and tag with the version name. We run the full suite of tests there again, and put the artifacts where the clients can then go get them after the release notes and guides are updated.

I suppose that other than to have a neat view of release versions there's not much need for master to be separate, but it is nice for visualizing as "actual release" vs work in progress.

blakeyrat

@sockpuppet7 If you don't mind the spam you can keep a branch for each release, and then just apply the patch fix to those as needed. You could also do it with tags and cherry-pick, but that's a bit nastier if the fix has to go WAY back in time.

Kian

@zecc said in Branches, Builds, and Deployment:

        OneFlow - a Git branching model and workflow | End of Line Blog

It seems to me that the main objection he has to gitflow, the 'no ff' merge strategy, is caused by his shitty tools and not by the branching strategy itself. Sure, if you sort commits by date, work that happens in parallel will show up mixed up. The solution to that is not to change the dates of the commits, or squash then or whatever. Just sort the commits by branch instead. If you have shitty tools that don't let you visualize it that way, that's not the fault of the branching model.

Sorry, I'm on mobile and it murdered that quote.

dfdub

@pleegwat We may work for the same company. Hello potential co-worker.

sockpuppet7

@kian said in Branches, Builds, and Deployment:

Just sort the commits by branch instead. If you have shitty tools that don't let you visualize it that way, that's not the fault of the branching model.

What tools let you sort by branch?

Kian

@sockpuppet7 don't know, my tools are shitty.

Greybeard

@unperverted-vixen said in [Branches, Builds, and Deployment](/post

I heard rumor of some teams cherry-picking from develop into master, which is just .

That's what we do.

It kinda invalidates the testing that happened against develop. Plus a problem with one change in develop has a good chance of preventing anything else from going forward.

Since we can't deploy multiple feature branches

Yeah, I figured having a high cost for creating deployment environments to be the most likely reason for having the anti-pattern. Aside from the irrational ones, such as "we can't have artifacts with release-format version numbers which haven't passed QA" or "we have to make sure we don't break staging."

I suppose in theory we could use feature branches for development, merge to development for QA testing, then when stuff is approved merge from the feature branch to master. But I just don't see a benefit to that approach, besides burning lots of developer hard drive space.

You could automate the creation and removal of deployment environments, thus allowing you to deploy multiple feature branches.

You could have feature branch builds take turns deploying and testing in that single not-staging environment.

Greybeard

@magus said in Branches, Builds, and Deployment:

@greybeard said in Branches, Builds, and Deployment:

Sure, when a change to a service doesn't break its own tests but does break the tests of its clients one has to do something to get some confidence that breakage would get noticed, diagnosed, and reported.
But why tie that to the length and boundaries of a sprint?

Because that's the point of sprints?

The point of sprints is to put a limit on requirements churn.

For every bit of code to remain in QA for an entire sprint before promotion then any change would have to enter QA and wait until the end of the current sprint. Then that entire artifact would have to remain in QA without change for the entirety of that subsequent sprint. It could then be promoted in the second sprint after it was last changed.

There is nothing about the point of sprints that prevents QA from being scheduled to start testing the change in the same sprint as it was made or which invalidates any testing done before the subsequent sprint boundary.

Greybeard

@sockpuppet7 said in Branches, Builds, and Deployment:

It's buying you that this way the master branch is a place where you'll apply hotfixes if needed.

Nope, hotfixes have to go through develop as well. If develop is busted, ain't no fixes going to production.

It's not the only way for doing that, but I don't see how other methods would be significantly better than that.

Hotfix goes into its own feature branch. Integration tests on the build from that branch pass, code review approved, merge to master.

Greybeard

@kian said in Branches, Builds, and Deployment:

I should clarify we don't have a service but rather an on premise product

For our legacy on-premise product I got rid of the master branch.

Used to be when we thought we were close to release we'd cut a release branch from master. People wouldn't get the memo and would commit code intended for the release into master after the release was branched.

So next release after cutting the release branch I deleted master. After that we would branch the next release off of the pending release whenever we were ready to start work on the next release. As people usually knew which release their change was intended for they would know which branch to commit into.

And we had automated merges from earlier pending release branches into later ones.

Magus

@greybeard said in Branches, Builds, and Deployment:

The point of sprints is to put a limit on requirements churn.

Not only that, in fact.

@greybeard said in Branches, Builds, and Deployment:

For every bit of code to remain in QA for an entire sprint before promotion then any change would have to enter QA and wait until the end of the current sprint. Then that entire artifact would have to remain in QA without change for the entirety of that subsequent sprint. It could then be promoted in the second sprint after it was last changed.

I have no idea what you're talking about.

@greybeard said in Branches, Builds, and Deployment:

There is nothing about the point of sprints that prevents QA from being scheduled to start testing the change in the same sprint as it was made or which invalidates any testing done before the subsequent sprint boundary.

And we do that. What's your point?

@greybeard said in Branches, Builds, and Deployment:

Nope, hotfixes have to go through develop as well. If develop is busted, ain't no fixes going to production.

That only works on products that only have one deployed version. Which is true of most web apps, but not even all of them.

dkf

We are developing a system that users install through specifically versioned releases, so we don't need to strictly keep master working. But we still find that highly desirable; we work in feature and bugfix branches (which do not need to build with every commit) and require that build/unit-test pass only at code review time (which immediately precedes the merge to master). Our integration tests are tricky (custom hardware in a nontrivial deployment) and slow (over a weekend), so we can't run them anything like as often. When we are going to do a release, we fork off a release branch. Those tend to be pretty short as we try to keep master working.

In terms of tech, we use git (with history rewriting disabled on shared branches) and GitHub, running our unit tests via Travis. Our integration tests don't work there; network trickiness means that individual test cases time out the build. They go much faster when we keep them on-site.

PleegWat

@dfdub said in Branches, Builds, and Deployment:

@pleegwat We may work for the same company. Hello potential co-worker.

Possibly, though I also figure there may be a number of large companies where one could tell such a story.

sockpuppet7

@greybeard said in Branches, Builds, and Deployment:

Nope, hotfixes have to go through develop as well. If develop is busted, ain't no fixes going to production.

That's not how gitflow works. You create a hotfix branch from master, and when it's done you merge it both into master and develop.

dfdub

@pleegwat said in Branches, Builds, and Deployment:

Possibly, though I also figure there may be a number of large companies where one could tell such a story.

Every large company that already existed before git probably has a homegrown VCS somewhere. The details you mentioned, however, are quite specific - that would be one hell of a coincidence.

I'm glad I use git for most of my work and only have to use that system every once in a while.

Unperverted Vixen

@greybeard said in Branches, Builds, and Deployment:

It kinda invalidates the testing that happened against develop.

Stuff gets tested again after the merge.

Plus a problem with one change in develop has a good chance of preventing anything else from going forward.

Not true, in my experience.

You could automate the creation and removal of deployment environments, thus allowing you to deploy multiple feature branches.

I suppose that's technically possible, if we requisitioned a bunch more hardware (we might be able to make existing web servers work since the sites won't be in use simultaneously, we hope, but we'd have to set up another 90+ SQL servers with 4 TB of SAN each). But as the person who'd be responsible for automating environment setups/teardowns, as well as all the build changes, and automated tooling to update configurations to match the temporary environments: .

I mean, if I saw the benefit to it, my attitude might be different, but it really does seem like a lot of work for no gain. (I actually think things would be worse - you could have contradictory changes in different feature branches and not realize the issue until after they both went to master.)

You could have feature branch builds take turns deploying and testing in that single not-staging environment.

That's not feasible - there's too many changes to be able to serialize them like that.

Kian

@greybeard said in Branches, Builds, and Deployment:

Used to be when we thought we were close to release we'd cut a release branch from master. People wouldn't get the memo and would commit code intended for the release into master after the release was branched.

Well, watching over the repo and ensuring people commit to the right place is part of my job description, so that problem is less of an issue. Master and develop branches are protected so code only gets into them through PRs, which I review. But we're a small team, not sure how well that scales.

Speaking of, I do have a question about how people handle a particular issue, which is one reason I do like more lineal histories: commits that can merge without conflict but that break the build. Say you have your development branch (whatever it is called), and two people start branches from the same commit, A and B. A changes the contents of a file included by B, but that B doesn't modify, checks his changes work where required, and merges it back to the development branch. B meanwhile is developing against the original file A changed, and when he finishes his changes he sees his branch builds fine, and there are no conflicts, so he merges as well. But because he coded against an old version of a file, the build breaks.

If we want to keep a more honest history, we could demand that before merging your branch into the development branch, if there have been commits done to it you merge back into yours, solve any issues that may arise, and then merge back into development. This keeps the "train of thought" of the developer more clear, as you can see more clearly what he knew when he made a certain commit and why a later fix might be required, and if you have a rule of "only make working commits to the branch", all your commits will continue to build as they did when you wrote them. An alternative is to require branches to be rebased and tested before merging back, so that the history is essentially lineal, and you will always have the latest working code at the root of your branch. But that might break older commits in your branch, which relied on things that were changed but that now count as part of your history. It would be strange to see changes that assume things that are no longer true at the point where the commit appears, but that were true when the developer wrote them.

dkf

@kian said in Branches, Builds, and Deployment:

Speaking of, I do have a question about how people handle a particular issue, which is one reason I do like more lineal histories: commits that can merge without conflict but that break the build. Say you have your development branch (whatever it is called), and two people start branches from the same commit, A and B. A changes the contents of a file included by B, but that B doesn't modify, checks his changes work where required, and merges it back to the development branch. B meanwhile is developing against the original file A changed, and when he finishes his changes he sees his branch builds fine, and there are no conflicts, so he merges as well. But because he coded against an old version of a file, the build breaks.

We guard against that by using the dual build system of GitHub/Travis. It does one build on the originating branch, and another build that is as if it is on master with the merge done (technically, this is by doing the merge locally to Travis and never pushing the result upstream). It's not perfect, as you can still have problems when you want to coordinate a set of pull requests across a group of repositories (e.g., one to alter an API in a library, and another to update the client of that library to use the API in the new way) but most of the time that isn't a big problem. Especially if you use the same branch name across each of the repositories.

We aren't interested in having a lineal history. The pattern of commits is more valuable to us than having things “clean” as it lets us also do things like tracking aggregate activity and so on. Life isn't linear.

HardwareGeek

@magus said in Branches, Builds, and Deployment:

we were thrown multiple bad states.

California, New York, New Jersey, ...?

boomzilla

@kian this is where I like hg. A commit like that would create an anonymous branch, since the parent of the cookie is not the head of the branch. Then you need to take the step of merging with the head at which point you discover the problems and fix them and you still have a record of the original commit.