The release - post mortem



  • The other day, I wrote about a release gone wrong. I thought you might appreciate the post mortem.

    After the last release, the senior managers decided they didn't want a repeat disaster of nobody knowing what anyone else was doing, so they hired a release manager for each team. They were supposed to coordinate, produce a dependency graph, and come up with a department-wide release plan.

    They failed.

    This release was as bad as the last one. Although it finally got straightened out (Monday morning at 3:30AM), until then, nobody knew if they were going to make it in time to keep all of our customers from being offline on Monday.

    The senior managers were not happy with the non-stop emails about all the problems for 48 straight hours so they ordered all the managers/release coordinators into a room at 8AM today to explain.

    This meeting resulted in a decision to hire a SENIOR release manager to coordinate the release managers who were coordinating the team managers who were coordinating the release.

    ...because yet another layer of beaurocracy is what will make this better.



  • Layer upon layer of managemen
    ?



  • @snoofle said:

    ...because yet another layer of beaurocracy is what will make this better.
     

    A fog hangs over the chain of command. The only visible thing is a bunch of uncoordinated people (the release managers).

    What lies beyond the fog is invisible, undetectable and thus of no consequence.



  • @pnieuwkamp said:

    Layer upon layer of managemen
    ?
     

    Funny thing; where I work there's pretty much a 1:1 ratio between programmers:everyone else



  • To heck with the billing practices at your client, I have the feeling you could make some real money by shorting the stock of your customers about the time each release comes out.



  • @snoofle said:

    ...because yet another layer of beaurocracy is what will make this better.
     

    To be fair, it's not another layer (of  bureaucracy?), it's a missing layer - one that should have been there in the first place.

    All of the release managers should project-manage the release activities in their group and establish dependencies, pinch points and checkpoints, but there should have been a programme manager accountable for overall transition planning, getting the RMs to play nicely together. That role doesn't play a part in any of the lower-level tasks, they interface with the RMs to ensure that part gets done.

    Not sure the title's correct, but I hope that senior RM is only managing the RMs and not their own particular release.

    At least that way there's only one person accountable for the fuckup final outcome.



  • @dhromed said:

    where I work there's pretty much a 1:1 ratio between programmers:everyone else
     

    Just like that diagram. One programmer requires:

    • one HR manager,
    • one marketing manager,
    • one security ....

  • Trolleybus Mechanic

    @Cassidy said:

    @dhromed said:

    where I work there's pretty much a 1:1 ratio between programmers:everyone else
     

    Just like that diagram. One programmer requires:

    • one HR manager,
    • one marketing manager,
    • one security ....

     

    I can't find the original, but in the off chance this hasn't slid through your inbox at some point since they turned on the Internet--

    [url="http://www.slideshare.net/AmSam/the-ant-and-the-lion-rethink-about-your-value-chain"]The Ant and the Lion[/url]

     



  • @Cassidy said:

    @snoofle said:

    ...because yet another layer of beaurocracy is what will make this better.
     

    To be fair, it's not another layer (of  bureaucracy?), it's a missing layer - one that should have been there in the first place.

    All of the release managers should project-manage the release activities in their group and establish dependencies, pinch points and checkpoints, but there should have been a programme manager accountable for overall transition planning, getting the RMs to play nicely together. That role doesn't play a part in any of the lower-level tasks, they interface with the RMs to ensure that part gets done.

    Not sure the title's correct, but I hope that senior RM is only managing the RMs and not their own particular release.

    At least that way there's only one person accountable for the fuckup final outcome.

     

    Future press release from snoofle's client:

    "We apologize for the fault in this release our our software. The senior release-manager has sacked all the release managers responsible for the software development"

    Following the next revision:

    "We apologize again for the fault in our software. The senior release-manage responsible for sacking all the release managers has been sacked".

     

     



  • @Cassidy said:

    All of the release managers should project-manage the release activities in their group and establish dependencies, pinch points and checkpoints, but there should have been a programme manager accountable for overall transition planning, getting the RMs to play nicely together.
     

    You are trolling, right? Please tell me you're trolling! I cannot stand the the idea that someone could really think like this, blurb out this language and still hang around The Daily WTF. That would debase this site.



  • Why hire a release manager for each team? Why not a single one? What kind of release process are you using?

    Listen, if your release takes more than 5 scripts and three hours, you're doing it WRONG WRONG WRONG!!!



  • @ubersoldat said:

    Listen, if your release takes more than 5 scripts and three hours, you're doing it WRONG WRONG WRONG!!!

    This. What are you building, a Space Shuttle?



  • @ubersoldat said:

    Listen, if your release takes more than 5 scripts and three hours, you're doing it WRONG WRONG WRONG!!!
     

    One release does... but if you repeatedly miss these small releases, they all eventually backlog into one HUGE deployment.



  • @ubersoldat said:

    if your release takes more than 5 scripts and three hours, you're doing it WRONG WRONG WRONG!!!

    Releasing my product is running one directory of 1+ sql scripts, and one shell script on each machine to pull the stuff from svn. Having done it personally, it took me 8 minutes.

    I am no longer allowed to log into the production machines (only dedicated production support folks are allowed access); we have SAs and DBAs for that. So I provide step by step instructions including cut-n-paste commands for the monkeys, and sit on the phone just in case they can't handle cut-n-paste, or log into the wrong machine...

    It's the six other teams that don't seem to understand how to release their own products, let alone how to make sure they play nice with all the connecting parts.

    @Others: the release managers for each team were orignally charged with coordinating stuff. They didn't. The new RM boss is now supposed to do that for them.  Personally, I figure any 15 year old with moderate organization skills could do the job, but we need professionals!


Log in to reply