We Got Busy



  • We just pushed a new release to production during a planned outage. During the smoke test, approximately half of the new functions didn't seem to work at all. We checked and verified that all the correct code, etc was in the build tree pulled from source control. We checked and verified that all the correct sql files were provided to the DBAs for installation.

    It turns out they ran all the DDL (structurally required) but only some of the DML (technically required but the DB won't throw errors if it's not run, you'll just get incorrect data).

    Why?

    "We got busy so we left it for later"

    When the hell is "later"?

    "We figured we'd do it tomorrow" (during business hours).

    No measure of business process is going to be able to counter the volume of sheer stupidity running rampant around here.

     



  • Automated deploy? Continuous deploy? And what exactly happened after? Did they run the DML? Or did you have to roll back?
    Surprising to me that you had to check the correct code. If you'd pushed to a staging server first, you'd at least know that the code you intend to push to prod is the correct stuff.
    In my experience, a couple of beers or donuts applied at the right time will ensure that your devops people are doing their stuff in a timely manner.



  • Daniel: "Yeah, our faucet's really leaking. Can you come fix it?"
    Mr. Miyagi: "Aye."
    Daniel: "Well, can I tell my mom when?"
    Mr. Miyagi: "When what?"
    Daniel: "When are you going to fix the faucet?"
    Mr. Miyagi: "After."
    Daniel: "After what?"
    Mr. Miyagi: "After after!"

     (From The Karate Kid, though if you don't know that I'm not sure you should be allowed to live in America....)


  • Java Dev

     Yay, I'm not allowed to live in America!



  •  Hm, this makes me wonder: Was there no specific communication with the DBAs telling them what to do and when to be finished? If they got all the information and got "busy" anyway, they should get a lot of spare time instead. ;-)



  • @t_wheeler said:

    (From The Karate Kid, though if you don't know that I'm not sure you should be allowed to live in America....)

    There is only one scene worth remembering in that movie, it's when the guy drives over a ghettoblaster with his bike when they crash the beach party.



  • Maybe the payroll department should get too "busy" and forget to cut their checks.



  • @t_wheeler said:

    Daniel: "Yeah, our faucet's really leaking. Can you come fix it?"
    Mr. Miyagi: "Hai."
    Daniel: "Well, can I tell my mom when?"
    Mr. Miyagi: "When what?"
    Daniel: "When are you going to fix the faucet?"
    Mr. Miyagi: "After."
    Daniel: "After what?"
    Mr. Miyagi: "After after!"

     (From The Karate Kid, though if you don't know that I'm not sure you should be allowed to live in America....)

    Nitpick

    He was Japanese (Okinawan), not Scots-Irish



  • I just got tired of this and modified our app so that it checks for a version number in the db and automatically runs all DDL and DML required on the first run of any updated software . There were lots of complaints by the db department when I suggested this, about how it would be innapropriate to run the scripts automatically and that manual tweaking was always necessary, and that the scripts needed to be run in the right order and yaddi yadda. So we just implemented it in the core team and deployed the changes without announcing it. Has worked flawlessly the last 18 months, and we haven't had a single complaint or problem caused by this procedure. And before we automated the db updates we were constantly having problems caused by PEBCAC updates.



  • @DrPepper said:

    Automated deploy? Continuous deploy? And what exactly happened after? Did they run the DML? Or did you have to roll back? Surprising to me that you had to check the correct code. If you'd pushed to a staging server first, you'd at least know that the code you intend to push to prod is the correct stuff. In my experience, a couple of beers or donuts applied at the right time will ensure that your devops people are doing their stuff in a timely manner.
    We don't have staging servers (they cost money). We just push from the blessed label in source control to prod. Hokey? Yes, but it can work if everybody follows procedures.

    I personally always do a quick test on whatever I deploy as a sanity check (does it at least come up and put through a trivial transaction?)

    I forced them to push the DML right then and there. Once it was done, everything worked fine.



  • @snoofle said:

    I forced them...

    Please tell us that this procedure involved the ClueBat(tm).



  • Ah, DBAs. The most useless employees in DevOps.



  • @Obfuscator said:

    I just got tired of this and modified our app so that it checks for a version number in the db and automatically runs all DDL and DML required on the first run of any updated software . There were lots of complaints by the db department when I suggested this, about how it would be innapropriate to run the scripts automatically and that manual tweaking was always necessary, and that the scripts needed to be run in the right order and yaddi yadda. So we just implemented it in the core team and deployed the changes without announcing it. Has worked flawlessly the last 18 months, and we haven't had a single complaint or problem caused by this procedure. And before we automated the db updates we were constantly having problems caused by PEBCAC updates.

    This is basically what I do. The deploy process should automatically run the patches. Simple.



  • @Obfuscator said:

    the scripts needed to be run in the right order and yaddi yadda.

    Easy fix.

    
    short cut = 0; 
    
    try {
            
            foreach(var script in sqlScripts) { 
    
              try {
                cut++;
    
                if(!script.executed)
                  scriptExecutable.execute(script);
    
                script.executed = true;
              } 
              catch(OverflowException) {
                throw;  //we won't have endless loops
              }
              catch(Exception)
              { 
                //thanks for playing. try again
              }
    
           }
    
    } 
    catch (OverflowException)
    {
      Console.write("Check your scripts moron");
    }
    finally {
      Console.write("DONE!");
    }
    
    

    Edit: the inner try is in the wrong place but you get the idea



  • @snoofle said:

    We don't have staging servers (they cost money).

    Better than spending money on staging, only to have it mirror dev instead of prod?



  • I was going to write a devil's-advocate defense of the DBAs, on the basis that they might not have had the importance of the DML explained to them, and as non-developers they couldn't be expected to infer that on their own. But no. I can't defend people who go "off piste" without first attempting to acquire an understanding of the risks they are taking on. Technical work requires a rigorous approach, and people who are willing to improvise like this in a production environment are nothing but a fucking liability.



  • @GNU Pepper said:

    I was going to write a devil's-advocate defense of the DBAs, on the basis that they might not have had the importance of the DML explained to them, and as non-developers they couldn't be expected to infer that on their own.

    That's not even acceptable behavior for your grandma. If you give her two patches and tell her to apply them while you make yourself a drink from the 30-year-old vodka under her sink and some of her glaucoma medication, and she only applies the first, somebody is getting the dentures slapped out of their mouth.

    And DBAs absolutely should know what DML is and why a DML patch would be necessary as part of a release. A DBA who doesn't understand that shit isn't a DBA, he's a goddamn lobotomy patient who accidentally opened the RDBMS management tool when he was looking for Spider-fucking-Solitaire.



  • @morbiuswilters said:

    @GNU Pepper said:
    I was going to write a devil's-advocate defense of the DBAs, on the basis that they might not have had the importance of the DML explained to them, and as non-developers they couldn't be expected to infer that on their own.

    That's not even acceptable behavior for your grandma. If you give her two patches and tell her to apply them while you make yourself a drink from the 30-year-old vodka under her sink and some of her glaucoma medication, and she only applies the first, somebody is getting the dentures slapped out of their mouth.

    And DBAs absolutely should know what DML is and why a DML patch would be necessary as part of a release. A DBA who doesn't understand that shit isn't a DBA, he's a goddamn lobotomy patient who accidentally opened the RDBMS management tool when he was looking for Spider-fucking-Solitaire.

    If DBAs and SAs don't push back, developers send all kinds of stupid shit in production then go home for the weekend, leaving undocumented chaos behind. The day you get the pager is the day you decide what to deploy.



  • @Ronald said:

    If DBAs and SAs don't push back, developers send all kinds of stupid shit in production then go home for the weekend, leaving undocumented chaos behind. The day you get the pager is the day you decide what to deploy.

    First off, as someone who does get paged, I do care about making sure deployments go well. But the morons applied half the fucking updates. It's not like they said "No, we'll deploy the whole thing later." They half-deployed, then apparently got confused and wandered off.



  • This kind of DBA behaviour reminds me of a rather excellent computer science lecture given by the legendary Richard Feynman:

    [url=https://www.youtube.com/watch?v=EKWGGDXe5MA]https://www.youtube.com/watch?v=EKWGGDXe5MA[/url]

    It's about an hour long, and has content that I'm sure will be a bit pedestrian for you guys (though you might be entertained by his excellent style of explanation).


    He has an analogy for how computers work, aimed at complete novices, where he draws parallels between a CPU and a really dumb (but fast) filing clerk.



    These sorts of DBAs remind me of those filing clerks. Except for the speed... they're dumb but slow.



  • Will there be a public execution? If not why not?



  • @eViLegion said:

    https://www.youtube.com/watch?v=EKWGGDXe5MA

    It's about an hour long, and has content that I'm sure will be a bit pedestrian for you guys (though you might be entertained by his excellent style of explanation).

     

    Saw that video in a recent feynman binge of mine. It's super-excellent.



  • @dhromed said:

    recent feynman binge of mine.

    I see..



  •  He's a feyn man, if you catch my drift.



  • @morbiuswilters said:

    @Ronald said:
    If DBAs and SAs don't push back, developers send all kinds of stupid shit in production then go home for the weekend, leaving undocumented chaos behind. The day you get the pager is the day you decide what to deploy.

    First off, as someone who does get paged, I do care about making sure deployments go well. But the morons applied half the fucking updates. It's not like they said "No, we'll deploy the whole thing later." They half-deployed, then apparently got confused and wandered off.

    Yeah, there's a huge difference between pushing back because code doesn't meet production standards, versus deciding not to run scripts because of being "busy". The former I expect and, on occasion, appreciate; the latter, not so much.



  • @morbiuswilters said:

    @GNU Pepper said:
    I was going to write a devil's-advocate defense of the DBAs, on the basis that they might not have had the importance of the DML explained to them, and as non-developers they couldn't be expected to infer that on their own.

    That's not even acceptable behavior for your grandma. If you give her two patches and tell her to apply them while you make yourself a drink from the 30-year-old vodka under her sink and some of her glaucoma medication, and she only applies the first, somebody is getting the dentures slapped out of their mouth.

    And DBAs absolutely should know what DML is and why a DML patch would be necessary as part of a release. A DBA who doesn't understand that shit isn't a DBA, he's a goddamn lobotomy patient who accidentally opened the RDBMS management tool when he was looking for Spider-fucking-Solitaire.

    As a DBA and reformed developer, I actually agree. A DBA shouldn't not get to pick and choose pieces of a deployment as part of an approved change. They should however have knowledge of DML and be able to voice their concerns during the development or approval phase of a change.

    And yes there are a ton of bad DBAs out there. But the good ones take the time to understand the developers needs.



  • re: DBAs deciding what to deploy:

    In this case, the DDL *and* DML were created by the application (as opposed to system) DBAs, and was all tested, QA'd and approved in advance (something new around here). The system DBAs just got distracted half way through, never got back to it and left everyone hanging.



  • @snoofle said:

    re: DBAs deciding what to deploy:

    In this case, the DDL and DML were created by the application (as opposed to system) DBAs, and was all tested, QA'd and approved in advance (something new around here). The system DBAs just got distracted half way through, never got back to it and left everyone hanging.

    As the expression goes: never ascribe to malice that which is adequately explained by incompetence.


Log in to reply