Oops...



  • One of my junior peers decided he wanted to learn emacs, so he began to use it. Naturally, he was always checking the man pages to see how to do this and that.

    This morning, he was adding some environment variables (for a new release) into our *many* config files in the $ROOT/deploy/conf tree for deployment. Somewhere along the line, he forgot which buffer contained which config file and without realizing it, saved the contents of one of the production config files to the development config file and vice versa.

    We're in the 3rd week of "QA" which means we're frantically trying to implement the features for which we just received requirements three days ago; ten of us are banging away. Inserting test data. Doing updates. Running massive queries on tables that supposedly only had a few hundred thousand rows (as opposed to the production tables with billions of rows).

    We were even truncating tables to get rid of data so that we could start clean.

    Fortunately, the only tables being truncated were the new ones for this release, so no actual data was lost.

    However, the customer service reps were swamped with calls complaining that our supposedly real time system was very slow. Tech support tracked it back to the database being bogged down by huge queries.

    We're lucky nobody tried to get rid of the test data from our main tables by doing a truncate....

    Fun times.



  • Moral of this story: always diff before commit.



  • TRWTF is using emacs. Now, let him use vim.



  • @ubersoldat said:

    TRWTF is using emacs. Now, let him use vim.
     

    Not unless it has Tetris, Eliza, and a Mayan calendar built in.



  • @da Doctah said:

    @ubersoldat said:

    TRWTF is using emacs. Now, let him use vim.
     

    Not unless it has Tetris, Eliza, and a Mayan calendar built in.

     

    Let's see ...

    • TeTrIs.vim : A tetris game in pure vim, at http://www.vim.org/scripts/script.php?script_id=172  

    • The Mayan calendar is easily available in the clock applet ... copies the date to the clipboard with a single click, too.
    • Eliza: vim users don't need that much psychological support, so it hasn't been necessary yet

    Conclusion? Emacs is a good OS, it just lacks a good editor .... (okay, that was really old ... so old like the old saying "eight megabytes and constantly swapping" ... today we'd need to say megamegabytes, as 8MB is not even small change anymore ;)

     

     



  • This is more of a failure of Dev/Prod separation than a text editor screw-up. The fact that developers have access to config files that allow write access to production would set off a thousand alarm bells where I work.



    We solve this problem in one of two ways:

    1. OS integrated authentication. The production process identity has the rights to production, we don't embed the username and password in a config file if it is at all possible to avoid it.
    2. Encrypted configs. When it is necessary to store a password in a config file and necessary for developers to work with them, we encrypt passwords and store the decryption keys on the production server in a location where developers don't have access to them.



      I've seen way too many instances of someone refreshing a dev system and accidentally pointing it to production to not be draconian about the enforcement of the above rules.


  • @snoofle said:

    One of my junior peers decided he wanted to learn emacs, so he began to use it. Naturally, he was always checking the man pages to see how to do this and that.


    I call fake. Emacs' man page has just the (short) list of commandline parameters, everything else is in Texinfo. (Reader of it, of course, is part of Emacs.)

    (Seriously, tho, I'm pretty sure snoofle just took the mental shortcut there. ;))



    And I doubly agree with Jaime; we DO have prod access, but we have separate user accounts for that. RunAs-ing an editor would be something you'd remebember doing... I think.



  • @pjt33 said:

    Moral of this story: always diff before commit.

    I would rather say: config is also code. At least in the sense that you should bundle conf and code and QA that in staging, rather than hope that changing conf between staging and prod won't affect the system.



  • Wait, maybe i misunderstood that. Did you mean that you changed the dev environment server addresses to point to the prod servers somehow? Do you not have any measures in place to prevent that sort of mistakes? Separate subnet or some deliberate manual step while deploying to prod?



  • All: We have a set of config files for each environment, all stored in svn. When we go to deploy, there's a script that yanks all the qa/prod versions (depending upon target environment) and merges them with the jars being released.

    In this case, he was intentionally adding new environment variables (that had been tested in the dev configs) to the corresponding configs for the other environments. These were then picked up as part of the build.

    They checked that the qa/prod files that were supposed to change were the ones that changed. They just didn't do diffs. Fortunately, what we were changing were new tables, so truncating an empty tables is a noop. It was the queries against fully loaded tables that caused the problem that led to finding the goof.

    Imagine if it hadn't been spotted; the next release would have had prod users pointing at the dev db. Lots of data would have been missing when they tried to access it.



  • @snoofle said:

    All: We have a set of config files for each environment, all stored in svn. When we go to deploy, there's a script that yanks all the qa/prod versions (depending upon target environment) and merges them with the jars being released.

    In this case, he was intentionally adding new environment variables (that had been tested in the dev configs) to the corresponding configs for the other environments. These were then picked up as part of the build.

    They checked that the qa/prod files that were supposed to change were the ones that changed. They just didn't do diffs. Fortunately, what we were changing were new tables, so truncating an empty tables is a noop. It was the queries against fully loaded tables that caused the problem that led to finding the goof.

    Imagine if it hadn't been spotted; the next release would have had prod users pointing at the dev db. Lots of data would have been missing when they tried to access it.

    I completely understood that. In my environment, switching the dev and production configs will result in an "access denied" error for both.



  • @Jaime said:

    I completely understood that. In my environment, switching the dev and production configs will result in an "access denied" error for both.
    Here too. But this is because there are completely separate servers and subnets and firewalls between them -- and developers don't even get close to prod servers or to the corresponding subnets.


Log in to reply