The darker side of sharing...



  • Most places I've worked have small, individual databases for each developer. There's a master script that can create the db to the last deployed configuration, and populate it with an account or two of obfuscated data from production. This way, you can just drop your database, run the script, apply any changes since the last release, and be back at a known configuration in a few minutes. Useful for recovering from runaway code, sql statements, and it insulates you from changes made by others.

    At this place, that would cost too much (I don't see how as it's the same amount of space we already share, just divied up as separate small databases). As such, all the development teams share one development database, and one test database. I successfully lobbied to get separate accounts designated for testing by each individual so as to avoid your test data causing my test to change.

    My account happened to be our biggest customer (load testing), and so has over 1BB records in the table.

    A week ago I was asked to test something slated for release today. For seven days, I've been going nuts chasing down seemingly random behavior. Finally, late this afternoon, I found the cause.

    Some moron on another team decided not to use the test account assigned to them, preferring instead to use the test account assigned to me. Without telling me. In an attempt to keep us from noticing the changed data, he reset the runId and last-updated timestamp to the values before he changed things.

    He forgot that our application uses accumulated records to make decisions, so the thousands of records he added caused all sorts of, what to me was, unexpected behavior.

    I hadn't figured it out by cutoff time, so the deployment was cancelled. This caused eight other dependent applications to cancel their deployments too.

    The auditors were emphatic it was due to a lack of testing, and this sort or instability would not be tolerated.

    Since I was supposedly running in an exclusive environment, I had no reason to think there would be, let alone go looking for thousands (among > one billion) of randomly added records that would affect my result. However, I had eliminated pretty much everything else, and on a hunch, I did a select-count by runId. There were more records than I expected. Further digging showed the extra records with the correct timestamp (so they looked like they were part of my run). What this guy didn't know about was that there is a source column, which indicated the data came from his system and not mine.

    In spite of the DBAs objecting, we will be getting our individual databases.



  • Not ms-access I hope :/



  • @snoofle said:

    Someone mucking about with wrong accounts... 

    What this guy didn't know about was that there is a source column, which indicated the data came from his system and not mine.

    OK, 1. How was this guy able to run as your account?  How did he get your password?  and 2. How did you not murder him when you found out what he had been doing?



  • @snoofle said:

    has over 1BB records in the table.

    So, over 443 records? Oddly specific choice of number.



  • @lolwtf said:

    @snoofle said:

    has over 1BB records in the table.

    So, over 443 records? Oddly specific choice of number.
    I am also wondering how many records "1BB records" is. One ByteyByte? One Billion Bytes? One BiByte? One Baby-Baby? One Whatever?


  • Considered Harmful

    @TheRider said:

    @lolwtf said:
    @snoofle said:

    has over 1BB records in the table.

    So, over 443 records? Oddly specific choice of number.
    I am also wondering how many records "1BB records" is. One ByteyByte? One Billion Bytes? One BiByte? One Baby-Baby? One Whatever?

    I first read it as "one billion billion", or 1,000,000,000². My jaw dropped a bit, until I realized the second B wasn't intended.



  • He must have saved the paper with all the user/password combinations he was given when they set up the accounts.

    What? You've never seen a list like that being casually handed out to everyone on the list? It is best practice here at my local university.



  • @TheRider said:

    @lolwtf said:
    @snoofle said:

    has over 1BB records in the table.

    So, over 443 records? Oddly specific choice of number.
    I am also wondering how many records "1BB records" is. One ByteyByte? One Billion Bytes? One BiByte? One Baby-Baby? One Whatever?
     

    Pretty sure I once ran across some table of Roman numerals that said B was used for 200.  It was a rare oddball part of the Roman empire or something, hadn't been distributed much beyond a limited circle of people, and I don't think they'd bothered to work out how it was used in subtractive form (was BD=500-200=300, BC=200+100=300, BB=200+200=400?  who knows?) but it had existed at one time.

    So "1BB records" might be "1x200x200" or "1,200,200".  Either of which could probably be described more easily in another way.

     



  • @snoofle said:

    Some moron on another team decided not to use the test account assigned to them, preferring instead to use the test account assigned to me.
    So what did he have to say for himself? Please tell me someone will rip him a new one.

    By the way if you have an old PSU lying around, you can make a pretty effective weapon out of it simply by grabbing the cables and swinging it around.



  • @snoofle said:

    The auditors were emphatic it was due to a lack of testing, and this sort or instability would not be tolerated.

    They're partly right.

    It's a lack of decent quality testing  that led to this situation, and that was down to the quality of the testing environment.

    I suppose the real question wasn't "how did it happen" but "how was it permitted to happen".

    @snoofle said:

    At this place, that would cost too much

    However, the cost of aborting the deployment of eight other dependent applications is still negligable compared to the cost of building and maintaining separate dev DBs, right?



  • @snoofle said:

    In an attempt to keep us from noticing the changed data, he reset the runId and last-updated timestamp to the values before he changed things.
     

    In other words, he knew he was doing something wrong and actively tried to cover his tracks. There is a short word for that: fraud.

    Did your HR or legal departments get involved? If so, what was the result? If I were a manager I'd be pushing for dismissal.

     



  • @all:

    Sorry, typo: 1BB was supposed to be one billion.

    By "account", I meant customer account (e.g.: you use customer A, and I use customer B). We no longer use group DB logins, you can only log in using your own. However, since we all run the applications against the same tables, we all have the same permissions.

    Our DBAs (about whom I've frequently stated that they don't seem to answer to anyone) are the ones who said it was too "expensive" to set up individual DBs for each developer (translation: we can't be bothered). The bean counters are the ones who will need to deal with the lost-opportunity for the delayed deployment. I haven't had the pleasure of seeing their reaction yet, so... TBD

    I view this as an opportunity to leverage the power of the auditors to get what I want (what we really should have had all along) as all I can do is ask, but they can mandate.

    As for how the sh** will hit the fan, this all triggered at the close of business last Friday, so I presume there will be assorted meetings and finger pointing come Monday. I took a data dump of what he did to cover my butt, so while I hope they make an example of this guy (they probably won't), I don't really care; it's just more billable time for me.



  • You don't really care?

    I'm not that fond of useless meetings :/



  • @snoofle said:

    Sorry, typo: 1BB was supposed to be one billion.
     

    Wait, is that an English billion, or an American billion?



  • @ASheridan said:

    Wait, is that an English billion, or an American billion?
     

    The common English billion == American billion, is it not?

    As for the rest of Europe, we still do million, milliard, billion, billiard, etc.



  • @ASheridan said:

    @snoofle said:

    Sorry, typo: 1BB was supposed to be one billion.
     

    Wait, is that an English billion, or an American billion?

    It's an Imperial billion, you know, because us Americans don't do metric billions...


  • @dhromed said:

    As for the rest of Europe, we still do million, milliard, billion, billiard, etc.

    Still knocking those balls around all day, I see.



  • @dhromed said:

    @ASheridan said:

    Wait, is that an English billion, or an American billion?
     

    The common English billion == American billion, is it not?

    As for the rest of Europe, we still do million, milliard, billion, billiard, etc.

    The Americans do billiards, too, but they prefer to call it "shooting pool".

     



  •  If I had a billiard I'd be shooting pool all fucking day.


  • Discourse touched me in a no-no place

    @dhromed said:

    @ASheridan said:

    Wait, is that an English billion, or an American billion?
     

    The common English billion == American billion, is it not?

    Yes - since the 1970s



  • @snoofle said:

    Our DBAs (about whom I've frequently stated that they don't seem to answer to anyone) are the ones who said it was too "expensive" to set up individual DBs for each developer (translation: we can't be bothered). The bean counters are the ones who will need to deal with the lost-opportunity for the delayed deployment. I haven't had the pleasure of seeing their reaction yet, so... TBD

     

    As a DBA I actually like it when dev's have their own local databases.  Helps keep things clean and less work for me in the long run. 

    I'm also a big fan of storing database assetts in source control and setting up continuous integration.  In other words a dev updates an object, say adding a column, and checks in the definition script to source control.  A tool is then used to auto generate a deployment script based on the version history, and is then run on a test environment. Boom we've taken the human equation out of the deployment. At least for the most part.  Plus the deployment script has at least received some testing.

    The one thing I wish though is that more devs knew how to read an execution plan.

     

     



  • Show of hands, who read the title and thought this was another Swamp thread?



  • @PJH said:

    Yes - since the 1970s
     

    Oh, I thought it was since like the 1790s.



  • @morbiuswilters said:

    Show of hands, who read the title and thought this was another Swamp thread?
     

    Only you.



  • @dhromed said:

    @morbiuswilters said:

    Show of hands, who read the title and thought this was another Swamp thread?
     

    Only your avatar.

    FTFY.



  • @ASheridan said:

    @snoofle said:

    Sorry, typo: 1BB was supposed to be one billion.
     

    Wait, is that an English billion, or an American billion?

    "African or European?", I think you mean.

     


  • :belt_onion:

    @dhromed said:

    @morbiuswilters said:
    Show of hands, who read the title and thought this was another Swamp thread?
     

    Only you.

    I definitely assumed it was, but was not too disappointed to see a snoofle post instead.

     



  • @heterodox said:

    @dhromed said:

    @morbiuswilters said:
    Show of hands, who read the title and thought this was another Swamp thread?
     

    Only you.

    I definitely assumed it was, but was not too disappointed to see a snoofle post instead.

     

    Swamp doesn't post in this forum anyway.



  • @lettucemode said:

    Swamp doesn't post in this forum anyway.

    He has (the uber thread started its life here) but his threads are moved to Funny Stuff and merged with the uber thread.


Log in to reply