Give the keys to your data to an idiot



  • I was asked to chase down why a production run was failing. Ok, I'm not usually the one to do that sort of thing as I don't even have access to the production machines, but it was something to do.

    Since I can't even get onto the production boxes to look at logs, let along debug, I set up my environment to point at the production environment. I'd run my own instance of the server so I could turn up the logging level, and see what was actually happening. Yes, I know it's a wtf that a supposedly locked down environment can be accessed like this, but that's a whole other world-o-wtf.

    Long story short, I track it down to a timestamp that has about 12 decimal places of non-zero digits after the milliseconds. Hmmmm, normally, we only store down to ms. After some digging and test queries, I prove that it's the extra digits that are being truncated by the driver, and so the sql update, which filters on the timestamp (that was read in from the same row), is mismatching because of the extra precision. 

    So how did those digits get altered? It's not the code; we never change them in the application. It's not any of the other applications as they don't even use that field. Ergo, it had to have been someone doing an update statement outside the code - by hand.

    But nobody except specific personnel are allowed to do that any more.

    Then maybe it's one of the specific personnel doing something they shouldn't.

    They ask, but nobody is owning up to it.

    I suggest they check the backup tapes to see when the data changed, then narrow it down by who logged in during that window.

    The intersection window whittles the list down to one name; a new DBA.

    "Oh that, yeah, see I was looking at a random record and tried an update, then must have forgotten to roll it back."

    Um, you do know that you're not actually supposed to change the data, right?




  • And that DBA was promptly given the boot...



  • How does your customer find these people?  Do they build a list of applicants from worst to best and then work down it till they hit some acceptable quality level (however low that may be)?



  • Not sure about the sensitivity of the data you're dealing with, but isn't perusing production data a WTF in and of itself, let alone updating it?  And based on how things go at your work, I know he was promptly reprimanded by being given a window office with a raise and an extra week of vacation, plus a lap-dance from the SVP's teenage daughter, right?



  • @Sutherlands said:

    And that DBA was promptly given the boot...

    Are you kidding? He was probably given an 'atta boy' for finally fessing up, and fast-tracked to promotion.

    This is Snoofle's company we're talking about, here.



  • @All: no, the new guy didn't get fired. No, they're not going to fire him.

    However, to cover my boss (the data is extremely sensitive in terms of privacy regulations), I made sure to be discussing it loudly enough to be overheard while walking past the auditor's open doors. They grabbed us and inquired. We told all. Let them handle it, or not as the case may be.

     



  • @snoofle said:

    I was asked to chase down why a production run was failing.
     

    @snoofle said:

    Since I can't even get onto the production boxes

    See, I'm not sure how you can allow this situation to arise.

    If they want you to do something, they should satisfy any pre-requisites you have to fulfil that assigned task.

    I've never heard of a detective being told that they should investigate and report upon a crime yet denied access to valuable information important to their inquiry. @snoofle said:

    "Oh that, yeah, see I was looking at a random record and tried an update, then must have forgotten to roll it back."

    Um, you do know that you're not actually supposed to change the data, right?

     

    Has anyone (or the auditors) asked how a new hire is granted this level of access? Failure in policy, or failure in HR?



  • @CodeNinja said:

    @Sutherlands said:
    And that DBA was promptly given the boot...

    Are you kidding? He was probably given an 'atta boy' for finally fessing up, and fast-tracked to promotion.

    This is Snoofle's company we're talking about, here.
    I see somebody doesn't read the tags.



  • @Sutherlands said:

    @CodeNinja said:
    @Sutherlands said:
    And that DBA was promptly given the boot...

    Are you kidding? He was probably given an 'atta boy' for finally fessing up, and fast-tracked to promotion.

    This is Snoofle's company we're talking about, here.
    I see somebody doesn't read the tags.



    Why would you assume that?

    Probably should have added a, 'Sarcasm', tag to my original post.



  • @Cassidy said:

    Has anyone (or the auditors) asked how a new hire is granted this level of access?
    He's a relatively new hire (about 3 months) whose job it is to support the production databases, so by definition he has access.

    Using it inappropriately, on the other hand...

    I have a clue bat, and though some folks around here kind of..., no, deserve it, I don't go around murdering people...



  • @Cassidy said:

    @snoofle said:

    I was asked to chase down why a production run was failing.
     

    @snoofle said:

    Since I can't even get onto the production boxes

    See, I'm not sure how you can allow this situation to arise.

    We didn't always have it like that. When we got swallowed by MegaCorp, as part of grabbing-the-reins, they removed production access from everyone except dedicated support people. This make sense from an auditing point of view. However, the support folks don't really have the expertise to diagnose certain kinds of stuff, so it's delegated to us. We're not supposed to be on those boxes - ever. Of course, in practice, we're given logins and passwords (by those who are supposed to have access) as their delegates to do the work.

    Stupid? You betcha.

    The first time it happened after the lock down, we couldn't get on so the problem festered for days. Our management unofficially decided to look the other way so that we could fix the problems.

    Now I get what I need to do the work, or obviously the work doesn't get done.

    Of course, the right way is to have support people who know what they're doing, but, after all, this is WTF-Inc.



  •  Snoofle, is your company hiring? In case you ever run out of WTF's to fix, I can throw some more in, keeping us both employed!



  • @snoofle said:


    Um, you do know that you're not actually supposed to change the data, right?


    But then, what's the point of having a DBA? 🙂

     



  • @tchize said:

    But then, what's the point of having a DBA? 🙂

    The DBA can change the structure. You know, like deleting a column of extremely-important customer data and then finding out that the backup system they're supposed to be looking after doesn't work?

    But then it seems Snoofle's company is used to sending out half a million letters to customers to ask them for their ZIP code again.


  • Winner of the 2016 Presidential Election

    @Qwerty said:

    But then it seems Snoofle's company is used to sending out half a million letters to customers to ask them for their ZIP code again.

    They can deliver mail without a ZIP code?


  • Discourse touched me in a no-no place

    @snoofle said:

    @All: no, the new guy didn't get fired. No, they're not going to fire him.

    However, to cover my boss (the data is extremely sensitive in terms of privacy regulations), I made sure to be discussing it loudly enough to be overheard while walking past the auditor's open doors. They grabbed us and inquired. We told all. Let them handle it, or not as the case may be.

     

    If it weren't for the fact that you're dealing with idiots, I'd assume that eventually people would learn that walking past the auditors with Snoofle is a bad idea.



  • @joe.edwards said:

    They can deliver mail without a ZIP code?
     



  • @joe.edwards said:

    @Qwerty said:
    But then it seems Snoofle's company is used to sending out half a million letters to customers to ask them for their ZIP code again.

    They can deliver mail without a ZIP code?

    I thought that was the joke as well...  If it was, I'm glad that you took the bullet instead.


  • @C-Octothorpe said:

    @joe.edwards said:

    @Qwerty said:
    But then it seems Snoofle's company is used to sending out half a million letters to customers to ask them for their ZIP code again.

    They can deliver mail without a ZIP code?

    I thought that was the joke as well...  If it was, I'm glad that you took the bullet instead.

    The joke was more the idea that you would need to ask, with the rest of the address you can just look up the ZIP code.



  • @locallunatic said:

    @C-Octothorpe said:

    @joe.edwards said:

    @Qwerty said:
    But then it seems Snoofle's company is used to sending out half a million letters to customers to ask them for their ZIP code again.

    They can deliver mail without a ZIP code?

    I thought that was the joke as well...  If it was, I'm glad that you took the bullet instead.

    The joke was more the idea that you would need to ask, with the rest of the address you can just look up the ZIP code.

    Seems you can deliver registered post without a signature, too (according to the other thread)

     


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.