The data is in which table?



  • A few weeks ago, I posted about a twit in another group who gave a demo, pointed to the production database and changed the configurations.

    Today, this same dimwit was giving part 2 of the demo, and had to (legitimately) point at production in order to show us how certain data was to be used for a report.

    Ok, reports are read-only so no harm no foul.

    Except that the client he randomly chose is one on which I happen to be currently working, and the data on-screen was wrong. Very wrong. I queried. He replied: we store that data in tables XXX and XXX_Hist. We had a problem in our code that accesses the primary table, so I just changed it to store the data directly into the history table. I'll fix it when I get a chance.

    Wait a minute; everyone uses your code for inserts. But everyone uses their own code for lookups and reports. Are you telling me that the data in the primary table is stale?  For how long has it been like this? A little over three months, but don't worry, all the data is in the history table.

    I grabbed the laptop from my boss, logged into production and did a quick comparison of all of the production reports that reference that table that go to clients for the past three months. They were all IDENTICAL! That never happens with data that changes daily.

    My boss and B+1 ripped into this idiot guy: you realize that we've been sending reports with stale data to our clients for three months!

    B+1 runs down the hall and brings back B+2 who goes absolutely bonkers, rips the guy a new one and fires him on the spot.

    Then he directs me and my boss to drop everything, pull all the data that's in the history table back into the primary table (which occurred without incident, so I don't know what the problem was in the first place), and rerun all the affected reports for the past three months. Then B+1 and B+2 would analyze and determine what remediation was necessary for the clients.

    Stuff like this always seems to involve offshore developers. Always.

    Why?



  • This is the same versioning system I use for my code! file.go, file_new.go, file-revision3.go, file--copy7.go, file-fix-bugfix-copy.go…



  • It just proves that the clients aren't reading their reports and you can stop producing them. Are these paper reports that go to hundreds of thousands of people like the last one? At least that one seemed to have a few people who read them and queried the company.

    @snoofle said:

    Then he directs me and my boss to drop everything,

    "Everything" being the emergency fix that you dropped everything else for last week?



  • @Qwerty said:

    you can stop producing them
    Not really. Some of them are real time (e.g.: MQ) messages that just feed blindly into other systems that use our data for reference data. Some of them take it as excel spreadsheets. Very few want the paper version.

    Only a few would be affected by the data from these tables as they were for a special type of client - from a new line of business we started about 5 months ago, so not heavily used. Nevertheless, they obviously want the data to be accurate as there is regulatory oversight.

    @Qwerty said:

    "Everything" being the emergency fix that you dropped everything else for last week?
    Actually, I finished that last Saturday, so I managed to get back to my work for a whole four hours this week.



  • [1] This guy is allowed to make unsupervised changes to the production system? (Oops! I am.)

    [2] Good moral to the story: any idiot will eventually be shot.

     



  • @snoofle said:

    B+1 runs down the hall and brings back B+2 who goes absolutely bonkers, rips the guy a new one and fires him on the spot.
    Ah, one of those rare and special occasions that TDWTF shows that there's some sense of justice left in this world.

     



  • @Severity One said:

    @snoofle said:

    B+1 runs down the hall and brings back B+2 who goes absolutely bonkers, rips the guy a new one and fires him on the spot.
    Ah, one of those rare and special occasions that TDWTF shows that there's some sense of justice left in this world.

     

    How to build a culture of accountability: when an apple is rotten, you don't fire the apple, you fire the apple PLUS the imbecile who kept it in the basket. For such a fuck-up B or B+1 should also go, otherwise it's like that fable with the ant who ends up fired.



  • @snoofle said:

    We had a problem in our code that accesses the primary table, so I just changed it to store the data directly into the history table. I'll fix it when I get a chance.
     

    Once again... he changed production.

    @snoofle said:

    rips the guy a new one and fires him on the spot

    Obligatory "TRWTF is that it should have happened sooner", but WIN!SQUARED! that light has finally dawned.

    @snoofle said:

    Stuff like this always seems to involve offshore developers. Always.

    Was this sacked person actually an offshore developer, then?

    I think there's a culture of outsourcers not understanding or appreciating the value of business data, and thus having a fairly carefree attitude to it importance. The fact he's cost the organisation so much with his last cockup should have rung warning bells. I'm surprised that he and his offshoring company weren't sued for the re-mediation costs.

     


  • FoxDev

    @Speakerphone Dude said:

    @Severity One said:

    @snoofle said:

    B+1 runs down the hall and brings back B+2 who goes absolutely bonkers, rips the guy a new one and fires him on the spot.
    Ah, one of those rare and special occasions that TDWTF shows that there's some sense of justice left in this world.

     

    How to build a culture of accountability: when an apple is rotten, you don't fire the apple, you fire the apple PLUS the imbecile who kept it in the basket. For such a fuck-up B or B+1 should also go, otherwise it's like that fable with the ant who ends up fired.

     

    That sounds a bit harsh; after all, this isn't a failure of the manager, it's a failure of process.

     



  • @snoofle: get a blog!



  • @ubersoldat said:

    @snoofle: get a blog!
    He has one .. and in a cunning move he has gotten someone else to maintain it for him.



  • @Cassidy said:

    I'm surprised that he and his offshoring company weren't sued for the re-mediation costs.
    This literally happened yesterday, so too soon to tell. It'll take about a week to rerun all those reports. Then the managers need to review the differences from what was actually sent to the clients, and determine the practical impact. Then the lawyers need to get involved due to the regulatory oversight. Lawsuits may well be coming; from us to the offshore folks, and from our clients to us.

    I doubt I'll be kept in the loop as I'm just a flunky, but if I hear anything...



  • @Speakerphone Dude said:

    when an apple is rotten, you don't fire the apple, you fire the apple PLUS the imbecile who kept it in the basket.
    I completely agree, but in this place, that would dump >90% of the staff.

    I don't know - it *could* be a good thing?!



  • @Speakerphone Dude said:

    @Severity One said:

    @snoofle said:

    B+1 runs down the hall and brings back B+2 who goes absolutely bonkers, rips the guy a new one and fires him on the spot.
    Ah, one of those rare and special occasions that TDWTF shows that there's some sense of justice left in this world.

     

    How to build a culture of accountability: when an apple is rotten, you don't fire the apple, you fire the apple PLUS the imbecile who kept it in the basket. For such a fuck-up B or B+1 should also go, otherwise it's like that fable with the ant who ends up fired.

    +1

     



  • @snoofle said:

    Stuff like this always seems to involve offshore developers. Always.

    Quite a few of these offshore developers come for a master's degree at our local university. From what I've seen (and what I've heard from some professor friends) they all come with identical resumes with 4.0 GPAs from their undergraduate work at the same no-name college in their home country so the university has no real way of weeding out the good candidates from the bad. Most of them are bad and can't even find the power button on a computer without step-by-step instructions. Then they lie and cheat their way through the graduate program, and they refuse to think for themselves. On test days the whole class will bunch up and let one guy do all the work and they just copy. Then they don't understand why the professors get mad at this.

    The one we had the misfortune of hiring as an intern at our company would not think. Ever. About anything. He was not capable of making decisions. He was the type of guy who asked for permission to take a restroom break and then followed up by asking which hand he should wipe with when he finishes, and then he'll ask you whether he should flush or not. Which wouldn't have been a problem if he could have learned from the experience and made correct decisions on his own the next time he encountered a similar situation. Every day with him seemed to start with a million questions, most of which I had already answered many times in the recent past.

    They're also very good at specifications. But it's sort of like that saying about following the letter of the law but not the spirit. You'll give them specs and they'll get the feature or project done, and it will work. But only with the test/example data you provided with the specs. Any deviation in input and the whole thing burns like a Colorado wildfire. The worst part is you may not notice for a few months because at a cursory glance everything works fine, but when you finally put it into production, databases implode, hard disks shatter, servers go nuclear, and the streets erupt with rioting and looting. Then you crack open the code to take a peek and facepalm desk wall floor sidewalk-after-jumping-from-tenth-story-window and scream "WHYYYYYYYY!!!!1!!??!?!?!??1" while gently rocking back-and-forth in a fetal position in the darkest corner of the office.



  • @Speakerphone Dude said:

    How to build a culture of accountability: when an apple is rotten, you don't fire the apple, you fire the apple PLUS the imbecile who kept it in the basket.
     

    @Severity One said:

    That sounds a bit harsh; after all, this isn't a failure of the manager, it's a failure of process.
     

    All process failures are down to management in some way:

    • those that own a flawed process and refuse to improve it,
    • those that are managing activities within the process and don't follow the procedures correctly. Either way, it's always down to someone, never something.

    I'm kinda in agreement with STD on this one. The UK is poor at accountability, so it's easy to hang a junior out to dry and protect the untouchable godfathers middle management. We dismiss those who carried out the order, but we don't see blame in those who issued the order.

    @snoofle said:

    @Cassidy said:
    I'm surprised that he and
    his offshoring company weren't sued for the re-mediation costs.

    This literally happened yesterday, so too soon to tell.
     

    No, I meant that other thing - using production for a demo and changing live data on the fly, thus costing the organisation quite a bit of wonga to fix. I suppose TRWTF was that this didn't ring alarm bells with management soon as this occurred - and he was able to continue unvetted.



  • @mott555 said:

    The worst part is you may not notice for a few months because at a cursory glance everything works fine, but when you finally put it into production, databases implode, hard disks shatter, servers go nuclear, and the streets erupt with rioting and looting. Then you crack open the code to take a peek and facepalm desk wall floor sidewalk-after-jumping-from-tenth-story-window and scream "WHYYYYYYYY!!!!1!!??!?!?!??1"
     

    The answer to your question is simply "because our QA procedures are simply a cursory glance".

    Things going mammaries-perpendicular in production implies they were never discovered during testing.



  • @Cassidy said:

    @snoofle said:

    @Cassidy said:
    I'm surprised that he and his offshoring company weren't sued for the re-mediation costs.
    This literally happened yesterday, so too soon to tell.
     

    No, I meant that other thing - using production for a demo and changing live data on the fly, thus costing the organisation quite a bit of wonga to fix. I suppose TRWTF was that this didn't ring alarm bells with management soon as this occurred - and he was able to continue unvetted.

    He was given a talk; basically a slap saying: don't do it again. He did the table-swap long before he did the configuration snafu; we just discovered them in a different order.

    In the places I've worked in the past, you were forgiven if you made an honest mistake in production, but if it was reasonably forseeable, or you were negligent, you got the axe.

    In this place, they've been putting formal corporate procedures in place, but they only enforce stuff like prod folks access prod, developers don't; they don't enforce stuff like: prod support folks (e.g.: this guy) should only access prod for appropriate reasons and not as their personal playground.

    I know for a fact both incidents got bucked up the chain to corporate security; hopefully they'll kick some butt as they've got a) the clout and b) the know how.

     


  • FoxDev

    @Cassidy said:

    @RaceProUK said:
    @Speakerphone Dude said:
    How to build a culture of accountability: when an apple is rotten, you don't fire the apple, you fire the apple PLUS the imbecile who kept it in the basket.
     

    That sounds a bit harsh; after all, this isn't a failure of the manager, it's a failure of process.

     

    All process failures are down to management in some way:

    • those that own a flawed process and refuse to improve it,
    • those that are managing activities within the process and don't follow the procedures correctly. Either way, it's always down to someone, never something.

    I'm kinda in agreement with STD on this one. The UK is poor at accountability, so it's easy to hang a junior out to dry and protect the untouchable godfathers middle management. We dismiss those who carried out the order, but we don't see blame in those who issued the order.

    Agree with you there (and been on the wrong end of it myself), but that's predicated on the management actually being at fault. Re-reading the original post, it seems that the fault was ultimately a breakdown in communication, so there's definitely a case to be made against (some of) the managers, as they allowed the communication to falter. However, I'm still not sure the first resort should be <AlanSugarVoice>You're fired!</AlanSugarVoice>, but then I wouldn't be against fines taken from salaries.



  • @snoofle said:

    In this place, they've been putting formal corporate procedures in place, but they only enforce stuff like prod folks access prod, developers don't; they don't enforce stuff like: prod support folks (e.g.: this guy) should only access prod for appropriate reasons and not as their personal playground.

    I know for a fact both incidents got bucked up the chain to corporate security; hopefully they'll kick some butt as they've got a) the clout and b) the know how.

     

    An American president once said "laws are useless without jails" (probably spelled "gaols") meaning that unless there's disciplinary action to enforce policies, there's no incentive to follow them.

    You can have all the policies in place. Unless this fuckwit is drenched in sticky Mountain Dew and stapled up by his goolies above a wasps' nest, people will continue to flout policy if they know there's no ramifications.

    Gotta send a clear message.

     



  • @mott555 said:

    They're also very good at specifications. But it's sort of like that saying about following the letter of the law but not the spirit. You'll give them specs and they'll get the feature or project done, and it will work. But only with the test/example data you provided with the specs. Any deviation in input and the whole thing burns like a Colorado wildfire.

    I too have noticed this phenomenon several times with offshore developers. I don't understand it. What's worse, it hasn't made us any better at making specs or test cases. So we're all doomed.

    I don't understand to what to attribute the expectation discrepancy. Communication, language barrier? Poor workplace culture? Just plain lazy developers? But it's so rampant. Is it a side-effect of the lack of context given to the offshores? Do we all just suck at explaining what we want from them? Perhaps developers are just as bad as clients as non-developers? Whatever the reason, I wish we didn't rely on them so much.



  • @Cassidy said:

    ...

    @Severity One said:

    That sounds a bit harsh; after all, this isn't a failure of the manager, it's a failure of process.
     

    ...

    No... I didn't say that. RaceProUK did.

     



  • @Ben L. said:

    This is the same versioning system I use for my code! file.go, file_new.go, file-revision3.go, file--copy7.go, file-fix-bugfix-copy.go

    TRWTF. This is literally the first time I've ever heard of somebody using Go.



  • @Severity One said:

    No... I didn't say that. RaceProUK did.
     

    Sorry, quoting failure on my part by javascript that can't determine which text is attributed to which poster when blocks are selected and "Quote" button is pressed.


  • FoxDev

    @Cassidy said:

    @Severity One said:

    No... I didn't say that. RaceProUK did.
     

    Sorry, quoting failure on my part by javascript that can't determine which text is attributed to which poster when blocks are selected and "Quote" button is pressed.

    Community Server is TRWTF.



  • @RaceProUK said:

    @Cassidy said:

    @Severity One said:

    No... I didn't say that. RaceProUK did.
     

    Sorry, quoting failure on my part by javascript that can't determine which text is attributed to which poster when blocks are selected and "Quote" button is pressed.

    Community Server is TRWTF.

     

    It just selects some text and puts quote tags around it. Do you expect CS's quote feature to parse the HTML of the source post in its entirety? This is rather unrealistic.

     



  • @dreaded said:

    It just selects some text and puts quote tags around it. Do you expect CS's quote feature to parse the HTML of the source post in its entirety?
     

    No, but I did expect it to quote the correct author.

    Whether or not that requires parsing the HTML of the course post, I care not.

    Accuracy, I care for.



  • The bad thing is that you have NO IDEA what other WTFs this guy's created before he got fired.

    The good thing is that he can't (accidentally?) sabotage the rest of the system(s) while being considered for disciplinary action/firing.



  • @zelmak said:

    The bad thing is that you have NO IDEA what other WTFs this guy's created before he got fired.

    That is why I read these forums every weekday!



  • @zelmak said:

    The bad thing is that you have NO IDEA what other WTFs this guy's created before he got fired.
    Sadly, when they rear their ugly heads, I'm usually the one who gets to deal with it. That's boring for me, but more often than not, it's worth posting about it here.



  • @Xyro said:

    I don't understand to what to attribute the expectation discrepancy.

    If I had to guess, I'd say to a culture whose education system is heavily geared toward making students learn facts rather than develop insight.



  • @All: Update: after the first bunch of reports were reran and compared, it turns out that while there were differences, they randomly averaged out to a net delta of zero over time. In other words, there was no net affect to the clients - over time, BUT we still need to report it to a) the clients, and then prove to them that it didn't cost them or their customers any money, and b) the regulatory agencies, which will force a formal audit.

    FWIW: IANAL: The lawyers squashed any talk of prosecuting this guy as he didn't do what he did with malicious intent (apparently, idiotic cluelessness is not a justification for suing somebody).

     



  • @snoofle said:

    Sadly, when they rear their ugly heads, I'm usually the one who gets to deal with it. That's boring for me, but more often than not, it's worth posting about it here.

    That's why banishment to the Phantom Zone is a superior option. You can still talk to them.



  • @RaceProUK said:

    @Cassidy said:

    @Severity One said:

    No... I didn't say that. RaceProUK did.
     

    Sorry, quoting failure on my part by javascript that can't determine which text is attributed to which poster when blocks are selected and "Quote" button is pressed.

    Community Server is TRWTF.

    Cassidy AND Community Server should be fired.



  • @snoofle said:

    apparently, idiotic cluelessness is not a justification for suing somebody

    In the US and Canada, a suit against an idiot is unlikely to succeed if he is your own employee, however this is a totally different thing if he is a director. Even if the company has been closed, civil actions against a director are made easy if he failed to his duties on one of the Big Three (you can even petition for criminal charges in that case):

    1. Gross negligence
    2. Witholding sales tax perceived from clients
    3. Witholding income tax retained from employees salary

    As my lawyer told me when he created my incorp: besides that, have fun!



  • @snoofle said:

    Stuff like this always seems to involve offshore developers. Always.

     

    Because off shore developers don't understand the business and no one is spending the monlths of time it would take to educate them.  If you give a goal and say "get to that goal" but don't define the rules of the game then don't be surprised if the players take short cuts that go agains the rules.

     You are lucky.  You grew up in an environment where the rules were taught along with growing up.  "Work hard, make money for the company, get along with the boss and you'll get promoted" is a rule.  But what if the rule was different, as it is in many developing parts of the world - "Work hard, make money for the company, and the boss's son will be able to get hired as your supervisor".

     



  • @snoofle said:

    @Qwerty said:
    you can stop producing them
    Not really. Some of them would be real-time, if the data was not stored in history tables for the past 3 months instead (e.g.: MQ) messages that just feed blindly into other systems that use our data for reference data. Some of them take it as excel spreadsheets. Very few want the paper version.
     

     FTFY



  • @snoofle said:

    apparently, idiotic cluelessness is not a justification for suing somebody
     

    But you're not suing him for a characteristic he exhibits, you're suing him for the actions his attributes have caused - and the unnecessary cost to your organisation. He's liable for that at least.




  • @snoofle said:

    That never happens with data that changes daily.
    A sub-WTF may be:  why wasn't this detected sooner?  Why didn't the report error out saying that there was no data for a particular date or date range?



  • I'm guessing the algorithm to fetch data either contains no checks for stale data, or the range filters are relative and not absolute (eg: last 30 days recorded). Whilst data was returned, reports ran.

    As someone else pointed out, how come the client didn't notice? Unless the data was presented in such a format that it was difficult to establish inaccuracies.


Log in to reply