I may have just dodged a bullet



  • So, some other folks in the office have been working on a ticket, apparently modifying some tables in an ancillary database that just about everyone uses.

    Person: Does <MAIN SYSTEM> use any of the <ANCILLARY SYSTEM> tables?

    Me: [Stammering/Stuttering] Uh, yeah, I'm pretty sure it does.

    Person: The reason I'm asking is that we've been working on <ANCILLARY SYSTEM> in test and I wanted to put the changes into production.

    Me: [Mouth agape] Um, I think that would be a bad idea. If you've changed the structure of tables (column types, sizes, etc.) and the code doesn't know about it, Bad Things(tm) will probably happen. There was nothing in the ticket to investigate changes to the software which uses those tables?

    Person: Okay, I guess I'll wait for <BOSS> to come back on Monday. Good thing I asked, or you'd be here all weekend cleaning it up.

    Me: No, whoever mucked up the database would be here backing out the changes.

    (I'm paraphrasing, of course ... but my question didn't get answered.)

     

    Perhaps I really do need to dust off my resume...



  • @zelmak said:

    Person: Okay, I guess I'll wait for <BOSS> to come back on Monday. Good thing I asked, or you'd be here all weekend cleaning it up.

    Me: I'll just tell them to call Person. He said he is responsible for these changes.



  • my mind was flailing around ... where is the change document? where is the test result? why isn't his in front of the change board and going thru the standard process?

    I mean ... seriously?


  • ♿ (Parody)

    @zelmak said:

    Person: Does <MAIN SYSTEM> use any of the <ANCILLARY SYSTEM> tables?

    Me: [Stammering/Stuttering] Uh, yeah, I'm pretty sure it does.

    Person: The reason I'm asking is that we've been working on <ANCILLARY SYSTEM> in test and I wanted to put the changes into production

    TRWTF is that you have only a partially configured test environment.



  • @zelmak said:

    Person: Does <MAIN SYSTEM> use any of the <ANCILLARY SYSTEM> tables?

    Me: [Stammering/Stuttering] Uh, yeah, I'm pretty sure it does.

    Person: The reason I'm asking is that we've been working on <ANCILLARY SYSTEM> in test and I wanted to put the changes into production.

    Me: [Mouth agape] Um, I think that would be a bad idea. If you've changed the structure of tables (column types, sizes, etc.) and the code doesn't know about it, Bad Things(tm) will probably happen. There was nothing in the ticket to investigate changes to the software which uses those tables?

    Person: Okay, I guess I'll wait for <BOSS> to come back on Monday. Good thing I asked, or you'd be here all weekend cleaning it up.

    Someone's figured out how dialog replacement in Skyrim's Radiant AI system works!



  • TRWTF is that the guy actually bothered to ask.



  • @zelmak said:

    my mind was flailing around ... where is the change document? where is the test result? why isn't his in front of the change board and going thru the standard process?

    I mean ... seriously?

    Probably wouldn't have helped. Last year I got called into a coworker's office on a Monday morning to help. The warehouse system was rejecting all of the orders that came in. It turns out that over the weekend someone installed a "clean-up" service from the application vendor and assumed that the word of the vendor that it wouldn't screw anything up was good enough. He filled out all the correct paperwork, tested it (but didn't do any regression tests), filed a change control, and defended the change to the Change Control Board.

    No amount of process will fix someone who never considers "I wonder if the system will still function after I do this".



  • @Jaime said:

    No amount of process will fix someone who never considers "I wonder if the system will still function after I do this".
     

    This really needs to be on stickers everywhere. And the wording is so accurate: "amount of process". The process is really just there for the process, and everyone is a slave of the process.



  •  Isn't that what integration tests are for - to make sure that changes to different systems don't interfere with each other?

    This is where a tool like opengrok comes in handy - the tool indexes all code in the codebase (including the application code, whatever sql/database scripts/files you store in VCS, etc.) so you can just ask it what is using the DeleteMe column on the XYNotSoImportantStuff table and it will spit back all of the files that reference it, like ApplicationCoreDAO.java in the centralCriticalLibrary project. 

    Of course having the tool doesn't guarantee that a given moron will look at it or submit the change request to the other applications/libraries that use the DeleteMe column before dropping it. 



  • @airdrik said:

     Isn't that what integration tests are for - to make sure that changes to different systems don't interfere with each other?

    This is where a tool like opengrok comes in handy - the tool indexes all code in the codebase (including the application code, whatever sql/database scripts/files you store in VCS, etc.) so you can just ask it what is using the DeleteMe column on the XYNotSoImportantStuff table and it will spit back all of the files that reference it, like ApplicationCoreDAO.java in the centralCriticalLibrary project. 

    Of course having the tool doesn't guarantee that a given moron will look at it or submit the change request to the other applications/libraries that use the DeleteMe column before dropping it. 

    The tool is the easy part. If you can get the moron to manually search the codebase, he'll eventually get tired of it and go searching for a tool. But if you show him opengrok, that won't help the underlying problem that he thinks the change has no risk because someone told him so.

    My workplace is full of people that spend all day looking for people that are willing to make their decisions for them. They keep forgetting that even if they are able to point at someone else and say "it was his fault", the system still broke. If you are the guy that happens to be standing next to every system failure, eventually it's going to catch up with you.



  • @airdrik said:

     Isn't that what integration tests are for - to make sure that changes to different systems don't interfere with each other?

    This is where a tool like opengrok comes in handy - the tool indexes all code in the codebase (including the application code, whatever sql/database scripts/files you store in VCS, etc.) so you can just ask it what is using the DeleteMe column on the XYNotSoImportantStuff table and it will spit back all of the files that reference it, like ApplicationCoreDAO.java in the centralCriticalLibrary project. 

    Of course having the tool doesn't guarantee that a given moron will look at it or submit the change request to the other applications/libraries that use the DeleteMe column before dropping it. 

    I have yet to ever find a tool that "just ask it what is using the DeleteMe column on the XYNotSoImportantStuff table and it will spit back all of the files that reference it". Even a simple case of (psuedo language)

    string s1  = "Dele";
    string s2 = "teMe";
    var content = row[s1+s2];

    will fail.

    At BEST it is a starting point. It is also trivial to have "logically unreachable" (there appears to be a code path that can reach it, but the combination if "real world" states is such that it will never occur).

    My preference is for instrumentation of the appliction. Have it record at runtime all of the call stacks that actually reference the item in question, and keep this as a repository (updated by automatic tests). IF your test suites (Unit, Integration, Design, Regression, Ad-Hoc) are comprehensive, you will get a much better picture of how your application fits together.



  • @Jaime said:

    No amount of process will fix someone who never considers "I wonder if the system will still function after I do this".

    While this is correct, it's the wrong perspective.  Unfortunately, we have far too much of the wrong perspective, because we have non-technical people trying to evade a technical problem.  It doesn't take someone skilled in computers to figure out how to fix the process to catch this.  It just requires someone skilled in dealing with details.  Even a lawyer can do it, in at least some instances.

    Your change control process doesn't mean a thing unless the process reviews the technical aspects of the changes and it is performed by people who are technically skilled to review those changes.

    The process we have at work has a change control board, which only reviews the 'what is this change and why is it being done and what else is being done at that time' level of detail, similar to the process you describe above.  However, there's a step before those questions are asked where one to two technical people (depending on various factors) review what is *actually* being done, and what tests were run to make sure the person performing the change didn't just test the change itself, but also tested to make sure the change didn't break other stuff.

    Summary: quantity of process may not fix someone who doesn't consider consequences, but quality of process may.



  • @tgape said:

    @Jaime said:
    No amount of process will fix someone who never considers "I wonder if the system will still function after I do this".

    While this is correct, it's the wrong perspective.  Unfortunately, we have far too much of the wrong perspective, because we have non-technical people trying to evade a technical problem.  It doesn't take someone skilled in computers to figure out how to fix the process to catch this.  It just requires someone skilled in dealing with details.  Even a lawyer can do it, in at least some instances.

    Your change control process doesn't mean a thing unless the process reviews the technical aspects of the changes and it is performed by people who are technically skilled to review those changes.

    The process we have at work has a change control board, which only reviews the 'what is this change and why is it being done and what else is being done at that time' level of detail, similar to the process you describe above.  However, there's a step before those questions are asked where one to two technical people (depending on various factors) review what is *actually* being done, and what tests were run to make sure the person performing the change didn't just test the change itself, but also tested to make sure the change didn't break other stuff.

    Summary: quantity of process may not fix someone who doesn't consider consequences, but quality of process may.

    In my story, the one who made the call not to regression test was the technical "expert". When you have third-party software, it's hard to have someone that can make a valid risk assessment. Sometimes the best option is to be paranoid. Our current process fix is to have me look at their changes for a technical viewpoint, but I'm a developer for completely unrelated software. Every recommendation I make is just common sense and the person who's role it is should be able to make every call I make.

    At the end of the day, "I have tested the mission critical aspects of this software and I guarantee that it won't shut the company down" trumps "I have made a technical assessment that the scope of testing is correct". The latter may save time and money, but it can't beat a full regression test for completeness.



  • @Jaime said:

    @tgape said:

    Summary: quantity of process may not fix someone who doesn't consider consequences, but quality of process may.

    In my story, the one who made the call not to regression test was the technical "expert". When you have third-party software, it's hard to have someone that can make a valid risk assessment. Sometimes the best option is to be paranoid. Our current process fix is to have me look at their changes for a technical viewpoint, but I'm a developer for completely unrelated software. Every recommendation I make is just common sense and the person who's role it is should be able to make every call I make.

    At the end of the day, "I have tested the mission critical aspects of this software and I guarantee that it won't shut the company down" trumps "I have made a technical assessment that the scope of testing is correct". The latter may save time and money, but it can't beat a full regression test for completeness.

    A high quality process probably would be paranoid and require at least integration tests for every change. Plus whatever tests the reviewer suggests, of course.



  • @airdrik said:

     Isn't that what integration tests are for - to make sure that changes to different systems don't interfere with each other?

    This is where a tool like opengrok comes in handy - the tool indexes all code in the codebase (including the application code, whatever sql/database scripts/files you store in VCS, etc.) so you can just ask it what is using the DeleteMe column on the XYNotSoImportantStuff table and it will spit back all of the files that reference it, like ApplicationCoreDAO.java in the centralCriticalLibrary project. 

    Of course having the tool doesn't guarantee that a given moron will look at it or submit the change request to the other applications/libraries that use the DeleteMe column before dropping it. 

    I have a hard time believing that a tool will find everything.  It can only look at things that you tell it to look at.  Otherwise it is searching your entire system for something that may not exist.

    Modern programming practices and design patterns solve this in a much better way. Try the adapter pattern to abstract the ancillary system from the main system.  And let the adapter be in the hands of the ancillary system developers.  Do this and any changes to said system will never affect the main system as long as only one rule is followed, "Do not change the interface of the Adapter" The way it translates calls can change but not the interface.  You can even swap out the entire ancillary system as long as the adapter interface is left unchanged, just re route those calls to the new anciallary system calls.

    The second rule broken here is one system bypassign another system to play with the second systems data.  Is the main system following al lthe business rules for the second system when making changes?  If it is only reading, you have still broken the rule because you need to much info about a system not of your concern.  Go through an adapter to get the data you need, let the anciallry system, which already knows how to get the data, retrieve it for you rather then rewriting all those queries or database calls.

     



  • @Bulb said:

    @Jaime said:
    @tgape said:

    Summary: quantity of process may not fix someone who doesn't consider consequences, but quality of process may.

    In my story, the one who made the call not to regression test was the technical "expert". When you have third-party software, it's hard to have someone that can make a valid risk assessment. Sometimes the best option is to be paranoid. Our current process fix is to have me look at their changes for a technical viewpoint, but I'm a developer for completely unrelated software. Every recommendation I make is just common sense and the person who's role it is should be able to make every call I make.

    At the end of the day, "I have tested the mission critical aspects of this software and I guarantee that it won't shut the company down" trumps "I have made a technical assessment that the scope of testing is correct". The latter may save time and money, but it can't beat a full regression test for completeness.

    A high quality process probably would be paranoid and require at least integration tests for every change. Plus whatever tests the reviewer suggests, of course.

    Riiiiiight.  We change control everything.  Could you imagine having to integration test every application in an entire VMware system when you update the client tools?  I once saw a change control for physically removing a server that had been powered off for a month, what do you test there?

    For every change, somebody has to decide what testng is appropriate.  If that person is stupid, the process can't fix it.



  • @Jaime said:

    For every change, somebody has to decide what testng is appropriate.  If that person is stupid, the process can't fix it.

    But the process can help by having decisions reviewed. Therefore any ONE stupid person (even the most intelligent/diligent person has stupid days) should not cause problems. The bigger the potential risk, the bigger (subject to reasonable constraints) the review panel should be. I try to make it a minimum of two, but with a strong preference for odd numbers [partially to avoid a deadlock]. I find that a 5 member panel (obviously depending on environment size) is often ideal (In a three member panel a potential dissenting voice may not speak up because they would be the "lone voice", hopefully in a 5 member panel there would be two dissents much more often than a single one)



  • @KattMan said:

    @airdrik said:

     Isn't that what integration tests are for - to make sure that changes to different systems don't interfere with each other?

    This is where a tool like opengrok comes in handy - the tool indexes all code in the codebase (including the application code, whatever sql/database scripts/files you store in VCS, etc.) so you can just ask it what is using the DeleteMe column on the XYNotSoImportantStuff table and it will spit back all of the files that reference it, like ApplicationCoreDAO.java in the centralCriticalLibrary project. 

    Of course having the tool doesn't guarantee that a given moron will look at it or submit the change request to the other applications/libraries that use the DeleteMe column before dropping it. 

    I have a hard time believing that a tool will find everything.  It can only look at things that you tell it to look at.  Otherwise it is searching your entire system for something that may not exist.

     It certainly doesn't find everything, but it does find most things - it is designed to index the all of the (static) text in the entire codebase (on a regular basis to keep up-to-date with changes) so that when you do a search it can provide you quickly with e.g. all references in the entire codebase where "foobar" is mentioned.  You still have to go wading through the references to determine what is relevant, and it certainly doesn't do quite as good a job as your IDE will with the entire code-base checked out (of course even your IDE won't find references obfuscated by eval-ing concatenated strings, though if you are doing that in your code then you guarantee that nothing short of running the code will detect all of the hidden references). But the point of having the tool is that it will assist you in finding such integration points without having to have all of the code checked out.



  • @airdrik said:

    @KattMan said:

    @airdrik said:

     Isn't that what integration tests are for - to make sure that changes to different systems don't interfere with each other?

    This is where a tool like opengrok comes in handy - the tool indexes all code in the codebase (including the application code, whatever sql/database scripts/files you store in VCS, etc.) so you can just ask it what is using the DeleteMe column on the XYNotSoImportantStuff table and it will spit back all of the files that reference it, like ApplicationCoreDAO.java in the centralCriticalLibrary project. 

    Of course having the tool doesn't guarantee that a given moron will look at it or submit the change request to the other applications/libraries that use the DeleteMe column before dropping it. 

    I have a hard time believing that a tool will find everything.  It can only look at things that you tell it to look at.  Otherwise it is searching your entire system for something that may not exist.

     

     

     It certainly doesn't find everything, but it does find most things - it is designed to index the all of the (static) text in the entire codebase (on a regular basis to keep up-to-date with changes) so that when you do a search it can provide you quickly with e.g. all references in the entire codebase where "foobar" is mentioned.  You still have to go wading through the references to determine what is relevant, and it certainly doesn't do quite as good a job as your IDE will with the entire code-base checked out (of course even your IDE won't find references obfuscated by eval-ing concatenated strings, though if you are doing that in your code then you guarantee that nothing short of running the code will detect all of the hidden references). But the point of having the tool is that it will assist you in finding such integration points without having to have all of the code checked out.

    And my second point which you snipped, was to say that all this isn't necessary if you follow proper abstraction.  Leave your references within the code domain where they belong.

    Then the only methods you need to look for in this manner are the adapter methods, but changing those should follow some other much more strict rule.



  • @blakeyrat said:

    Someone's figured out how dialog replacement in Skyrim's Radiant AI system works!
     

    Indeed.  I got pretty disapointed after I discovered the repition after doing a bunch of these side quests.  Concluded that it wasn't cutting edge at all.

    And worse, I fucked up my Skyrim experience by doing a shit load of the side quests and the non-story line quests like the thieves guild.  So I didn't get very far in the story before I got bored with it and disapointed about the Radiant AI system.  So now I'm all decked out in obsidian shit and can kill a dragon by mearly looking at it, but haven't played it in over a month. I stopped at the story quest where you battle the dragon the first time on that tall mountain peak.  He's rather difficult as leveled-up as I was.

     



  • @Jaime said:

    No amount of process will fix someone who never considers "I wonder if the system will still function after I do this".

    Disagree: any process that brings about change must feature risk management in some measure, so I would expect any deployment process to include activities surrounding the creation of a backout plan and criteria for invocation. Failure is down to one of two things:

    • those activities missing from process documentation, in which case the process is flawed and the individual can't be blamed for following it - process owner needs a kicking
    • those activities being included for someone not following the process, in which case whoever was responsible for process control needs a kicking. 

    @tgape said:

    While this is correct, it's the wrong perspective.  Unfortunately, we have far too much of the wrong perspective, because we have non-technical people trying to evade a technical problem.  It doesn't take someone skilled in computers to figure out how to fix the process to catch this.  It just requires someone skilled in dealing with details.  Even a lawyer can do it, in at least some instances.

    Your change control process doesn't mean a thing unless the process reviews the technical aspects of the changes and it is performed by people who are technically skilled to review those changes.

    A properly-documented process includes roles & responsibilities for the performed activities: or to put your sentence more generically: a poor choice of role to fulfil those specific responsibilities will contribute to process failure.

    All that can happen post-situation is an analysis that led to the failure, and improvement opportunities identified: learn by those mistakes and take action to prevent them from happening.



  • @Cassidy said:

    @Jaime said:

    No amount of process will fix someone who never considers "I wonder if the system will still function after I do this".

    Disagree: any process that brings about change must feature risk management in some measure, so I would expect any deployment process to include activities surrounding the creation of a backout plan and criteria for invocation. Failure is down to one of two things:

    • those activities missing from process documentation, in which case the process is flawed and the individual can't be blamed for following it - process owner needs a kicking
    • those activities being included for someone not following the process, in which case whoever was responsible for process control needs a kicking. 

    We just had a change go wrong yesterday.  It went through the whole change control process.  I looked at the rollback plan when the shit started to hit the fan and is was complete gibberish.  Our problem is that we have 1500 IT employees and 900 applications to support.  Very few applications have more than one person who is qualified to make any judgement at all, so if the person in charge of an application is a moron, we are screwed.  In this case, there was a technical peer reviewer, but he's the one that wrote the useless rollback plan.  We have a solid process and everyone followed it, but the process was unable to compensate for the lack of skill of those performing the change.

    The only process improvements we need are the hiring process and the performance review process.  These two need to be busted down to broom duty.



  • @Jaime said:

    We just had a change go wrong yesterday.  It went through the whole change control process.  I looked at the rollback plan when the shit started to hit the fan and is was complete gibberish

    So:

    • nobody reviewed the backout plan to see if it was gibberish before proceeding with the change
    • nobody tried the rollout process on a testbed to identify what could go wrong - let's just proceed with the change
    • nobody tested the backout plan to see if the gibberish would have worked. 

    That about right?

    @Jaime said:

    Our problem is that we have 1500 IT employees and 900 applications to support.  Very few applications have more than one person who is qualified to make any judgement at all, so if the person in charge of an application is a moron, we are screwed

    Do you have clear lines of ownership of the applications? Irrespective of who is a moron and who isn't, do you have some means (eg: RACI) to determine who IS accountable? If you continuously get a moron who gives it the thumbs-up, then it's their name repeatedly against a failed change, and at some point that will attract the attention of higher-ups.

    @Jaime said:

    In this case, there was a technical peer reviewer, but he's the one that wrote the useless rollback plan.

    Summary: the author and reviewer is the same person, we don't have any policy governing independent reviewing, ergo there is no real quality control. Fortunately, if his signature is listed as "creator" and "reviewer" then it ain't too hard to do finger-pointing.

    @Jaime said:

    We have a solid process and everyone followed it, but the process was unable to compensate for the lack of skill of those performing the change.

    Then there's a flaw in the policy governing the process.. but when all's said and done, you've got someone accountable for the failure and if nothing can be found wrong with the process then the finger points at the person.

    @Jaime said:

    The only process improvements we need are the hiring process and the performance review process.

    That. You got someone higher-up who you can flag up these issues? I sell this to management as "cost to the firm" and "lost business/customers/reputation", rather than a whine about someone being slightly incompetent.

     

     

     



  • @Cassidy said:

    @Jaime said:

    We just had a change go wrong yesterday.  It went through the whole change control process.  I looked at the rollback plan when the shit started to hit the fan and is was complete gibberish

    So:

    • nobody reviewed the backout plan to see if it was gibberish before proceeding with the change
    • nobody tried the rollout process on a testbed to identify what could go wrong - let's just proceed with the change
    • nobody tested the backout plan to see if the gibberish would have worked. 

    That about right?

    Technically moron #1 wrote the rollback plan and moron #2 reviewed it.  In reality, moron #1 simply threw up his hands and submitted it for review blank and moron #2 filled it in with gibberish.  Unfortunately, that technically qualifies as a review.  Nobody tried it.  When I asked what they actually planned to do, they had a reasonable explanation, but their actual plan didn't resemble what they wrote.

    @Cassidy said:

    @Jaime said:
    Our problem is that we have 1500 IT employees and 900 applications to support.  Very few applications have more than one person who is qualified to make any judgement at all, so if the person in charge of an application is a moron, we are screwed

    Do you have clear lines of ownership of the applications? Irrespective of who is a moron and who isn't, do you have some means (eg: RACI) to determine who IS accountable? If you continuously get a moron who gives it the thumbs-up, then it's their name repeatedly against a failed change, and at some point that will attract the attention of higher-ups.

    Yes, we have RACI's for everything.  In our corporate culture, not following the process will get you fired (official policy is two cases of changes without change control result in unconditional termination, no time limit).  Not being competent just gets you sent to training in Las Vegas.@Cassidy said:
    @Jaime said:
    In this case, there was a technical peer reviewer, but he's the one that wrote the useless rollback plan.

    Summary: the author and reviewer is the same person, we don't have any policy governing independent reviewing, ergo there is no real quality control. Fortunately, if his signature is listed as "creator" and "reviewer" then it ain't too hard to do finger-pointing.

    Nope.  Two people, both morons.  We actually had two implementation disasters this week.  One was moron #1 with moron # 2 as a reviewer and the other was moron #2 with moron #1 as a reviewer.  I sat in the pre-deployment meetings for both changes, and I made additional testing recommendations for both.  Both failures were directly a result of my recommendations not being followed correctly.  They didn't refuse to do the additional testing, they simply bumbled it to the point of not finding the problem.


  • @Cassidy said:

    @Jaime said:
    We just had a change go wrong yesterday. 

    100% agreement with Cassidy on this one, especially regarding who TESTED the rollback!



  • @Cassidy said:

    @Jaime said:
    The only process improvements we need are the hiring process and the performance review process.

    That. You got someone higher-up who you can flag up these issues? I sell this to management as "cost to the firm" and "lost business/customers/reputation", rather than a whine about someone being slightly incompetent.

    This year is the first year I'm going to get less than a stellar review.  The difference this year is that we've lost of lot of good people leaving us with a higher concentration of morons than in the past.  I lost my cool a few times due to frustration.  Getting along counts more towards our performance reviews than doing your job well.  So, I do have a higher-up and he has flagged an issue -- me.



  • @Jaime said:

    Technically moron #1 wrote the rollback plan and moron #2 reviewed it.  In reality, moron #1 simply threw up his hands and submitted it for review blank and moron #2 filled it in with gibberish.  Unfortunately, that technically qualifies as a review.  Nobody tried it.  When I asked what they actually planned to do, they had a reasonable explanation, but their actual plan didn't resemble what they wrote.

    A review should be a "Read Only" operation. If there are changes required, the review FAILS and the item is rejected until it is re-submitted with the appropriate changes (this can be streamlined, but not eliminated). Therefore the policy should [IMPO] never have allowed "it for review blank and moron #2 filled it in"  with gibberish or otherwise.



  • @TheCPUWizard said:

    @Jaime said:

    Technically moron #1 wrote the rollback plan and moron #2 reviewed it.  In reality, moron #1 simply threw up his hands and submitted it for review blank and moron #2 filled it in with gibberish.  Unfortunately, that technically qualifies as a review.  Nobody tried it.  When I asked what they actually planned to do, they had a reasonable explanation, but their actual plan didn't resemble what they wrote.

    A review should be a "Read Only" operation. If there are changes required, the review FAILS and the item is rejected until it is re-submitted with the appropriate changes (this can be streamlined, but not eliminated). Therefore the policy should [IMPO] never have allowed "it for review blank and moron #2 filled it in"  with gibberish or otherwise.

    Sure, that's the intention of how the process should be followed.  However, had the read-onlyness of the document been enforced by technical means, then moron #2 would have simply emailed moron #1 the text to paste into the rollback plan section of the document.  Our corporate policies encourage paying lipservice to the process with draconian rules around filling in the document and lax punishments for failure in execution.  I've seen rollback plans for emergency data changes in the middle of the day that look like this:

    Pre-deployment:
    1.  Back up production database to xxxx

    Rollback:
    1.  Restore production database from backup at xxx

    The plan is technically sound and testable, except for the fact that it wipes out all changes made between the two steps (the actual work the company does).  Stupid people cannot be made smart through process.  This rollback plan will survive almost any process scrutiny you throw at it, but anyone with a clue can tell it's a horrible idea.



  • @Jaime said:

     I've seen rollback plans for emergency data changes in the middle of the day that look like this:

    Pre-deployment:
    1.  Back up production database to xxxx

    Rollback:
    1.  Restore production database from backup at xxx

    The plan is technically sound and testable, except for the fact that it wipes out all changes made between the two steps (the actual work the company does).  Stupid people cannot be made smart through process.  This rollback plan will survive almost any process scrutiny you throw at it, but anyone with a clue can tell it's a horrible idea.

     It should fail the tests. The test environment should have a constant stream of transactions (like production), and the test should verify constency. This "framework" should be outside the *specific* test for the proposed change so that it is run on *every* change that is being tested.

     note: I am not disagreeing with you per-se, and do sympathize with your situation - but "process" cn be made fairly bullet proof (never perfectly) against most "morons". The fundamental idea is that of continuous improvement. When the "process fails" the failure must be identified, and the process adjusted such that the same type of situation can not "make it past the gate". Withn a fairly short period (especially if the proces is in heavy use) it becomes farily robust. Of course, the only thing that can help " and lax punishments for failure in execution", is to get buy-in from a higer level of management, so that the person who failed to enact the "punishment" is themselves punished by their boss...



  • @TheCPUWizard said:

    but "process" cn be made fairly bullet proof (never perfectly) against most "morons".

    To me, this is big red flag. Your process is a band-aid for the real problem: incompetent employees. You have your smart people spending significant amounts of their time creating and implementing process entirely to protect the company from its own employees. The entire goal of process should be to save the company money and headaches by reducing the error rate. The solution is to get rid of the incompetent employees, not waste the valuable time of your competent employees in the hopes they can (mostly) mitigate the problems caused by incompetents.


  • ♿ (Parody)

    @morbiuswilters said:

    @TheCPUWizard said:
    but "process" cn be made fairly bullet proof (never perfectly) against most "morons".

    To me, this is big red flag. Your process is a band-aid for the real problem: incompetent employees. You have your smart people spending significant amounts of their time creating and implementing process entirely to protect the company from its own employees. The entire goal of process should be to save the company money and headaches by reducing the error rate. The solution is to get rid of the incompetent employees, not waste the valuable time of your competent employees in the hopes they can (mostly) mitigate the problems caused by incompetents.

    To me, this is a big red flag. If you think you can simply get rid of the morons, you're living in a fantasy land. Also, there's simple human error, to which anyone is prone. I think TheCPUWizard had it pretty much dead on, and not just as far as IT, but for any aspect of business.

    Also, from a business perspective, if you can get a system where the required level of competence is lower, but the end product is the same, you can increase your productivity and competitiveness. This sort of thing helps the smart people, too, since they don't have to spend so much effort on even the simplest parts. Also, a good process makes it easier to do things correctly (and correctly more quickly) than to make mistakes.



  • @boomzilla said:

    @morbiuswilters said:
    @TheCPUWizard said:
    but "process" cn be made fairly bullet proof (never perfectly) against most "morons".

    To me, this is big red flag. Your process is a band-aid for the real problem: incompetent employees. You have your smart people spending significant amounts of their time creating and implementing process entirely to protect the company from its own employees. The entire goal of process should be to save the company money and headaches by reducing the error rate. The solution is to get rid of the incompetent employees, not waste the valuable time of your competent employees in the hopes they can (mostly) mitigate the problems caused by incompetents.

    To me, this is a big red flag. If you think you can simply get rid of the morons, you're living in a fantasy land. Also, there's simple human error, to which anyone is prone. I think TheCPUWizard had it pretty much dead on, and not just as far as IT, but for any aspect of business.

    Also, from a business perspective, if you can get a system where the required level of competence is lower, but the end product is the same, you can increase your productivity and competitiveness. This sort of thing helps the smart people, too, since they don't have to spend so much effort on even the simplest parts. Also, a good process makes it easier to do things correctly (and correctly more quickly) than to make mistakes.

    There's a third option: hold meetings to train employees on proper use of the process. That way you don't have to fire anyone and don't waste manhours idiot-proofing processes. You might not be able to make dumb people smart, but you can direct them along a path.



  • @morbiuswilters said:

    @TheCPUWizard said:
    but "process" cn be made fairly bullet proof (never perfectly) against most "morons".

    To me, this is big red flag. Your process is a band-aid for the real problem: incompetent employees. You have your smart people spending significant amounts of their time creating and implementing process entirely to protect the company from its own employees. The entire goal of process should be to save the company money and headaches by reducing the error rate. The solution is to get rid of the incompetent employees, not waste the valuable time of your competent employees in the hopes they can (mostly) mitigate the problems caused by incompetents.

     In addition to what Boomzilla posted (Which I agree with), in most (at least USA) places getting rid of " incompetent employees" requires documentation, and this documentation must be free of any potential bias, otherwise lawsuits may follow. The best way (both IMPO and in "real" legal guidance I have recieved) is to make sure the is a PROCESS which captures the incomptent behaviour along with the ramification such that it can be shown to be unbiased, repetitive (a single event usually will not qualify), impactful, and that there have been reasonable attempts at remediation.



  • @boomzilla said:

    If you think you can simply get rid of the morons, you're living in a fantasy land.

    Well, it's them or me. Finding a new job is easy. I'd rather spend my time effectively rather than sending out memos reminding people they need to unzip their pants before they start pissing.

    @boomzilla said:

    Also, there's simple human error, to which anyone is prone.

    Of course. I'm not against all process, but if you find yourself creating process solely for the purpose of protecting against people who should know better, that's an entirely different situation.

    @boomzilla said:

    Also, from a business perspective, if you can get a system where the required level of competence is lower, but the end product is the same, you can increase your productivity and competitiveness.

    Isn't that the theory behind outsourcing everything to India? You get what you pay for. If I have a choice between hiring 1 smart guy or 10 mediocre guys and 1 smart guy to make sure they don't fuck up, I'm going with the single smart guy.

    @boomzilla said:

    This sort of thing helps the smart people, too, since they don't have to spend so much effort on even the simplest parts. Also, a good process makes it easier to do things correctly (and correctly more quickly) than to make mistakes.

    I believe process should exist to help the good people do a better job, not to help the mediocre people do a somewhat-better-than-mediocre job.



  • @TheCPUWizard said:

    ...in most (at least USA) places getting rid of " incompetent employees" requires documentation, and this documentation must be free of any potential bias, otherwise lawsuits may follow.

    That's not very hard to do. I've been involved in firing several bad developers and it doesn't require an elaborate process to catch them up.

    Edit: I should point out it's never pleasant to fire somebody, but I'm not shy about doing it if I have to. I know they will land on their feet because there are countless organizations out there that will hire mediocre people in the belief that their almighty process can turn a frog into a prince. An amusing corollary: the incompetent developers are usually the ones most hell-bent on generating loads of deadweight process (this isn't to imply that all process-happy people are incompetent). It seems they are aware that loads of process gives them a niche they can survive in. They always end up back at big enterprise shops, happily churning out process-abiding WTFs for their moribund departments. These are the same people who will give an estimate of 18 months to do something that should take 3 months, because they intend to rewrite the entire thing in Java making ample use of SOAP for no discernible reason.



  • @morbiuswilters said:

    @TheCPUWizard said:
    ...in most (at least USA) places getting rid of " incompetent employees" requires documentation, and this documentation must be free of any potential bias, otherwise lawsuits may follow.

    That's not very hard to do. I've been involved in firing several bad developers and it doesn't require an elaborate process to catch them up.

    Edit: I should point out it's never pleasant to fire somebody, but I'm not shy about doing it if I have to. I know they will land on their feet because there are countless organizations out there that will hire mediocre people in the belief that their almighty process can turn a frog into a prince. An amusing corollary: the incompetent developers are usually the ones most hell-bent on generating loads of deadweight process (this isn't to imply that all process-happy people are incompetent). It seems they are aware that loads of process gives them a niche they can survive in. They always end up back at big enterprise shops, happily churning out process-abiding WTFs for their moribund departments. These are the same people who will give an estimate of 18 months to do something that should take 3 months, because they intend to rewrite the entire thing in Java making ample use of SOAP for no discernible reason.

    I am not "Shy" about it either. In the past 36 months, I had six clients where there were repeat problems caused by specific individuals. This was documented and presented to client management. Suprisingly, at four of the clients, the individuals in question were (eventually) fired based on this information. In three of the four cases (75%) the company was sued and there was a financial impact to the company (I believe most settled, but I know at least 2 went to court). There were multiple reasons given, including lack of a written (and enforced) policy regarding the actions. In the one case I had to get involved in, I had to testify that there were indeed others at the company who had made "similar mistakes" (not of the same severity or frequency), and the ex-employee's lawyer was able to use this to argue bias.

    Even 10 years ago, this would have been fairly rare (at least based on my experience). But with the current economy, and the litigous nature of American society, I believe many people are resorting to this if/when they have problems with finding new employment.


  • ♿ (Parody)

    @morbiuswilters said:

    @boomzilla said:
    This sort of thing helps the smart people, too, since they don't have to spend so much effort on even the simplest parts. Also, a good process makes it easier to do things correctly (and correctly more quickly) than to make mistakes.

    I believe process should exist to help the good people do a better job, not to help the mediocre people do a somewhat-better-than-mediocre job.

    Yeah, you've made abundantly clear that you don't really understand these concepts. In a perfect world, I wouldn't disagree with you on this point, but the real purpose is to simply improve the firm's performance at the a given level of effort, regardless of individual skill. Most firms won't be able to avoid mediocre (or worse) employees, so they don't have the luxury of tailoring their process to just the highly effective. Of course, that's not really a one dimensional metric, and a good system will help people who have very different strengths and weaknesses.

    @morbiuswilters said:

    @TheCPUWizard said:
    ...in most (at least USA) places getting rid of " incompetent employees" requires documentation, and this documentation must be free of any potential bias, otherwise lawsuits may follow.

    That's not very hard to do. I've been involved in firing several bad developers and it doesn't require an elaborate process to catch them up.

    Of course, every place is different. You reminded me of Some Observations About Firing. The comments are especially interesting, as they give a fairly wide perspective on the various reasons why people don't get fired.



  • Could be worse you could be working on a program that has incompetent coworkers and no processes.



  • @TheCPUWizard said:

    I am not "Shy" about it either. In the past 36 months, I had six clients where there were repeat problems caused by specific individuals. This was documented and presented to client management. Suprisingly, at four of the clients, the individuals in question were (eventually) fired based on this information. In three of the four cases (75%) the company was sued and there was a financial impact to the company (I believe most settled, but I know at least 2 went to court). There were multiple reasons given, including lack of a written (and enforced) policy regarding the actions. In the one case I had to get involved in, I had to testify that there were indeed others at the company who had made "similar mistakes" (not of the same severity or frequency), and the ex-employee's lawyer was able to use this to argue bias.
    So true.  At my previous church this occured with the school's janitor (the school being part of the church).  He was a lazy idiot that would just talk all day and not do any actual work.  The church's head trustee had to step in since no one else was doing anything about it and eventually got the guy fired.  The idiot then tried to sue the church for firing him, fortunately the head trustee had collected a ton of evidence to show how incompitent he was, so the case got dropped rather quickly.

    People in the US love to sue, and even recently Yahoo is now trying to sue Facebook.  We really need some reform over how suing works.



  • @boomzilla said:

    Yeah, you've made abundantly clear that you don't really understand these concepts. In a perfect world, I wouldn't disagree with you on this point, but the real purpose is to simply improve the firm's performance at the a given level of effort, regardless of individual skill.

    I understand just fine; you're the one who seems to think process is free and that tons of process can make up for a lack of intelligence. I hope you enjoy wasting your life in a dead-end job, sending out memos reminding people to breathe and to keep beating their hearts.

    @boomzilla said:

    Most firms won't be able to avoid mediocre (or worse) employees, so they don't have the luxury of tailoring their process to just the highly effective.

    So I should waste my time because other companies are staffed by morons? I didn't get into this career because I wanted to babysit people or spend all day in meetings. Did you?



  • @TheCPUWizard said:

    The test environment should have a constant stream of transactions (like production), and the test should verify constency. This "framework" should be outside the *specific* test for the proposed change so that it is run on *every* change that is being tested.
    Excuse me while I clean the Pepsi that shot out my nose off the keyboard....

    That works great for the 20 multi-million dollar applications we have.  But for the other 880 applications, just setting up a test environment that robust would have killed the project in the first place.  I would rather pay for good people than insist that every application be guarded by millions of dollars of idiot-proofing. Some should, but certainly not all, or even the majority.



  • @Anketam said:

    Could be worse you could be working on a program that has incompetent coworkers and no processes.

    1. I'd quit. 2) If for whatever reason I couldn't quit, I'd prefer no process. It's going to be a clusterfuck either way, but at least it's easier to slack off with no process. Plus it's funnier when the shit hits the fan.


  • @Anketam said:

    We really need some reform over how suing works.

    Don't hold your breath (or do, and then sue me when you pass out and hit your head on the coffee table). The people who write the laws benefit from a cumbersome legal system with ample opportunities for large-figure lawsuits. It's like creating a pedophile task force to help clean up child pornography.


  • ♿ (Parody)

    @morbiuswilters said:

    @boomzilla said:
    Yeah, you've made abundantly clear that you don't really understand these concepts. In a perfect world, I wouldn't disagree with you on this point, but the real purpose is to simply improve the firm's performance at the a given level of effort, regardless of individual skill.

    I understand just fine; you're the one who seems to think process is free and that tons of process can make up for a lack of intelligence. I hope you enjoy wasting your life in a dead-end job, sending out memos reminding people to breathe and to keep beating their hearts.

    What? Where did I imply that process is free? It's true that for certain issues, process can make up for a lack of intelligence. Or attention to detail, etc, etc. Of course, super duper controls aren't appropriate for every environment, as Jaime mentioned, and I'm sorry that I didn't put in enough extra clauses with caveats to make that clear (insert blakeyrant about proactive pedantic dickweedery here).

    There's always a cost, but you need to compare that to the benefits. And of course, the goal is to make a process that more than pays for itself. In the context of this thread, that would be by reducing down time and recovery operations.

    @morbiuswilters said:

    @boomzilla said:
    Most firms won't be able to avoid mediocre (or worse) employees, so they don't have the luxury of tailoring their process to just the highly effective.

    So I should waste my time because other companies are staffed by morons? I didn't get into this career because I wanted to babysit people or spend all day in meetings. Did you?

    This line of attack is purely from your imagination. I never said anything about meetings or memos. I'm sorry for your poor experience in dead end jobs and the resulting lack of imagination. Frankly, one benefit of a well defined (and effective) process is that it's easier to weed out the true morons, and there's less babysitting required.

    Instead of thinking about sending memos around for signature, think about having a sane configuration management environment, including your issue tracker and source control, so that you can understand what's in each build, and what level of testing you've completed. Sure, smart people working really hard can keep track of that in their heads for some projects (and for small enough projects, maybe that's appropriate), but that's a pretty ineffective way to work on non-trivial stuff.

    As an analogy, consider the difference between C and C#. One benefit of C# is that you've eliminated a whole category of error in the form of string buffer overflows without sending a single memo or requiring any meetings. You may commence stretching this analogy beyond its limits to show...something.

    My guess is that you've actually set up some sort of process that's fairly decent. But for some reason, you're equating "process" with bureaucracy and the memos and meetings that go with it.



  • @morbiuswilters said:

    @Anketam said:
    We really need some reform over how suing works.
    Don't hold your breath (or do, and then sue me when you pass out and hit your head on the coffee table). The people who write the laws benefit from a cumbersome legal system with ample opportunities for large-figure lawsuits. It's like creating a pedophile task force to help clean up child pornography.
    No plan on holding my breath.  I know it is just wishful thinking, but can't I have dreams?



  • @Jaime said:

    Nope.  Two people, both morons.  We actually had two implementation disasters this week.  One was moron #1 with moron # 2 as a reviewer and the other was moron #2 with moron #1 as a reviewer.

    Okay, my bad - I assumed the probability of encountering two morons simultaneously in the same chain of events was quite low. Seems... not.

    @Jaime said:

    I sat in the pre-deployment meetings for both changes, and I made additional testing recommendations for both.  Both failures were directly a result of my recommendations not being followed correctly.  They didn't refuse to do the additional testing, they simply bumbled it to the point of not finding the problem.

    This brings the red mist for me. What is the bucketfucklet point of sitting in on a meeting, providing contributions if they're simply ignored? Worse still, if this situation COULD have been avoided but wasn't? Shit me a fuckbrick.

    @Jaime said:

    The difference this year is that we've lost of lot of good people leaving us with a higher concentration of morons than in the past.

    Remove a gold ring from the pig sty, some shit's gonna fill the recently-vacated spot...

    @Jaime said:

    I lost my cool a few times due to frustration.

    Don't blame you, given the tales you're posted here. Wouldn't recommend it, but I can see why (and it's happened to me on occasion. Now I play passive-agressive smug shit)

    @Jaime said:

    Getting along counts more towards our performance reviews than doing your job well.  So, I do have a higher-up and he has flagged an issue -- me.

    Your face don't fit anymore, bitch. Polish up that CV and prepare to jump ship.

    It sounds like your workplace is becoming a moron magnet... pretty soon you'll be the only astronomer at a meeting of the Flat Earth Society. For the sake of your sanity, frustration and anger, GTFO.

    In the meantime, keep posting WTFs. Snoofle's been somewhat slow this week.

     



  • @Anketam said:

    Could be worse you could be working on a program that has incompetent coworkers and no processes.

    No, I couldn't - or rather, I wouldn't.

    If I found myself stuck in this position, I'd first establish a process so I knew what the hell it was I was trying to do.

    Then I may try to encourage the morons others to follow this process to at least provide consistency of output and ensure that my deliverables matched theirs (or vice versa).

    Yeah, I know this looks like I'm taking ownership and showing initiative and all that... but bollocks if I'm in a position where my effort could amount to nothing and I had the opportunity to do something about it and did fuckall. It's not in my nature to follow sheep.

    Note: I'm reading "incompetant" as "don't yet know how to" rather than "don't show any interest in knowing how to and will resist any attempts at being shown how to". Fundamentally, most people are receptive to change if sold on the benefits and properly engaged. Behaviour can be shaped with the right tools n techniques.



  • @boomzilla said:

    There's always a cost, but you need to compare that to the benefits. And of course, the goal is to make a process that more than pays for itself. In the context of this thread, that would be by reducing down time and recovery operations.

    Once again, I never stated I had anything against process. What I detest is process that attempts to compensate for outright stupidity. I have never seen that become more valuable than just getting rid of the problem in the first place.

    @boomzilla said:

    This line of attack is purely from your imagination. I never said anything about meetings or memos. I'm sorry for your poor experience in dead end jobs and the resulting lack of imagination. Frankly, one benefit of a well defined (and effective) process is that it's easier to weed out the true morons, and there's less babysitting required.

    That isn't at all what we are talking about here. Re-read the thread. We aren't weeding out the morons; you guys have explicitly said that it is not possible to get rid of the morons and that we must create process to compensate.

    @boomzilla said:

    My guess is that you've actually set up some sort of process that's fairly decent. But for some reason, you're equating "process" with bureaucracy and the memos and meetings that go with it.

    No, you guys are the ones who equated process with bureaucracy. Maybe that's not what you intended, maybe your process is sane, but it's what you stated. Process should not exist for the sake of curtailing incompetent individuals who have no business in the position they are in. I don't care how much process you have, if a surgeon doesn't know the difference between a liver and a lung I am not going to let him operate on me. Nor would I implement a rule that said all surgeons (including the intelligent ones) must verify the organ they are operating on against a photo in a textbook and have the signoff of 2 other surgeons and at least 1 manager before they cut.



  • @Cassidy said:

    Note: I'm reading "incompetant" as "don't yet know how to" rather than "don't show any interest in knowing how to and will resist any attempts at being shown how to". Fundamentally, most people are receptive to change if sold on the benefits and properly engaged. Behaviour can be shaped with the right tools n techniques.

    Am I wearing a tweed jacket with leather patches? No? Then I'm not your fucking teacher. I'm more than happy to impart knowledge but I'll be damned if I'm going to take someone who knows nothing and teach them from the ground up.



  • @Cassidy said:

    @Jaime said:
    Getting along counts more towards our performance reviews than doing your job well.  So, I do have a higher-up and he has flagged an issue -- me.

    Your face don't fit anymore, bitch. Polish up that CV and prepare to jump ship.

    It sounds like your workplace is becoming a moron magnet... pretty soon you'll be the only astronomer at a meeting of the Flat Earth Society. For the sake of your sanity, frustration and anger, GTFO.

    In the meantime, keep posting WTFs. Snoofle's been somewhat slow this week.

    I'm holding out hope.  I have a lot of say in the hiring process and we recently extended offers to two people I recommended (both accepted).  Both hires will be over moron #2.  If I can get a strong hire over moron #1 the tide can shift back.  By simultaneously adding some talent back to the pool and showing these two that screwing up gets people hired above them instead of promoting them and hiring under them, a strong message can be sent.  On the other hand, if the new hires don't work out, then I'm probably out.


Log in to reply