Change Manglement



  • So we've just implemented a new change management process. As part of this, any changes to production systems need to be documented in full, and reviewed by a coworker from the same team and a manager before changes are made. All well and good, makes sense, no complaints. Well....maybe one.

    I wanted to make a change to one of our mail appliances. My implementation plan was as detailed as it needed to be for any competent engineer - "back up the config. Change x from y to z. Commit the changes. Repeat on all appliances in the pool."

    I submitted it for review. It got rejected - not enough detail. I queried it.

    Boss: It fails at the first step. I don't know how to back up the config. I also don't know where to find any of these values that you're changing.

    Me: I'm making the change, and I do know how to back up the config.

    Boss: But what if you're not here?

    Me: I am here. I want to make the change right now.

    Boss: Granted, but it should be step by step - What if someone needs to roll back your change?

    Me: Everyone in the team knows how to back up the config, and where to find the values that are being changed.

    Boss: But what if it's not one of your team? You need to document exactly where to click or what to type, step by step, so anyone could do it.

    Me: I don't want anyone to do it. If it's not one of my team, they'd better stay the fuck away from my appliances.



  • @timbstoke said:

    You need to document exactly where to click or what to type, step by step, so anyone could do it.
     

    NO. BAD BOSS. BAD.



  • @dhromed said:

    @timbstoke said:

    You need to document exactly where to click or what to type, step by step, so anyone could do it.
     

    NO. BAD BOSS. BAD.

    I could be wrong

     

    You're not.

    The boss needs to uderstand that Change Management is just about approval/rejection of the change - "should we do it? Y/N?" not "how do we do it?"

    You don't put project implementation plans in a business case.

    There is some truth in what he says - the procedures do need to be documented. Your RFC then could just reference those procedures, rather than repeat verbatim what to do.

     



  • I will be sure to tell this to the surgeon when they tell me what exactly they are going to do during the surgery:

    I don't know how to do step 5 where it says you are going to perform the brain surgery.  Could you add more detail to it so that someone that is not on your team can do it?

     

    Also detailed steps can easily be messed up:

    There was this one time with a patch that required a sysadmin to run a rather long sql update command and they managed to mess it up.  What happened was the step which had the sql command was broken on to two lines for readibility.  The second line started with the where clause.  The sysadmin copied and pasted only the first line and when it did not execute he was like oh the developer forgot the ; rather than copy the rest of the command found on the second line.  Needless to say the 5 second update command started taking hours to run at which point they called asking why it was taking so long.  At this point we were able to figure out how he screwed it up, and was able to roll it back without it commiting.



  • @timbstoke said:

    ...


    Me: I don't want anyone to do it. If it's not one of my team, they'd better stay the fuck away from my appliances.

    Has your boss been using the word fungible when referring to your team? 


  • Trolleybus Mechanic

    @timbstoke said:

    Boss: But what if it's not one of your team? You need to document exactly where to click or what to type, step by step, so anyone could do it.
     

    Whose going to do it? You? Ah, that's what you're getting at. You want to take on my workload. Okay, I'll rewrite it, but first I'll need some crayons.



  • @Cassidy said:

    There is some truth in what he says - the procedures do need to be documented. Your RFC then could just reference those procedures, rather than repeat verbatim what to do.

     

    Oh, I agree, and our documentation is impeccable. It's also stored in a secure area, accessible only to those with authority to actually work on the appliances.

    A quote that was actually used: "Your RFC should include enough detail so that, if necessary, one of the helpdesk guys can implement it."

    No it shouldn't. If anything, it should contain little enough detail to weed out incompetent engineers - if they need to ask for help, they need to have their access revoked until they know what they're doing.



  • @timbstoke said:

    @Cassidy said:

    There is some truth in what he says - the procedures do need to be documented. Your RFC then could just reference those procedures, rather than repeat verbatim what to do.

     

    Oh, I agree, and our documentation is impeccable. It's also stored in a secure area, accessible only to those with authority to actually work on the appliances.

    A quote that was actually used: "Your RFC should include enough detail so that, if necessary, one of the helpdesk guys can implement it."

    No it shouldn't. If anything, it should contain little enough detail to weed out incompetent engineers - if they need to ask for help, they need to have their access revoked until they know what they're doing.

     

    The problem with that policy is that it not only "weeds out" incompetent engineers, but good-but-inexperienced ones as well.  What happens if the brilliant new guy needs to ask for help because he doesn't know how to reticulate the splines as specified in step 5, but could easily figure it out if someone took half an hour to walk him through the process?

     



  • @timbstoke said:

    Boss: But what if it's not one of your team? You need to document exactly where to click or what to type, step by step, so anyone could do it.
     

    You tell him the step by step document can be made, and it will take you 80 hours.



  • @Mason Wheeler said:

    @timbstoke said:

    @Cassidy said:

    There is some truth in what he says - the procedures do need to be documented. Your RFC then could just reference those procedures, rather than repeat verbatim what to do.

     

    Oh, I agree, and our documentation is impeccable. It's also stored in a secure area, accessible only to those with authority to actually work on the appliances.

    A quote that was actually used: "Your RFC should include enough detail so that, if necessary, one of the helpdesk guys can implement it."

    No it shouldn't. If anything, it should contain little enough detail to weed out incompetent engineers - if they need to ask for help, they need to have their access revoked until they know what they're doing.

     

    The problem with that policy is that it not only "weeds out" incompetent engineers, but good-but-inexperienced ones as well.  What happens if the brilliant new guy needs to ask for help because he doesn't know how to reticulate the splines as specified in step 5, but could easily figure it out if someone took half an hour to walk him through the process?

     

    I assume that all new hires get access to your production environment from day one at your place of business?



  • @pkmnfrk said:

    @Mason Wheeler said:

    @timbstoke said:

    @Cassidy said:

    There is some truth in what he says - the procedures do need to be documented. Your RFC then could just reference those procedures, rather than repeat verbatim what to do.

    Oh, I agree, and our documentation is impeccable. It's also stored in a secure area, accessible only to those with authority to actually work on the appliances.

    A quote that was actually used: "Your RFC should include enough detail so that, if necessary, one of the helpdesk guys can implement it."

    No it shouldn't. If anything, it should contain little enough detail to weed out incompetent engineers - if they need to ask for help, they need to have their access revoked until they know what they're doing.

     

    The problem with that policy is that it not only "weeds out" incompetent engineers, but good-but-inexperienced ones as well.  What happens if the brilliant new guy needs to ask for help because he doesn't know how to reticulate the splines as specified in step 5, but could easily figure it out if someone took half an hour to walk him through the process?

    I assume that all new hires get access to your production environment from day one at your place of business?

     

    Not at all. I've been here 4 years now and I have yet to touch a production system.  That's simply Not For Developers.  But new hires do get full access to the development side of things from day one, including VMs that can emulate production systems and servers filled with backup copies of databases taken from various clients.  Why shouldn't they?

     



  • @timbstoke said:

    A quote that was actually used: "Your RFC should include enough detail so that, if necessary, one of the helpdesk guys can implement it."

    No it shouldn't. If anything, it should contain enough detail to indicate what processes need to be followed, which can then be cross-referenced to determine the level of capabilitiy required to execute the activities required of that process.

    FTFY. Ask your boss if he's confusing RFCs with helpdesk instructions.

    @Mason Wheeler said:

    @pkmnfrk said:

    @Mason Wheeler said:

    The problem with that policy is that it not only "weeds out" incompetent engineers, but good-but-inexperienced ones as well.  What happens if the brilliant new guy needs to ask for help because he doesn't know how to reticulate the splines as specified in step 5, but could easily figure it out if someone took half an hour to walk him through the process?

    I assume that all new hires get access to your production environment from day one at your place of business?

     

    Not at all. I've been here 4 years now and I have yet to touch a production system.  That's simply Not For Developers.  But new hires do get full access to the development side of things from day one, including VMs that can emulate production systems and servers filled with backup copies of databases taken from various clients.  Why shouldn't they?

    I think you missed the point. pkmnfrk is satirically highlighting that the brilliant new inexperienced guy is NOT a good choice of someone that needs to follow a defined process when deploying an approved change (but this is dependent upon the business impact and risk the change carries). The brilliant new guy can easily observe the process and learn but would not be responsible for performing the actual work.

    For the same reason all new hires at your place get full access to the dev side of thingsbut not to production - it's a damage limitation (well, risk-reduction) measure.

    As an aside: we were asked to create setup instructions for some of our training courses and we added in just the right amount of knowledge that someone technical (and fairly experienced in software deployment) would have no trouble following them, but those inexperienced would struggle. I argued against my team leader against step-by-step handholding instructions, on the grounds that we weren't teaching people how to do their own setups - if they claim to have the technical expertise, they should be capable of following the instructions no problem.  As a result, we either had business-level people balking at the instructions who then called us in to perform the setup, or techies did it pretty much perfectly but still emailed in to confirm they'd done it right. It really was a night-and-day contrast; there were very few inbetween.



  • There's really only one way to deal with someone like that; give them exactly what they want. To the n-th degree.

    Write the LONGEST, MOST DETAILED procedure you can to do the job, including body movements, mentioning to press the ENTER key after each line, etc.

    For example:

    1. Log into server 1.2.3.4 as user x with password y
    1.1. Procedure to open a putty/whatever window here
    1.2. Procedure to enter the login command here. Remember to press ENTER after typing the command. You should see a prompt that looks like this: xxx
    2. Log into the database server s as user u with password p
    2.1. Enter the command:
         sqlplus u/p@s
    2.2. Press ENTER to submit the command
    2.3. Verify you are presented with a prompt that looks like this: xxx
    ...
    5000. Log out 
    5000.1. Enter: exit
    5000.2. Press ENTER
    

    Congratulations! You have finished procedure xxx!

    After a couple of submissions like that, you'll be getting requests to dial it down and make your change requests more <what they should be>, which is exactly what you want.

     



  • @snoofle said:

    Write the LONGEST, MOST DETAILED procedure you can to do the job
     

    .. when ${boss} asks you to draw up an RFC.

    Then you can fulfill progress reports over the next several weeks showing him how far your detailed instructions have come, and how much more there is to go before someone on the helldesk is able to successfully follow a complex procedure without any prior knowledge or skills.



  •  I tried that once.  BOSS+1 read it and said "Yes, that's exactly what we need.  Do more like this."

     



  • Doesn't change management have a "Possible Risks" section? If so, put "Possible risk: Someone who doesn't know what they are doing tries to do this and fucks the whole thing up. Mitigation: Don't let people who don't know what they are doing anywhere near the system. It's not rocket science."

    As for the "keep implementation out of change control", I'm with that as well. The only point of change control is to give people who might be affected by your change a chance to say "whoah" before you make the change.



  •  We can do better than that, snoofle. Code up a script that turns "type ls /usr" into

    1. Place both hands near the keyboard, but not touching any keys.

    1. b. CAUTION: Proper safety precautions and methods should be folowed while using a keyboard to prevent serious injury. Refer to ISO39547("Reccomended typing practices in the workplace versionn 25, 2010") before proceeding.

    2. Move a finger over the l key

    2. a. the l key is in the middle row, near the right side.

    2. b. The finger reccomended according to  ISO39547("Reccomended typing practices in the workplace versionn 25, 2010") is the left ring finger.

    3. Lower the finger until it makes light contact with the l key

    3. a. CAUTION: Be careful not to touch other nearby keys.

    4. depress the l key fully for between 2ms and 45ms. 

    4. b. CAUTION: failure to depress the key fully may result in malfunction.

    4. c.  CAUTION: failure to depress the key for at least 2ms may result in malfunction.

    4. d. CAUTION: failure to depress the key for less than 45ms may result in malfunction.

    5. Move a finger until it is over the s key.

    5. a. the s key is in the middle row, near the left side.

    5. b. The finger reccomended according to  ISO39547("Reccomended typing practices in the workplace versionn 25, 2010") is the left ring finger.

    6. Lower the finger until it makes light contact with the s key

    6. a. CAUTION: Be careful not to touch other nearby keys.

    7. depress the s key fully for between 2ms and 45ms. 

    7. b. CAUTION: failure to depress the key fully may result in malfunction.

    7. c.  CAUTION: failure to depress the key for at least 2ms may result in malfunction.

    7. d. CAUTION: failure to depress the key for less than 45ms may result in malfunction.

    ... and so forth. So complete, any monkey could do it! (I have to work with instructions that are nearly as 'complete' on a daily basis. So much time spend wading through the verbiage to find the actual instructions. So easy to omit an important step because you couldn't find it!)

     



  • @DCRoss said:

     I tried that once.  BOSS+1 read it and said "Yes, that's exactly what we need.  Do more like this."

     

    I'm pretty sure that's what would happen here too.



  • @havokk said:

    Doesn't change management have a "Possible Risks" section?
     

    <pedant>The process of "Change Management" doesn't, but fundamentally an RFC is a business case for a change proposal, so should contain some element of risk management. How much risk management is to be included on the RFC should be determined by the Change Management policy - but to include step-by-step implementation details on a change request by policy is over-bureaucratic.

    @timbstoke said:

    @DCRoss said:

     I tried that once.  BOSS+1 read it and said "Yes, that's exactly what we need.  Do more like this."

    I'm pretty sure that's what would happen here too.

    Still isn't a reason not to do it. You're simply flagging up that following those instructions is unnecessarily time-consuming, and auditing the Change Management process will either show the large cost for each RFC or the small number of submitted RFCs due to the time taken to complete the RFC content according to policy.

    AIUI: you're still being paid to write, every day you spend upon writing it.. right? (unless you're being paid upon deliverables)



  • @Anketam said:

    I will be sure to tell this to the surgeon when they tell me what exactly they are going to do during the surgery:

    I don't know how to do step 5 where it says you are going to perform the brain surgery.  Could you add more detail to it so that someone that is not on your team can do it?

     

    And then when the surgeon gets runs over a by a bus on the way to hospital, they call in a proctologist and give him the detailed steps.

     



  • @robbak said:

    2. b. The finger reccomended according to  ISO39547("Reccomended typing practices in the workplace versionn 25, 2010") is the left ring finger.

    3. Lower the finger until it makes light contact with the l key

    3. a. CAUTION: Be careful not to touch other nearby keys.

    4. depress the l key fully for between 2ms and 45ms. 

    4. b. CAUTION: failure to depress the key fully may result in malfunction.

    4. c.  CAUTION: failure to depress the key for at least 2ms may result in malfunction.

    4. d. CAUTION: failure to depress the key for less than 45ms may result in malfunction.

    Wonderful. This can only come from a very, very irritated mind.



  • @DCRoss said:

     I tried that once.  BOSS+1 read it and said "Yes, that's exactly what we need.  Do more like this."

     

    "Ok. This documents cost the company $X. I can even detail it more for 2*$X."

    Hey, once I've spent $15k on adminitrative costs to aquire 3 $50 ssl certificates. Yep, I was working to the government, there was no other way to do that.

     



  • The standard deployment procedures for my team include a dated folder out on a share with .SQL files to be run.  Each filename is prefixed with a number, indicating the order to be run in.  The standard procedures state to run each one in sequence.

     So, they decide to share the workload and hand it off to an offshore team that had been trained in our procedures.  They decide they don't want to take so much time and they know better, so they start each script without waiting for the previous script to complete.  Error messages?  No problem, those must just be warnings.  All scripts done executing -- good, close the ticket.

     Needless to say, the databases were not in exactly the state they were supposed to be in after the install.

     


  • ♿ (Parody)

    @The Bytemaster said:

    The standard deployment procedures for my team include a dated folder out on a share with .SQL files to be run.  Each filename is prefixed with a number, indicating the order to be run in.  The standard procedures state to run each one in sequence.

    Right. So TRWTF is not putting them all into one script. Or having a master script that executed those scripts.



  • @boomzilla said:

    @The Bytemaster said:
    The standard deployment procedures for my team include a dated folder out on a share with .SQL files to be run.  Each filename is prefixed with a number, indicating the order to be run in.  The standard procedures state to run each one in sequence.

    Right. So TRWTF is not putting them all into one script. Or having a master script that executed those scripts.

    In my experience you can't fix stupid with better procedures.



  • @robbak said:

    2. Move a finger over the l key

    2. a. the l key is in the middle row, near the right side.

    You forgot to specify what keyboard layout you were using with the directions to where the key is located.


  • ♿ (Parody)

    @Rick said:

    @boomzilla said:

    @The Bytemaster said:
    The standard deployment procedures for my team include a dated folder out on a share with .SQL files to be run.  Each filename is prefixed with a number, indicating the order to be run in.  The standard procedures state to run each one in sequence.

    Right. So TRWTF is not putting them all into one script. Or having a master script that executed those scripts.

    In my experience you can't fix stupid with better procedures.

    No, but you can prevent some mistakes. I'm obviously not defending the clueless here, but if you write the script so that it can't fail due to parts being run out of order, that will always be superior to relying on a correct manual execution, which is vulnerable to at least typos, carelessness and cluelessness. None of which are mutually exclusive, of course.



  • @boomzilla said:

    @The Bytemaster said:
    The standard deployment procedures for my team include a dated folder out on a share with .SQL files to be run.  Each filename is prefixed with a number, indicating the order to be run in.  The standard procedures state to run each one in sequence.
    Right. So TRWTF is not putting them all into one script. Or having a master script that executed those scripts.

    That too.  It was a process that was arround before I was here - and before the DBA group was handled like it is now.  It now remains as a side effect from other items -- they have forced "script lockdown" several days before implementation.  With the type of changes we typically have to do, we sometimes barely get the request in soon enough to make the arbitray deadline, - so different members are working on different scripts and droping them into the folder for approval just before the lockdown deadline.  It is kind of WTFery, but it works for now until we get a better process in place and as the systems mature in this last upgrade so they don't need so many last-minute stored procedure changes.



  • @The Bytemaster said:

     So, they decide to share the workload and hand it off to an offshore team that had been trained in our procedures.  They decide they don't want to take so much time and they know better

    BUSTED!


  • Discourse touched me in a no-no place

    @Cassidy said:

    @The Bytemaster said:

     So, they decide to share the workload and hand it off to an offshore team that had been trained in our procedures.  They decide they don't want to take so much time and they know better

    BUSTED!



  • @PJH said:

    @Cassidy said:

    @The Bytemaster said:

     So, they decide to share the workload and hand it off to an offshore team that had been trained in our procedures.  They decide they don't want to take so much time and they know better

    BUSTED!

    Would it ruin this joke too much if you explained it to us dumb Americans?


  • Trolleybus Mechanic

    @robbak said:

    4. depress the l key fully for between 2ms and 45ms. 

    4. b. CAUTION: failure to depress the key fully may result in malfunction.

    4. c.  CAUTION: failure to depress the key for at least 2ms may result in malfunction.

    4. d. CAUTION: failure to depress the key for less than 45ms may result in malfunction.

     

    You didn't explicitly specify what should occur after that 2ms-45ms window. I followed your instructions, and ended up with a series of "l" across the screen. Without explicit instructions telling me to release the "l" key by releiving pressure and moving my finger in an swift, upwards motion, how can you expect me to deploy your changes?

     



  • @Someone You Know said:

    Would it ruin this joke too much if you explained it to us dumb Americans?
     

    The man in te picture is [url=http://tvtropes.org/pmwiki/pmwiki.php/Series/FatherTed]Father Ted[/url], who has a history of financial impropriety particularly the "Lourdes incident"  which involved a him holding on to a large sum of charitable donations while he took a trip to Las Vegas.  The blue logo is from Ulster Bank which is owned by RBS and has has some difficulties recovering from last Summer's minor, almost inconsequential IT-related hiccup which has left them holding on to large sums of their customers' deposits and being unable to give it back for weeks.

    There, now it isn't funny any more.

     



  • @DCRoss said:

    There, now it isn't funny any more.

    Thanks. Good work.


Log in to reply