Code Manglement (literally)



  • Hey it's been a while since I posted here, so I think I'm due.  I finally left the little shop of horrors, so I can finally post some of the resulting WTF's without accidentally identifying myself and getting fired.  Today's WTF:  Worst Practices in Code Management

    We did custom software.  As in every single customer installation was unique like a legacy snowflake.  We didn't use any code management software, we kept a directory structure for each customer system in our office.  We used it for testing and development of further customizations.  At the customer site, we maintained two directory structures:  one for production and one for testing.  Our procedure was that all code changes were made on the customer's test system, tested there, and then migrated to the production directories.  This was fairly simple with a little discipline; don't combine too many enhancement projects, test all changes thoroughly, and then simply copy the executables, libraries, etc over to the production directories.

    It got hairy on large projects that took 6 months or longer to implement.  In theory, you enforce a code freeze on the customer.  They're not allowed to modify their test system for the duration, we would take a backup snapshot of their test system and load it on one of our office computers.  We would do our development and testing in our office, then migrate the changes to their test system for more testing, and then move it to production using the above procedure, and release the onsite code freeze.

    Oh, wait, it gets better.  That's the ideal.

    Most of the time I worked with a PM who was a total softie and couldn't enforce a code freeze on the customers.  He also couldn't lean on them to have any sort of discipline, like cleaning data or being prompt with requirement feedback or anything.  So many of our projects were delayed for months with the ball firmly in the customer's court.  Meanwhile, they of course would whine that they had changes to make to their system.  Technically minor changes, but important for their business.  So this PM let them call the shots which made more work for me.  This all came to a head with my last project before I left.  It went sort of like this:

    Customer asked for large hardware migration plus software enhancements.  We decided to do this in a two-phase approach.  First we'd do the hardware migration, then we'd work on the software.  Their system had many interfaces to external hardware and software, so even the hardware migration meant loads of testing (and porting some of their software).  We had several developers working on the project, and I was team lead.  In order to avoid code-freezing them, the PM made us copy their test system to another test system on *their* computer and renamed the new copy "development".  So on the customer's computer, we have Production, Test, and Development.   Test was the customer's playground, and Development was ours.

    But that wasn't good enough, we also needed a copy of their system in our office.  Despite the complexity, this all went fine for the first phase.  Several months into the second phase we noticed some code inconsistencies between our systems and theirs.  Discovered that some of my coworkers had modified programs onsite, one coworker made his changes to Test, another one made changes to Development.  Meanwhile I was unknowingly changing our copy of their system.  When the problem came to light, I was told to merge all the changes together and make sure the systems were all working and not let it happen again.  I was also told to do a better job of a) managing my coworkers, and b) managing the code changes.  Remember we didn't have any code management in place.  The best I could do was maintain a list of modified files in Excel and pester my coworkers to tell me when they changed something.

    And I should point out that "merging changes" meant manually - load both versions of a file into an editor, compare them and copy changes from one to the other.  We had a diff utility, which I used, but it was incapable of telling me *why* the modules were different, only that they were.  So I had to use my intuition to figure out what was deleted, inserted, simply changed.  And if you're still conscious after this nightmare description enough to wonder; yes I had to do this repeatedly and got pretty stinking tired of it.

    When the whole thing melted down, as I'm sure you guys know it had to, I was held responsible. I was also treated as a moron for suggesting the use of modern code management tools like subversion.  It's your job to do this, stop trying to avoid responsibility!



  • So let me get this straight: you didn't even have source control in place?  From what I recall of your previous posts, and the current scenario you describe, it sounds like the place had some connection to manufacturing.  If I'm correct, wouldn't that sort of dumb-ass approach fail all kinds of certification tests and the like?  And if I'm completely off-base, what line of business was this madhouse operating in?



  • Healthcare...  :P



  • @C-Octothorpe said:

    Healthcare
     

    Sounds like a resonable assumption to me.



  • At some point you just have to say "screw management" and install a subversion (or whatever) server on your desktop for developers to use. This is ridiculous.

    Of course that assumes your co-workers give a crap.



  • @blakeyrat said:

    Of course that assumes your co-workers give a crap.

    This.

    Herding cats is easier than trying to push something "new" like version control onto a group of old, change-resistant curmudgeons.



  • @blakeyrat said:

    At some point you just have to say "screw management" and install a subversion (or whatever) server on your desktop for developers to use. This is ridiculous.

    Of course that assumes your co-workers give a crap.

     

    They didn't. 



  • @locallunatic said:

    @C-Octothorpe said:

    Healthcare
     

    Sounds like a resonable assumption to me.

     

    Reasonable, but wrong.  It was 911 dispatching systems.  Because near-real-time critical  systems don't need no steenking source management tools.



  • I know my OP was a bit tedious to read, it was THAT complicated to describe what we did.  I got really resentful toward the end because in order to save the customer from having to lift a finger, the PM was making me do TONS of extra work.  I started to wonder if he would make me create and then manage five, six, seven copies of their system.  He made other developers do that too, because I used to manage the disk farm in our office and I would routinely see multiple copies of different customer systems.



  • @jetcitywoman said:

    @locallunatic said:

    @C-Octothorpe said:

    Healthcare
     

    Sounds like a resonable assumption to me.

     

    Reasonable, but wrong.  It was 911 dispatching systems.  Because near-real-time critical  systems don't need no steenking source management tools.

     

    Ah, so it was even more of a defects are a big deal.  For some reason the bigger deal defects are the worse the process to make the code seems to be (at least in many cases, or maybe it's just a reporting bias due to these place's bugs being a big deal nature).



  • @jetcitywoman said:

    I would routinely see multiple copies of different customer systems.

    That's a favourite of mine. 

    Root\app

    Root\new_app

    Root\app_bak

    Root\app_from_companyname

    Root\app_2011-01-02

    Root\app_real_new

    Ok hotshot, which is the current version?

    PS - this is how it looked in source control!!!



  •  I would think that in high-risk software with extreme sensitivity to downtime, you'd implement best practices and enforce them stringently.  Not at this place, though.



  • Flavor Flav called and wants to thank your former employer for helping his career.



  •  I keep thinking of details I left out.  While most of our systems were written in Cobol, making the source control a headache but not a nightmare, the specific project I outlined as taking the flak for was written in Assembler.  So yeah, try to imagine manually merging changes from several developers in a 50 year old language that is the equivalent of mowing the lawn with tweezers.  You don't just glance at the code and go "oh, he was adding this feature".



  • Definitely not subversion. You want something that can precisely carefully track common ancestors regardless of how the commits are being shipped around or built upon. You want a DVCS, like git or bzr or hg. Definitely not subversion.



  • @jetcitywoman said:

    And I should point out that "merging changes" meant manually - load both versions of a file into an editor, compare them and copy changes from one to the other.  We had a diff utility, which I used, but it was incapable of telling me *why* the modules were different, only that they were.  So I had to use my intuition to figure out what was deleted, inserted, simply changed.
     

    Too little too late, I know, but I'd just like to give a heads up that Total Commander has all of this.

    # full directory sync/compare with indications of what's new, different or deleted, with all the cool filtering/view options;
    # file compare that tells you what's changed, even within a line (eat shit, VSS!), and you can edit the files inside the differ to immediately copy these diffs to the other file, or just plain edit to taste (eat more shit, VSS!).

    Yeah, I really can't live without TC any more :)

     



  • DVCS +1



    You wouldn't even need to get anyone else to use it. You could just have a DVCS on your own machine with different branches/repos/whatever to represent each environment. It's no silver bullet but at least its under control, the DVCS can track common ancestry and you have a record of about who changed what and sort of "when".



    Obviously too late, but DVCS is great for analogous, and much less disastrous situations.



  • @blakeyrat said:

    At some point you just have to say "screw management" and install a subversion (or whatever) server on your desktop for developers to use. This is ridiculous.

    I'd just install a version control system for me to use. The distributed ones (git, mercurial or bazaar) can simply version a directory without installing anything anywhere, so they are well suited for that. Than when merge time comes, I'd dig the base out of it and be quickly done and maybe others would notice the point in having some version control. And if not, well, it's their problem.

    In fact a former colleague of mine actually does just that. They deploy on VMS servers that have filesystem which remembers old versions, so it's deemed sufficient by the management, but does not help you track changes you do on another machine, like your development box. So when he takes snapshot of the application to work on, he commits it to git locally, than works on it and when he's ready to deploy, he takes current snapshot, commits it to git again, merges his work and deploys the result. He is sure what he's doing and if the others don't give a damn, well, it's their problem.

    @blakeyrat said:

    Of course that assumes your co-workers give a crap.

    This does not.



  • @blakeyrat said:

    At some point you just have to say "screw management" and install a subversion (or whatever) server...

    ...which is why it's called subversion.



  • @Bulb said:

    They deploy on VMS servers that have filesystem which remembers old versions,

    You... you wrote that in present tense. VMS servers still exist?

    @Bulb said:

    So when he takes snapshot of the application to work on, he commits it to git locally,

    VMS exists at the same time as git?



  • @blakeyrat said:

    @Bulb said:
    They deploy on VMS servers that have filesystem which remembers old versions,

    You... you wrote that in present tense. VMS servers still exist?

    @Bulb said:

    So when he takes snapshot of the application to work on, he commits it to git locally,

    VMS exists at the same time as git?

    Maybe it's like computer science steampunk..?



  • @dhromed said:

    eat shit, VSS!

    +1

    @dhromed said:

    eat more shit, VSS!.

    +2

    I hate, and still am forced to use VSS, even though using .Net 4 and VS2010... Explain that one to me...



  • @blakeyrat said:

    @Bulb said:
    They deploy on VMS servers that have filesystem which remembers old versions,

    You... you wrote that in present tense. VMS servers still exist?

    @Bulb said:

    So when he takes snapshot of the application to work on, he commits it to git locally,

    VMS exists at the same time as git?

     

    It does, and in fact the system I was working on was VMS.  Just like the other poster's organization, my manager depended completely on the file versions maintained by the OS.  Well and of course the vigilance of the developers to comment their changes (but you can probably guess what the discipline was like there) and purging the versions was strictly forbidden.



  • @Bulb said:

    @blakeyrat said:
    At some point you just have to say "screw management" and install a subversion (or whatever) server on your desktop for developers to use. This is ridiculous.

    I'd just install a version control system for me to use. The distributed ones (git, mercurial or bazaar) can simply version a directory without installing anything anywhere, so they are well suited for that. Than when merge time comes, I'd dig the base out of it and be quickly done and maybe others would notice the point in having some version control. And if not, well, it's their problem.

     

    Well, after I suggested we use some version control system, management pointed out to me quite clearly that they wouldn't and that it was MY problem.  I want to know how you would use your own and not end up exactly where I ended up, on a project with multiple developers working on the same code.  You'd have to get their versions and merge them into your VCS tree.  And if you botch it they'll just point to your use of the VCS as the source of the problem.  I'm guessing you've never worked for an organization quite like the one I was in!



  • @C-Octothorpe said:

    Root\app

    Root\new_app

    Root\app_bak

    Root\app_from_companyname

    Root\app_2011-01-02

    Root\app_real_new

    Ok hotshot, which is the current version?

    Reminds me of the mistake most people in video editing make, putting "FINAL" in the file name for a video that is to be delivered to the client. Inevitably, the client would review it, find a mistake and ask for a re-render with the fix. This sort of back and forth is unavoidable most times, and not too painful. However, filenames start morphing into "Video FINAL 2 final REALLY FINAL 3.mov"



  • *wanders over to Square Enix HQ and suggests a new title for the next Final Fantasy game...*

    Hey guys, how about Final Fantasy XLII: Archons' Advent Part 7: Gratuitous Use of Punctuation Symbols!!! No Really It Is The Final Fantasy This Time: We Promise!!!


Log in to reply