Source control of the damned



  • As I'm finally about to leave my current job, I figured I'd vent a bit by posting what is by far the biggest recurring WTF I've experienced in the time I've spent working here: our source control system and its associated processes.

    The good news is, of course, that we have a source control system. The bad news is that everyone tries to avoid using it as much as possible. The worse news is that it's actually for the best that we avoid using it.

    The smarter programmers among us have our own little unofficial subversion or git repositories, either personal or for a small team. These unofficial repositories sit on individual developers' workstations and generally aren't backed up -- at least not regularly and certainly not automatically. The source for a particular project is put into the official source control system only when a release is imminent.

    The official source control system is IBM Synergy. Having never used this system outside of this company, I'm not sure if it's the product that sucks or just that the way our workflow is designed is completely brain-dead, but regardless, nobody (including the Configuration Management department, who oversees source control and releases) seems to have much of any idea how to use it. Despite this, our company pays tens of thousands of dollars every year in licensing fees for this system that nobody understands and the developers universally hate, largely because management is fixed on the idea that "more money = better than" and is very reluctant to consider free source control systems, since they must by definition be crap. And of course whoever was behind the decision to use Synergy in the first place can't stand the idea of having their ego bruised by it being replaced.

    Part of the fundamental problem with our source control system is that most of the software we develop is Linux-based for embedded systems. Yet, everyone in the company who isn't a software developer, including our IT department, doesn't know Linux from their rear end, and would prefer not to touch it with a ten-foot pole. As a result, our Synergy server, like all of our other servers, is a Windows server. This is where the problems start.

    You see, a Windows Synergy server apparently doesn't care very much about preserving the case of file names, POSIX attributes, UNIX-style line endings, and all sorts of other things that Linux build systems care very much about. As a result, if you check your Linux software into our official source control system, and then you check it out again and try to build from that, you are in for a world of hurt. We get around this by checking in a giant gzipped tar archive of our project instead of checking in file-by-file. This of course means that there is no meaningful revision history. You can't do diffs, you can't even see easily which files changed between particular versions unless you go through the trouble of checking out both archives, and recursively diffing them yourself, and there certainly aren't commit messages available for you to reference. This is what leads the competent among us to host our own subversion and git repositories where we can be afforded such luxuries.

    Even when all you're trying to check in is a single archive, the process for doing so is a nightmare. These archives are about 250MB, and the Synergy server is ancient and slow as molasses on a witch's tit. Once you've exported the source code from your subversion or git repository and tar'ed it up, checking in a new version of the archived software involves the following steps (which all must be done in a GUI on a Windows PC):

    • Find the Synergy project for the previous version
    • Copy the entire project, giving it a new version number (takes at least 30 minutes)
    • Create a task for updating the project
    • Check out the newly copied project (takes at least another 30 minutes)
    • Replace the old archive with the new archive
    • Instruct Synergy to scan for changes to the project
    • Tell Synergy to accept the changes (takes 45 minutes to an hour)
    • Check in the project (takes another 30 minutes or so)
    • Mark the task as complete (takes about 10 minutes)

    So checking in your software to the official source control easily takes half a day. But that's just the beginning of the fun! Remember I said that the only time we do this is when a release is impending. What does the process for preparing for a release entail?

    • Fill out a word document template requesting an official build of the software for testing
    • E-mail this word document to Configuration Management
    • Wait for up to several days while CM builds the software
    • Retrieve from CM a physical CD with all sorts of official "virus checked!" etc stamps on it, hopefully containing your binaries
    • Load the binaries from the CD to the device, and do the testing
    • If anything goes wrong, fix the software and start over from the very beginning (checking in a new archive to Synergy)
    • Otherwise, proceed to the paperwork nightmare that is the release process itself.

    And the process that CM follows to create the official binaries has a few WTFs of its own. First of all, rather than just checking out the Synergy project that the developer just spent all that time creating, they first create a completely new copy of the project based on the one the developer just checked in, which will never be used for anything except immediately for checking out the software to build. Next, since CM and IT are both allergic to Linux, CM has no Linux PCs to build on, so they build on Windows PCs running Linux in VMWare. The VMWare machine images were of course created by developers, and each project has its own VMWare linux image that CM uses to build its software. Owing to the fact that the CM build computers are old, not to mention the inherent slowness in using VMWare, a build that takes maybe forty minutes on a developer's PC can take upwards of three hours on the official CM build PC. And, of course, should anything go wrong during the build, CM will either completely fail to notice and deliver you a CD without binaries on it, or a developer will be called into the CM office to troubleshoot.

    Finally, just this week, one of the project managers voiced his opinion that, in addition to using the above process for formal testing prior to a release, we should follow this procedure to produce an official set of binaries every time we want to load new software onto one of our devices, even during active development. That is, every time I add a new feature or fix a bug or correct a typo, I should perform all of the above actions (taking at least two full days and perhaps as much as a week) before I can actually try to run the updated software on its target device. And it's not like we're making nuclear warhead delivery mechanisms or anything here; these are very mundane non-critical systems, and developers here continuously update the software on their test units, reloading them dozens of times every day. 

    As much as it might be neat to get paid to browse the web 99% of my time while I'm waiting for Synergy or for CM to do their things, I think that might get old after a while, and so I'm quite glad that I won't be around to see what comes of that proposition.



  • Tell me you're not government funded .. please!



  • @OzPeter said:

    Tell me you're not government funded .. please!
     

    We sell some stuff to the government, but we're not a government contractor and most of our customers are other companies.

    We are, however, government regulated, which partially explains the paperwork nightmare of the release process that I didn't go into detail over.


  • 🚽 Regular

    I'm a little confused as to why you can't always keep the codebase on your own internal SVN server and when you are ready to make a new build, just put it in Synergy as a tarred file (which one could say is a bastardization of tagging). Afterall, the only time CM is interested in what's in Synergy is when you're ready to make a new build for them anyway, correct? I don't see why CM would ever be in control over source control if they're really interested in the finished product.



  • @Tyler said:

    The official source control system is IBM Synergy. Having never used this system outside of this company, I'm not sure if it's the product that sucks or just that the way our workflow is designed is completely brain-dead, but regardless, nobody (including the Configuration Management department, who oversees source control and releases) seems to have much of any idea how to use it. Despite this, our company pays tens of thousands of dollars every year in licensing fees for this system that nobody understands and the developers universally hate, largely because management is fixed on the idea that "more money = better than" and is very reluctant to consider free source control systems, since they must by definition be crap. And of course whoever was behind the decision to use Synergy in the first place can't stand the idea of having their ego bruised by it being replaced.

    <slander>Perhaps whoever is pushing Synergy was taken out to dinner by the IBM rep...or received more [i]personal[/i] favors.</slander>

    Seriously, just burning the new archive onto CD or DVD and archiving [b]that[/b] would be much cheaper.  Come to think of it, AccuRev is light years ahead of Synergy (based on your description), and its Windows-based server [b]doesn't[/b] mangle UNIX-based source and script files.  While I don't personally think that AccuRev is the most wonderful commercial (read: expensive) CM system, it [b]does[/b] work, and it mostly doesn't lose things.  Its merge editor even works if you keep and eye on it...



  • @RHuckster said:

    I'm a little confused as to why you can't always keep the codebase on your own internal SVN server and when you are ready to make a new build, just put it in Synergy as a tarred file (which one could say is a bastardization of tagging). Afterall, the only time CM is interested in what's in Synergy is when you're ready to make a new build for them anyway, correct? I don't see why CM would ever be in control over source control if they're really interested in the finished product.

     

     This is basically what we do do right now, as I tried to describe in the post. The WTFs are:

    • The company doesn't officially support any useable form source control, so it risks having its software and the associated revision history completely lost or otherwise fucked up
    • "Tagging" the software into Synergy takes multiple hours
    • Getting an approved-for-testing build from a "tag" takes multiple days
    • Testing for release involves several "tagging" iterations
    • Someone is seriously suggesting that we use this "tagging" process for every single change we make, no matter how minor or at what point in the development process we are

  • 🚽 Regular

    @Tyler said:

     This is basically what we do do right now, as I tried to describe in the post. The WTFs are:
     

    Well, you seemed to indicate that you're not always using your own source control.

    @Tyler said:

    Someone is seriously suggesting that we use this "tagging" process for every single change we make, no matter how minor or at what point in the development process we are

    Right, forgot about that part. That's a major pain. However, even barring that, if you have access to a non-WTF source control and can therefore make more changes while your soon-to-be-old tag is processing, you don't really have to spend 99% of your time looking at lolcats and YouTube videos. ;)



  • I can see why you're leaving.  This obliterates everything that's good about source control.   A flat file-system based solution would be better, at least that could be fully scripted. 

     Off topic: "Slow as molasses on a witch's tit." sounds really dirty to me.



  •  @MarkJ said:

    @Tyler said:

    The official source control system is IBM Synergy. Having never used this system outside of this company, I'm not sure if it's the product that sucks or just that the way our workflow is designed is completely brain-dead, but regardless, nobody (including the Configuration Management department, who oversees source control and releases) seems to have much of any idea how to use it. Despite this, our company pays tens of thousands of dollars every year in licensing fees for this system that nobody understands and the developers universally hate, largely because management is fixed on the idea that "more money = better than" and is very reluctant to consider free source control systems, since they must by definition be crap. And of course whoever was behind the decision to use Synergy in the first place can't stand the idea of having their ego bruised by it being replaced.

    <slander>Perhaps whoever is pushing Synergy was taken out to dinner by the IBM rep...or received more personal favors.</slander>

    Seriously, just burning the new archive onto CD or DVD and archiving that would be much cheaper.  Come to think of it, AccuRev is light years ahead of Synergy (based on your description), and its Windows-based server doesn't mangle UNIX-based source and script files.  While I don't personally think that AccuRev is the most wonderful commercial (read: expensive) CM system, it does work, and it mostly doesn't lose things.  Its merge editor even works if you keep and eye on it...

    <pedantics>Technically, that's libel.</pedantics>



  • @frits said:

     Off topic: "Slow as molasses on a witch's tit." sounds really dirty to me.
     

    Depends on the witch.



  • The process itself sounds great, it just takes too damn long.



  • @dhromed said:

    @frits said:

     Off topic: "Slow as molasses on a witch's tit." sounds really dirty to me.
     

    Depends on the witch.

    "I'll choose Bachelorette number THREE, Chuck!"

    [GRAMS: "Lollipops and Roses"]



  • @Tyler said:

    We get around this by checking in a giant gzipped tar archive of our project instead of checking in file-by-file.

    [ ... ] @Tyler said:
    These archives are about 250MB

    [ ... ] @Tyler said:

    • Find the Synergy project for the previous version
    • Copy the entire project, giving it a new version number (takes at least 30 minutes)
    • Create a task for updating the project
    • Check out the newly copied project (takes at least another 30 minutes)
    • Replace the old archive with the new archive
    • Instruct Synergy to scan for changes to the project
    • Tell Synergy to accept the changes (takes 45 minutes to an hour)
    • Check in the project (takes another 30 minutes or so)
    • Mark the task as complete (takes about 10 minutes)

    You know, it could be that Synergy is only sucking quite so hard because of your mode of usage.  Huge big impenetrable binary blobs are generally what all SCMs handle worst.  I know it sounds nuts, but you might actually be better off using non-compressed 'shar' archives to package up the source trees for insertion into the repository, both in performance terms, and also because you'd be able to get at-least-somewhat-meaningful history and changesets!




  • Wouldnt non-compressed commit get mangled?



  • @RHuckster said:

    However, even barring that, if you have access to a non-WTF source control and can therefore make more changes while your soon-to-be-old tag is processing, you don't really have to spend 99% of your time looking at lolcats and YouTube videos. ;)

    The "surfing web 99% of time" was not the current state of things but rather what would happen if the idiotic proposal to always use Synergy goes through.



  • @Shinhan said:

    Wouldnt non-compressed commit get mangled?

    Linefeeds probably, but if the project only uses unix-style linefeeds that's rather easy to fix. Mixed linefeeds are harder to restore.

    A shar archive will store the case-preserved filenames and permissions while still being plain text, allowing meaningful (to some degree) diffs to be generated.



  • @dhromed said:

    Depends on the witch.


    Missed one! Mmmmmmmmm...... Good job in real life she's not a minor :-)



  • @MeesterTurner said:

    @dhromed said:

    Depends on the witch.

    Missed one! Mmmmmmmmm...... Good job in real life she's not a minor :-)
    Not my taste. But you missed three.


  • @MeesterTurner said:

    @dhromed said:

    Depends on the witch.

    Missed one! Mmmmmmmmm...... Good job in real life she's not a minor :-)
     

    We have a different idea of what constitutes Mmmmmmmm but that's okay.



  •  Sounds like you are getting out just in time.

    Just wanted to mention that while I've had no recent experience with Synergy, I can see from the system requirements page for the product that it does support Linux. I suspect that your company just has a really old version and/or they haven't properly configured it to support *NIX style file paths.  

     



  • @Zecc said:

    @MeesterTurner said:

    @dhromed said:

    Depends on the witch.

    Missed one! Mmmmmmmmm...... Good job in real life she's not a minor :-)
    Not my taste. But you missed three.
    You missed one yourself.


  • @DeepThought said:

    Just wanted to mention that while I've had no recent experience with Synergy, I can see from the system requirements page for the product that it does support Linux. I suspect that your company just has a really old version and/or they haven't properly configured it to support *NIX style file paths.  
    He mentioned that in the OP. They're the only department with any understanding of Linux, so obviously if they're going to have a SCMS setup it'll run Windows configured by idiots.



  • @RHuckster said:

    I'm a little confused as to why you can't always keep the codebase on your own internal SVN server and when you are ready to make a new build, just put it in Synergy as a tarred file (which one could say is a bastardization of tagging).

     

     Yo dawg we hurd you like revision control so we put a source code manager in your source code manager so you can manage revisions while you manage revisions.

     YES!!!!!!



  • @ShatteredArm said:

     @MarkJ said:

    @Tyler said:

    The official source control system is IBM Synergy. Having never used this system outside of this company, I'm not sure if it's the product that sucks or just that the way our workflow is designed is completely brain-dead, but regardless, nobody (including the Configuration Management department, who oversees source control and releases) seems to have much of any idea how to use it. Despite this, our company pays tens of thousands of dollars every year in licensing fees for this system that nobody understands and the developers universally hate, largely because management is fixed on the idea that "more money = better than" and is very reluctant to consider free source control systems, since they must by definition be crap. And of course whoever was behind the decision to use Synergy in the first place can't stand the idea of having their ego bruised by it being replaced.

    <slander>Perhaps whoever is pushing Synergy was taken out to dinner by the IBM rep...or received more personal favors.</slander>

    Seriously, just burning the new archive onto CD or DVD and archiving that would be much cheaper.  Come to think of it, AccuRev is light years ahead of Synergy (based on your description), and its Windows-based server doesn't mangle UNIX-based source and script files.  While I don't personally think that AccuRev is the most wonderful commercial (read: expensive) CM system, it does work, and it mostly doesn't lose things.  Its merge editor even works if you keep and eye on it...

    <pedantics>Technically, that's libel.</pedantics>

    Because it's written instead of spoken?



  • @Lingerance said:

    @DeepThought said:
    Just wanted to mention that while I've had no recent experience with Synergy, I can see from the system requirements page for the product that it does support Linux. I suspect that your company just has a really old version and/or they haven't properly configured it to support *NIX style file paths.  
    He mentioned that in the OP. They're the only department with any understanding of Linux, so obviously if they're going to have a SCMS setup it'll run Windows configured by idiots.
    I think [b]Lingerance[/b] has identified TRWTF here.



  • If you could kick out ego-guy, the money requirement might be satisfied by perforce. Although you might have considered that ages ago and now I'm just rubbing salt in your obviously painful wound :-)

    The support you get for the money is actually top notch too. Although I have my beefs with perforce' architecture, the people who answer their phones are worth the money, and it does really work as advertised.


Log in to reply