The Big Project: replacing the engine in a running car going 60

  • I was given the critical role of managing (and largely executing) The Big Project at my company.  You know, the kind of software project that could save a lot of money, improve every employee’s life, increase efficiency and thus revenue, cure cancer, increase the size of your manhood, etc. - or end up being such a time-consuming, costly, boondoggle that the company falls apart.  And, needless to say, the kind where success means moving on up in the world, and failure meaning the potential destruction of my career.  Or maybe I’m just overdramatic.

    Namely, it’s developing a centralized internal system to replace all of our legacy ones, which are presently being held together by that all-purpose glue called Excel, and long-forgotten magicks from the dark ages of computing when terms like “core memory” actually referred to a piece of hardware.  I like to call it "slaying the Kraken".

    I’ll be replacing systems that have been in place for 30 years, validating then migrating all of the data to the new system, and linking it to the new accounting/payroll/TPS/warehouse system in a single swoop so that there isn’t a second of downtime.

    As the analogy goes:  changing out the engine on a running car going 60 on the interstate.

    So, I was wondering if anyone here had experience with these kinds of projects, what it was like, and how common success versus failure has been.  General advice, aside from “run far, run fast”, might be helpful.

    Otherwise:  Any good books about this sort of thing?  i.e. like The Mythical Man Month

  • Create a scalable, modular structure, then migrate applications one bit at a time. Do not, under any circumstances, try to tackle this beast in one go. A monolithical application will only die half-finished after having absorbed most of your financial resources and a few lives. I'm a fan of skeletal applications where you can swap out and add components. Foobar2000 is an excellent example of this, though its size may not be comparable to your project.

    To stick with your metaphor: build a new car on top of the one currently driving, then roll it off the back when it's done, overtake the old one and let it crash into the railings.

    And don't expect to be done next year.

  • Whatever you do, plan it, and don't write a single line of code before you've mapped out what you are going to do. prototypes excluded of course.

    Secondly as dhromed already said, tackle little bits at a time.  And spend time researching and describing how it works now, how the data is structured, what data you have, and only then how you are going to do it in your version.

    Based on that data create milestones, targets, tasks or stories or whatever development methodology you are comfortable with, but make sure you know what has to be reasonably finished when. Not just because it makes it easier to tell your boss when stuff will be finished, but also to limit yourself in the amount of polish you might want to apply and what would be reasonable.

    Depending on how the system interfaces, you might want to begin with creating empty versions of each component, creating internal api's to each other where needed. These can just give and return expected but realistic values, but this will help you in not forgetting that component X needed some info from component B, which you just abstracted away deeply in your system. In essence it avoids too tightly coupled systems, while still giving you the ability to tackle one part at a time.
    Initially this can be done on paper for your technical design, but it can be helpfully to later translate these to actual code via unit tests or just writing some stub code.

    Please note that you don't need to describe absolutely everything before hand, but you should have the big lines drawn out for the entire system, and per bit you tackle you should first solve it on paper. Especially the more detailed bits will change as you work, because you will find better ways to do it, or you might have overlooked something while writing the docs. Be sure to update your docs to reflect this, ideally at the end of the project the docs should still describe how the system works.

    In terms of how detailed it should be, well that's personal preference. Personally I only describe the important bits, perhaps the objects and public methods. How those objects work internally, well that's best decided when writing the code itself. Basically if you are describing boilerplate code, or are describing really detailed implementation details, it is too deep.  But that's just my personal preference.

  • Just do parallel testing.  Keep the legacy around for a week and make people do their work on both legacy and the new system, and compare the outputs. This way, if you have to rollback, you can.

  • Get it in front of the users as soon and as often as you can. They won't really know what they want, and even if you do what they ask for, it'll be wrong, and the only way to find out is to have them touching the system.

    Also, don't be afraid of the 80% solution, especially when going further in one area will really drag you down in another.

  • @boomzilla said:

    Get it in front of the users as soon and as often as you can.


  •  Thanks for all the advice, everyone.

     It's times like these that make me glad I was a business major.  I think if I were comp-sci I'd be totally lost on the accounting system integration.  Though, if I never see another GL code in my life it'll be too soon.

  • "Whackin' the Kraken" more like.

  • @Salami said:

    Just do parallel testing.  Keep the legacy around for a week and make people do their work on both legacy and the new system, and compare the outputs. This way, if you have to rollback, you can.

    Sadly, this system does not work. No business user want to do additional data entry.

  • I'd go at it this way: plan two Big Projects: the Big Project, and a set of eventually-disposable shims and glue that make the Big Project able to disguise itself as pieces of what it's replacing. Data migration would then happen piecewise, online and transparently. The first major goal should be to have Big Project cover only what the legacy system currently does; obviously you'd architect it with an eye to the future, but you don't want to be changing business processes and business-critical software at the same time if you can possibly avoid doing so.

    Do the back end first. Initially, feed it with data scraped and massaged from the legacy system. Make a clean separation between the back end proper and the legacy-system scraper shims; you should eventually be able to simply stop using those.

    Next, do queries and reports. First implement shims that generate legacy-system-compatible reports based on data held in the new back end, and build automated tests that compare those with corresponding reports generated directly from the legacy system. Don't move on until that stuff has stopped generating WTF.

    Now you're ready to duplicate the Big Project back-end, adjust the data-scraping shims to keep both back-end clones updated in parallel, and put one of them into production along with Big Project's legacy-system report emulators. You can then start training your users to generate the exact same reports and queries they're accustomed to, using the new facilities instead of the legacy ones; if you've done your job right this should cause minimal pain and negligible productivity loss. Note that because the entire old system is still there and still being kept up to date, just queried less and less often, you can still fall back to it quickly while you fix the inevitable unexpected Big Project report and query bugs.

    You could actually start implementing native Big Project queries and reports at this point, trial them with a carefully selected set of clueful users, use the trial results to control the progressive disabling of general access to legacy and legacy-compatible queries and reports, and attend swiftly to the howls of pain and outrage that such disabling will inevitably cause.

    Only when the Big Project back end is known to be solid and everybody is using queries and reports run solely against it, so that you no longer care about maintaining data integrity in the legacy system except insofar as you're still scraping newly entered data from it, should you start implementing native Big Project data entry. Going this way saves you the pain of needing to have your Big Project data entry facilities push data into the legacy system as well. You've already got separate testing and production back ends set up, so you can do cutovers from legacy data entry to Big Project data entry in a disciplined piecewise fashion.

    And when the last legacy data entry scraper has been made redundant, you can have a little party.

  • And once again I am TRWTF, having failed to notice that this is a necro post until after contributing a long and redundant screed.

    So, which way did you tackle it, and how's it going?

  •  @flabdablet said:

    And once again I am TRWTF, having failed to notice that this is a necro post until after contributing a long and redundant screed.

    So, which way did you tackle it, and how's it going?


    Basically,  we decided to go with a more or less fool-proof strategy.

    We threatened to sue the ERP system vendor for breach of contract if they didn't have us up and running by X date.   In response, we got dedicated support people and additional resources to help with the migration at no extra charge.

    We cleaned some of the data in the old system, imported it into the new one, and did some parallel testing - had both systems running simultaneously, and compared the outputs.  Once the results looked good and we figured out the shortcomings, we did a fresh import of data from the old system into the new one, and over the course of the Christmas holiday (1 week), we made the transition to finally go-live with the new system.  The old system would remain online just for reference, in read-only mode.

    Throughout the process, we regularly brought the users in to see the system and did lots of trainings.  It brought a number of the staff closer together, in that we were all sharing the pain of having to deal with the new ERP, and got to have lots of fun telling crazy stories of all the bugs, glitches, and just downright stupidity of the system.Ultimately, it was a system that not a single person in the compay wanted, not even the upper management team.  But we paid a lot of money, put in a lot of work, and had a contract.  So we were stuck with it for years to come.

    The migration was anything but smooth.  Some months after go-live, we discovered the inventory management sub-system was double-counting inventory during warehouse transfer; inflating our inventory valuations by a significant margin and screwing up a whole bunch of stuff.

    Not long after that, I discovered that none of the price changes we'd made since go-live actually took into effect.  Over the course of some months, the internal screens and displays showed order entry operators and product managers one price, and customers were charged a different price.  The price displayed varied with the screen it was presented on.  Most screens lied about the real price.  So we were over and undercharging customers tens of thousands of dollars (and sending out that bad pricing data to resellers), cumulatively, for all that time the system error went unnoticed.  I only noticed it because I have a profound mistrust for ERP systems, and regularly go through transaction records and add up figures by hand.  Once I discovered the discrepancy, it just took a couple SQL queries and transaction logs to confirm it. The CFO was less than pleased.

    I could go on and on.  Maybe write a book about it.  😛

    So, that is the abridged version.

    I have since left the company.  Decided to go some place with better pay, better hours, and where the WTFs were primarily coming from clients and not internally from within my company.

    When I left, some six months after go-live, the system was still a mess, there were massive delays in various projects, data still needed to undergo a lot of cleanup (multi-week project), and we needed to purchase/build even more systems to make up for the rather large shortcomings of the new ERP system.  It was unanimously agreed among upper management that they would migrate to a new ERP system as soon as the contract expired for the new one that was just purchased.

    In short - the company survived, and is limping along with crippled internal systems.

    But, to me, that is a million miles and a million years ago.

    Sometimes, though, I am haunted by the whisper of the name of that ERP system in my darkest dreams.


Log in to reply

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.