The Reporting Nightmare



  • Hi All,

    Long time reader, first time poster so go easy on me.

     A long time ago I couldn't decide what to do with my life, except that it had to involve computers. I decide to take a working year out before university, experience the working world (beyond shelf-stacking in the local electronics store) and hopefully find out what I was really interested in.

    As it turns out, it was one of the best choices I ever made. I managed to land a job with a local firm, but the job was a bit "wtf". My role was supposed to be programming support to management, but it ended up being "translator of management BS into specifications for Out Sourcers". You see, at the time I joined, the company was in the process of creating its first software product. To do this, they'd hired a programming firm. However the product was fairly complex, and they often had problems conveying what they wanted (my first meeting with them resulted in my manager drawing the layout of a new user interface feature on the back of a business card). My job, was to take the improvements and turn them into VB apps which we could then demo to the actual developers. Now that was a WTF in itself - because to make the proof of concepts work, I had to re-implement almost the entire thing (If I knew then what I know now - I wouldn't have!). 

     Anyway, we're still not on the real WTF. The real WTF came in the form of the lovely reporting mechanism. It basically consisted of a set of templates, created by the user,into which they placed various fields.The reporting system would run thought the product for various "objects", each object had its own template and when the reporting system came across a particular object type loaded the template and replaced the fields. Simple right? The outsourcers decided to implement this all in house. Reporting generation for a simple 5 "object" project took over 2 hours.Typical real-world projects contained 100's of objects, so lord knows how long that would take (we never tested it... It took too long).

    My manager decided this reporting was all lovely, but it needed graphics. Perhaps a diagram of the flow of information from one object to the next. It shouldn't be a problem, we have a window that displays that already... just add it to the report right? 4 Months of billable time later, and they were done. The simply 5 "object" report... now took 4 hours, and the whole system depended upon Microsoft Word 97. If you had a document already open in Word at the time ... well, hope you saved it.

    Towards to the end of my 12 months, the company decided to part ways with the outsourcers. It was becoming too expensive to use them, they would fix one thing, break another, it was a nightmare. So, during the parting of ways they got hold of the latest source code and dumped it directly in my lap. With 1 month to go until University (which frankly couldn't come soon enough at that point) I was tasked with taking a look an making various fixes/improvements. The fixes didn't take long, but the improvements.. well top of the list was the reporting. So I broke into the code, and after a couple of days of investigation found the following;

    For each report, the code did the following;

    1. Open a window and make it invisible quickly, as we will use the Windows RichEdit control to hold the entire report (in RTF format) and this new window will be its parent.

    2. Because of a limitation in the RichEdit control we were using (not the windows standard), create a new RichEdit for each object;

    For each object in the project;

    1. Load the object by opening up a window on screen. Quickly make the window invisible (how quickly depended on what else you were doing at the time - and these windows were "always on top"). All calculation code was done in the user interface you see.

    2. Identify the object type by reading the window caption.

    3. Load the correct template from the database into the destination RichEdit control.This normally took a while,because it was saved across various VARCHAR 255 fields. I'm not kidding.

    4. Do a ReplaceAll against every possible field for that object against the template. There was a different function for every object type, all 92 of them. Some of the objects had a couple of hundred fields.

    5. Do a search for the graphics field text. Since no field delimiters were used (they were removed by the developers because according to their code comments, they were confusing), the following also matches; graphic, Graphic, GRAPHIC

    6. If we find the graphics field text, make the window visible and take a screenshot of it as WMF and put it on the clipboard (thus blowing away with whatever the user had on there - no attempt is made to keep the previous contents)

    7. Start Microsoft Word using a COM object, and create a new document.

    8. Paste the graphic into Word. Copy it from Word.

    10. Paste the graphic into a Bitmap object. Edit the Bitmap object to remove the window border, caption and buttons (using fixed heights and widths - so on different machines it would cut off important information)

    11. Copy the Bitmap Object back into the clipboard

    12. Close Microsoft Word.

    13. Move the mouse to the location of the Microsoft Word "save this document" message box, click "No". After a bug report that this was annoying, they fixed it so a key was pressed instead.

    14. Paste the Bitmap into the correct location of the Graphics field in the report template. Do not remove the original field text.

    The only thing missing was the Table. Bare in mind, this was one among many, many serious WTF's in that code (the odd colour interface, the messy database structure, the "network support").

    My fix... which I was mighty proud of at the time;

    First, I replaced the whole Word dependency with a DLL to take a screenshot of the client area only (so no editing the bitmap required) and in a Bitmap directly. Didn't even have to use the clipboard. Dropped from 4 hours, back down to 2. Management were impressed.

    Then I discovered memo fields (having never worked with databases before). That 2 hours, now took 1 hour. Management were ecstatic

    Then I discovered the mistake of having the UI doing the calculations. I saved the calculations to the database (I was young, it was quicker). Less then 15 minutes to run the whole test report. Management pretty much wanted my babies.

    That ladies and gentlemen paid for my entire University education. That, and replacing the entire product in 12 months (and not in VB) while at University. Now that was an eye opener...


  • :belt_onion:

    The only thing I'm particularly surprised by is that using Microsoft Word and editing a bitmap took two hours in and of itself. That must have been some fairly insane code.

    @deathy said:

    go easy on me.

    Haha, you must be new he-- oh, wait.



  • It sounds like your first experience in the real world was, sadly, typical.

    OTOH, without first having formal education in technology, you seem to have had a knack for it. Kudos!



  • What snoofle said.

     



  • @heterodox said:


    The only thing I'm particularly surprised by is that using Microsoft Word and editing a bitmap took two hours in and of itself. That must have been some fairly insane code.

    Normally, it wouldn't, but they were starting and stopping Microsoft Word every time. The delay, from what I remember, came from the constant starting and stopping of Word. Editing the bitmap, while messy code, was actually reasonably fast. 


    @snoofle said:


    It sounds like your first experience in the real world was, sadly, typical.

    In my second ever job, I came across plenty of WTF's... but everyone remembers their first...

    @snoofle said:


    OTOH, without first having formal education in technology, you seem to have had a knack for it. Kudos!

    High praise from Mr Snoofle, thankyou! 


  • Garbage Person

     Actually, I'm rather impressed by that course of recovery. Patching out the most heinous crimes first, and THEN replacing the whole damned thing.

     Now, the real question at hand is: What country were the outsourcers from?



  • I can think of two, right about now... India and Pakistan.

    But that's just my 0,02€ worth... or my trolling... whichever tickles your fancy.



  • @Weng said:

     Now, the real question at hand is: What country were the outsourcers from?

     

    The scary thing is.. they were from the UK, the same as the company hiring them. 

     



  • Sounds like fairly typical shoddy as hell British engineering.



  • Awesome story. A lot better than front page content.
    Still... I have a bit of a problem believing you handled this so masterfully without any prior experience or formal education... but hey, I'm not saying you're not just talented.



  • @snoofle said:

    It sounds like your first experience in the real world was, sadly, typical.

    OTOH, without first having formal education in technology, you seem to have had a knack for it. Kudos!

     Holy Sh--... I actually agree with Snoofle <grin>... Kudos!


  • Garbage Person

    @nexekho said:

    Sounds like fairly typical shoddy as hell British engineering.
    British IT is ALARMINGLY shit even compared to their normal levels of engineering expertise. Case in point: Accenture has HUGE operations there, and is headquartered in Ireland now.



  • It just goes to show: just because you know how to code doesn't mean you know how to code.



  • @Weng said:

    @nexekho said:

    Sounds like fairly typical shoddy as hell British engineering.
    British IT is ALARMINGLY shit even compared to their normal levels of engineering expertise. Case in point: Accenture has HUGE operations there, and is headquartered in Ireland now.

    I think you're recalling the problems with government IT projects and mistakenly extrapolating them to "British IT" in general.  Big fat-cat corporate outsourcers milking the government for pork and delivering crappy quality because of ineffectual oversight is not a problem unique to the UK, but we certainly have it bad here.  Outside the public sector, however, there are a lot more success stories.  Think of companies like ARM, or CSR, for example...




  • I thought that British IT was composed of a wacky trio of an Irishman, a socially inept black man, and a crazy female redhead manager.


  • Garbage Person

    @DaveK said:

    I think you're recalling the problems with government IT projects and mistakenly extrapolating them to "British IT" in general.  Big fat-cat corporate outsourcers milking the government for pork and delivering crappy quality because of ineffectual oversight is not a problem unique to the UK, but we certainly have it bad here.  Outside the public sector, however, there are a lot more success stories.  Think of companies like ARM, or CSR, for example...
    Sorry, Accenture is headquartered in the British Isles. Every single other project could be an enormous under-budget/under-schedule/over-featured success and they'd drag the average down to "raging shitpile".

    This is highly unfair because Accenture didn't originate there, but they clearly chose to relocate there for a reason.

     

    (Has anyone EVER heard a success story with Accenture involved?)



  • @lettucemode said:

    I thought that British IT was composed of a wacky trio of an Irishman, a socially inept black man, and a crazy female redhead manager.

    Y'know, it couldn't hurt to try turning Government IT procurement off and back on again.




  • @Weng said:

    @DaveK said:

    I think you're recalling the problems with government IT projects and mistakenly extrapolating them to "British IT" in general.  Big fat-cat corporate outsourcers milking the government for pork and delivering crappy quality because of ineffectual oversight is not a problem unique to the UK, but we certainly have it bad here.  Outside the public sector, however, there are a lot more success stories.  Think of companies like ARM, or CSR, for example...
    Sorry, Accenture is headquartered in the British Isles. Every single other project could be an enormous under-budget/under-schedule/over-featured success and they'd drag the average down to "raging shitpile".

    Ah, so by "British", you mean "From the Republic of Ireland" do you?  I'm sure that Ian Paisley would appreciate your attitude, but I don't know if anyone would consider U2 to be "a British rock band".




  • @deathy said:

    My fix...

    ...awesome course of action of original poster

    Can I have your babies?



  • Sounds like they just automated the task they were doing by hand, without bothering to wonder if perhaps there were more efficient ways... or assuming that because the only way they knew to dump a window's contents to a bitmap involved pasting into word, that was the only way it could be done.



    Closing and restarting Word for every screenshot, now that's insane. Did they leave any hint why? Did you try making it re-use the existing instance to see if everything went (even more) horribly wrong?



  • @DaveK said:

    @Weng said:

    Sorry, Accenture is headquartered in the British Isles.

    Ah, so by "British", you mean "From the Republic of Ireland" do you?  I'm sure that Ian Paisley would appreciate your attitude, but I don't know if anyone would consider U2 to be "a British rock band".

     

    British Isles != United Kingdom.

     



  •  @lolwtf said:

    Sounds like they just automated the task they were doing by hand, without bothering to wonder if perhaps there were more efficient ways... or assuming that because the only way they knew to dump a window's contents to a bitmap involved pasting into word, that was the only way it could be done.

    Closing and restarting Word for every screenshot, now that's insane. Did they leave any hint why? Did you try making it re-use the existing instance to see if everything went (even more) horribly wrong?

    You assume they left alot of comments.

    I will say they were extremely good at inserting a copyright notice at the top of every unit/module.But then you would be too, if you had an automated tool for doing that.

    I did consider just re-using the same instance of word, but remember that management wanted the word dependency gone. So I focused my goal on that first. From what I remember, they restarted word every time because they left word in an "unknown" state - they didn't know what they'd done and didn't want to find out.



  • @Watson said:

    @DaveK said:

    @Weng said:

    Sorry, Accenture is headquartered in the British Isles.

    Ah, so by "British", you mean "From the Republic of Ireland" do you?  I'm sure that Ian Paisley would appreciate your attitude, but I don't know if anyone would consider U2 to be "a British rock band".

     

    British Isles != United Kingdom.

    Yes, that's geographically the case, but I think we're really talking about economic rather than geographical units here.  Look at Weng's argument: he's saying that "British IT" is shit because Accenture, owing to its enormous size, drags down the average.  I'm countering that it's nonsensical to smush together the separate economies of the UK and the RoI in one comparison; it's as misleading a comparison as if I was claiming that "American IT" was shit, and then it turned out I was counting Mexico in together with the USA.

    I think the only valid conclusion Weng can draw from his data is that Accenture IT is shit.




  • @DaveK said:

    Yes, that's geographically the case, but I think we're really talking about economic rather than geographical units here.  Look at Weng's argument: he's saying that "British IT" is shit because Accenture, owing to its enormous size, drags down the average.  I'm countering that it's nonsensical to smush together the separate economies of the UK and the RoI in one comparison; it's as misleading a comparison as if I was claiming that "American IT" was shit, and then it turned out I was counting Mexico in together with the USA.

     Actually you are leaving out all of the Central and South American countries.....



  • As well as the remaining North American countries.



  • I wasn't making a comprehensive list.  Weng didn't throw in all the European countries, just one other country, so I didn't throw in all the continental American countries, just one.



Log in to reply