Integration WTF



  • Background: I work at a private university, doing website development. Office politics are usually the worst WTF I have to deal with, and though these are bad indeed, they still pale in comparison to some of those I've heard here. In this particular WTF, we've been working on sending the data captured by our web forms to our lead-tracking and student management software package so that we don't have to use the wooden-table practice of receiving an email from the form and (gasp) retyping it into the software package.

     I won't get into all the WTFs here stemming from miscommunication in this project (which range from benign things such as someone else missing a phrase in an email--leading to a month's lost productivity--to outright WTFs with managers running into my office screaming bloody murder because I'm a part time worker and full-time student, and thus not available every moment of every day to answer an email within 30 minutes of when it was sent). No, here the software itself is the TRWTF.

    When most of you develop an integration between software packages, you presumably use a semi-sane method of passing data between applications. INI files, CSV files, direct web services requests, even XML isn't really that bad in certain scenarios. However, this integration is... shall we say, "special". Apparently, the only format accepted by the software produced by this vendor is DB2. No ifs, ands, or buts. The vendor also ***HIGHLY RECOMMENDS*** (to quote them) that we produce a CSV and send it to their special "custom-engineered conversion utility" to be converted to DB2. Therefore, we paid said vendor a substantial sum of money to send an "integration engineer" (think: highly paid consultant) to our location to perform the integration. Despite the demands on my time, I managed to produce a decent exporter that would take the POST data from each form submission, convert the field names to their absurd standards (8 characters maximum for a column name? Seriously?), produce a tab-delimited CSV, and use an FTP upload to transfer the file to the antiquated desktop machine running the bridge software--which is sitting in someone's office, next to his desk, without surge protection or power backup. The bridge software runs nightly, and is supposed to convert the CSV to a DB2 file. The software itself is scheduled to run shortly thereafter to import the DB2 file.Though this seemed rather wooden-table-esque to me, I went along with it and did my part.

     The integration engineer spent three days on the campus, typing merrily away. Any delay on my part for such an absurd reason as going to class or doing homework was met with an instant firestorm from people from the department--after all, this was a highly-paid consultant that was being held up by a mere student worker. After three days, he goes home. No one hears from him or his company for a month. Then, suddenly, the head of the department charges into my office, yelling "You need to call the integration engineer NOW!" Mystified as to what the problem could be, I called the integration engineer. He then spends the next 15 minutes explaining the problem: apparently, their "custom-engineered conversion utility", highly promoted by the company and modified "just to our specifications" by the integration engineer, is incapable of dealing with such concepts as a new field in the file, or a different field order, or a different field name, or... well, pretty much any difference whatsoever. We had added a new field to one of the web forms asking a question about how the user had heard of us, and my part of the integration had dutifully created a new 8-character-titled field in the CSV file for that information--which their software will happily track, as it's valid and useful marketing information. However, the conversion utility died miserably when this file was read. Apparently, instead of doing any strange, sane thing such as ignoring unknown fields, or looping through the field names and transferring known field names to the corresponding field in the DB2 table, the conversion utility did a straight column-by-column copy, straight from the CSV file to the DB2 file. This shouldn't have been a problem--after all, I added the new form field at the end of the form, so it was accordingly placed at the CSV file--but the column-by-column copy didn't stop once it had reached the end of the DB2 file. No, indeed--instead, it kept parsing the CSV file and trying to copy data into a nonexistent DB2 column.

    I offered to write the department a better piece of software, but was refused on the grounds that "we've already invested too much in this piece of software--it'll work great after just a couple more updates, right?"

    Moving right along...



  • Look at it this way; this little episode will give you more practical training for what you'll encounter in the real world than anything else you'll learn at school!

     



  • @Kharel said:

    "we've already invested too much in this piece of software--it'll work great after just a couple more updates, right?"
    Any psychology professors at that university?

    (Any unit testers?)



  •  @Xyro said:

    Any psychology professors at that university?

    (Any unit testers?)

     I wish--there are indeed psychology professors, but unfortunately no unit testers. Or test environments. Or testing staff of any kind. Or anything relating to testing, for that matter--we make changes on a live website day-to-day that gets thousands of visitors without a second thought. One mistyped character and, due to the (lack of) error handling and logging, the whole site begins to display a blank white screen, with error data logged into a password-protected directory that, apparently, no one has the password for. The only way to actually get at the error code that's occurring is on the server itself--and guess what? We don't have physical or VNC control over the server. We're stuck using SSH to connect to the server, log in, find the file we're interested in, and enter "php file.php" every time we want to see the cause of a syntax error. The many suggestions I've made for such obscene things as a subversion repository, testing environment, some sort of peer review system, or at least better access to the server have all been rejected due to lack of IT personnel.

    As far as experience, I'm lucky that way--I've had several years of professional software development experience on everything from web servers to desktop applications to--you guessed it--embedded hardware without a filesystem. I've worked as everything from a salaried employee to an independent contractor. Each of those environments had their own WTFs, but nothing approaching the sheer scale and quantity present here, especially in the office politics. I'm seriously considering finding a job with another local development company as soon as possible--I know several people in one that's hiring, and I'll just bet they might pay a competent software developer more than $8.25 an hour (yes, you read that right--for some reason the university does not pay students more than $8.25 per hour, no matter what type of work they do).



  • SEGREGATION NEVER, INTEGRATION FOREVER!

    Wait, what?



  • Just out of curiosity, when you are referring to a DB2 file are you talking about an IXF formatted file or something else entirely? I ask because I know that importing a CSV file's contents into DB2 is a relatively trivial process. Perhaps the bridge process is merely running the OTB DB2 import feature. If that's the case, then your school almost definetly got sold a bill of goods on this one.

    While I understand and sympathize with your predicament, I must say that I'm not entirely surprised that adding a column to the CSV file caused the import process to blow up. As a rule of thumb, any changes that impact an integration point should be reviewed first by whoever supports that integration point to ensure no adverse affects. 



  • @Kharel said:

    produce a tab-delimited CSV

    Speaking of philosophy, if it's tab-delimited, then in what way is it a CSV file?



  • @blakeyrat said:

    @Kharel said:
    produce a tab-delimited CSV
    Speaking of philosophy, if it's tab-delimited, then in what way is it a CSV file?
     

    In this case think "character separated values" rather than "comma separated values". Many applications, including DB2 and common spreadsheet applications allow you to use characters besides commas to delimit the fields in a CSV file.



  • @DeepThought said:

    @blakeyrat said:

    @Kharel said:
    produce a tab-delimited CSV

    Speaking of philosophy, if it's tab-delimited, then in what way is it a CSV file?
     

    In this case think "character separated values" rather than "comma
    separated values". Many applications, including DB2 and common
    spreadsheet applications allow you to use characters besides commas to
    delimit the fields in a CSV file.

    Yes, but when that happens it is no longer a CSV and becomes something else. (That's my position and I'm sticking to it!)



  • @blakeyrat said:

    Yes, but when that happens it is no longer a CSV and becomes something else. (That's my position and I'm sticking to it!)
     

     According to wikipedia it's referred to as a DSV file and I too generally prefer the use of commas in CSV files, but don't really care enough to make a big deal out if it.



  • Doesn't seem like a WTF at all...unless the importer requirements specified that he support arbitrary new columns, you provide unexpected input, and you get unexpected results.  Setting up a DB2 import is not tough, but probably still takes 1-2 days with setting up the environment, getting the requirements, and testing.  Add a day for consultant type things like getting access, and it doesn't sound like the uni got fleeced at all.  Probably you would be able to manage it yourself if you knew the layout of the DB2 tables. 



  • @blakeyrat said:

    @DeepThought said:

    @blakeyrat said:

    @Kharel said:
    produce a tab-delimited CSV
    Speaking of philosophy, if it's tab-delimited, then in what way is it a CSV file?
     

    In this case think "character separated values" rather than "comma separated values". Many applications, including DB2 and common spreadsheet applications allow you to use characters besides commas to delimit the fields in a CSV file.

    Yes, but when that happens it is no longer a CSV and becomes something else. (That's my position and I'm sticking to it!)

     

    I'm with you.

    It's a ^ISV.



  •  @huai said:

    you provide unexpected input, and you get unexpected results
     

    I'm agree. The story doesn't say whether the import script died gracefully or crapped out, but when somebody screwed around with your input file, it's not a bad idea to stop and error out. Unless the specs read: import whatever you're given and assume it's more or less ok.

    The story contains plenty WTFs from all sides, but here the OP should have realised this could be a problem.



  • @Kharel said:

    Then, suddenly, the head of the department charges into my office, yelling "You need to call the integration engineer NOW!"
     

     

    Office? I call shenanigans.



  • @SQLDave said:

    @Kharel said:

    Then, suddenly, the head of the department charges into my office, yelling "You need to call the integration engineer NOW!"
     

     

    Office? I call shenanigans.

     

    We-e-ell, when Kharel says "office", it were only a hole in t' ground wi' a bit o' cardboard pulled over it, but it were an office to us!



  • When I worked for a university as a student-type I somehow wound up with a private office (each four walls, ceiling, desk, PC, printer, door with lock) and a desk in a shared office (5 walls, ceiling, 2 people, 2 desks, 2 PCs, printer, door with lock) at opposite ends of campus. I also had a state-issued cell phone.

    Aside from the rate of pay (I topped out at $14.65/hr at a public university--5 years ago, for goodness sakes--with an admin assistant who carefully watched the pay charts and immediately filed the paperwork to push me to the max each time "max" changed) the "Office" line sounds perfectly plausable.

     Lincoln


  • :belt_onion:

    @lincolnjkc said:

    • private office (each four walls, ceiling, desk, PC, printer, door with lock)
    • shared office (5 walls, ceiling, 2 people, 2 desks, 2 PCs, printer, door with lock)

     

    No floor? Bummer.



  • @bjolling said:

    No floor? Bummer

    It's entirely possible that there was one of those, too, but there was generally so much crap on the floor* in my office I can't swear to it

     

    *- Also known as "the biggest shelf that I could get the university to buy"



  • @lincolnjkc said:

    so much crap on the floor

    Seriously, dude, use the toilet.



  • @bjolling said:

    @lincolnjkc said:

    • private office (each four walls, ceiling, desk, PC, printer, door with lock)
    • shared office (5 walls, ceiling, 2 people, 2 desks, 2 PCs, printer, door with lock)

     

    No floor? Bummer.

    Classic rookie mistake.  You have to have a floor, otherwise they can easily tunnel to freedom.  I recommend no less than 12" of concrete with a mat of steel reinforcement at 6" on center.  #3 bar is usually sufficient, but #4 bar gives you better piece of mind.


  • @bstorer said:

    piece of mind.
     

    You fucking dyslectic imbecile.



  • @dhromed said:

    @bstorer said:

    piece of mind.
     

    You fucking dyslectic imbecile.

    I meant what I said, bitch.  Clearly I was explaining that #4 rebar will tell you what it is thinking, while #3 will sit there all quiet like a little bitch.


  • @bstorer said:

    Clearly I was explaining that #4 rebar will tell you what it is thinking, while #3 will sit there all quiet like a little bitch
     

    You're a lying cunt, fuckface.



  • @dhromed said:

    You're a lying cunt, fuckface.
    The only thing I lied about was having a cunt.  I didn't think you'd mind when you found out, what with you being a fag and all.



  • @bstorer said:

    The only thing I lied about was having a cunt.  I didn't think you'd mind when you found out, what with you being a fag and all.
     

    Truly, I was wonderin' about your extraordinarily tight vagina.

    And the smell.

    Oh god the smell.

    It's like something died in there.

    Like morbius' wang.



  • @dhromed said:

     

    It's like something died in there.

    A lot of things have died in there.  Dozens, hundreds maybe.  It's to be expected, really, when the only things they have for sustenance are the rotten corpses of those who have gone before and whatever mold or fungus can be scraped from the walls of that dark, dank place.  Wait, we are talking about the oubliette in my basement, right?


  • @bstorer said:

    Wait, we are talking about the oubliette in my basement, right?
     

    In a way, we are.



  • @dhromed said:

    morbius' wang
    What is it with you and wangs today?  I thought you liked dicks.



  • @belgariontheking said:

    @dhromed said:

    morbius' wang
    What is it with you and wangs today?  I thought you liked dicks.

    He's being frugal.  A wang is just a Chinese knock-off dick.


  •  Well, this WAS a nice thread until our resident special-needs posters started dry humping each other.





  •  You are all dicks.



  • @Zylon said:

     Well, this WAS a nice thread until our resident special-needs posters started dry humping each other.

    PROTIP: You can still post, regardless of what others are discussing.  But, no, whining like a bitch is good, too.


  • @bstorer said:

    PROTIP: You can still post, regardless of what others are discussing.  But, no, whining like a bitch is good, too.

    See folks? It's ironic because whining like a bitch is exactly what bstorer is passive-aggressively doing in this very post. Apparently he doesn't enjoy being called out on his compulsive infantile thread-shitting.



  • @Zylon said:

    @bstorer said:

    PROTIP: You can still post, regardless of what others are discussing.  But, no, whining like a bitch is good, too.

    See folks? It's ironic because whining like a bitch is exactly what bstorer is passive-aggressively doing in this very post. Apparently he doesn't enjoy being called out on his compulsive infantile thread-shitting.

     

    You realize they're just pulling you in, slowly but surely.



  • @Zylon said:

    whining
     

    You should leave your window open tonight, and I will give BTK a call.

    Leave milk and cookies by the sill -- he likes that.



  • @Zylon said:

    @bstorer said:

    PROTIP: You can still post, regardless of what others are discussing.  But, no, whining like a bitch is good, too.

    See folks? It's ironic because whining like a bitch is exactly what bstorer is passive-aggressively doing in this very post. Apparently he doesn't enjoy being called out on his compulsive infantile thread-shitting.

    You started it, fag.

Log in to reply