One transaction at a time, damnit
-
So I start working at this company that creates software that deals with different types of financial transactions. In fact, they deal with aggregating a lot of different types of transactions together, figuring out where they are supposed to be sent, and sends them over to a 3rd party to process. The developers basically code interfaces to 3rd party processors of these financial transactions.
Being new, I would sit in a cube and kind of listen to what was going on with the other developers while trying to learn the ropes. This asian guy sits behind me and I notice he is on the phone quite a lot. He is not very fluent in english, and most of the time he is on the phone he appears to be on a conference call and getting interrupted a lot. All I hear is:
"But..."
"Um..."
"No, that's not..."
*loud sigh of frustration*
"Okay, but..."
I finally learn that he has been having three-way calls with one of our customers and representatives from this third party processing company that he is trying to write an interface to. Apparently, his integration is taking far too long and just would not work. Eventually, the guy gets fired.
I guess, being the new guy, I get his project to try to figure out how to deliver it. So I start working on it and discover that the 3rd party processor has an SDK that is used to process transactions through their services. I start reading the SDK documentation and it is a series of simple COM interfaces that are called to do whatever it is you want to do. As I get to the back of the manual, I notice a section about "Registry Keys Used". I start reading it and realize they are dumping their entire message structure into the registry as the SDK COM components make calls to each other. Yes, instead of just having COM marshal their data objects across apartments, they were dumping them to a static set of registry keys. Given that our application was mutlithreaded, this wasn't going to work too well. It worked so UN-well, in fact, that the modem port you wanted to use for one transaction would get overwritten between thread calls and cause all sorts of nasty problems as one thread tried to read from a modem that it didn't even have initialized (since the other thread overwrote the COM_PORT_TO_USE reg key in preparation for processing a second transaction).
This sort of explained why it always worked on his box, but never when they tried to do pre-production load testing. He never tried more than one transaction at a time, would test it, declare it working, then ship it to the customer where it would promptly crash and burn. I'm not sure if his lack of recognizing the problem of using the same registry keys for each thread was going to be a problem, or if the people that actually wrote the damn SDK that were on the phone couldn't explain why it didn't work is the WTF. Maybe both, but it gets better.....
One of the first things I do is run a test transaction to basically kick the tires of the SDK. I usually have my CPU utilization monitor open, and I noticed that while the SDK dialed the modem and processed the transaction, my CPU utilization slammed to 100%. I also noticed that the transaction would fail fairly frequently on my box. I did a quick check of the ex-employee's box to compare it with my own and confirmed the problem.....
Being new, I got a newer computer that was quite a bit faster than the ex-employee's computer. The 100% CPU utilization was likely the SDK developer doing something like:
for (int x = 0; x<1000000000; x++);
while the SDK code waited for the response from the modem. Since my fast computer counted that high faster than the ex-employee's computer, there was not always data ready to be read in the COM port buffer. Thus, an error. This third party processor is a rather large company, how in the HELL did they let this shit get out?
But, of course, it gets better....
So I report to management that this project is impossible to complete because the SDK, to put it bluntly, was written by someone that should not have been allowed to even own a programming book let alone execute a compiler. There was no possible way in hell to serialize all transactions over a single connection for the volume of transactions expected, plus each transaction took 100% of the CPU to process. So the inevitable conference call happens.
I get on the phone and explain to the company reps what exactly is wrong with this SDK. Their response:
"It is working as designed."
I ask for the source code so I can fix it (or reverse engineer their communication protocol and write my own interface).
"We can't give out that source code." (Translation: You caught us and we're too embarassed to give it out. We swear we didn't let the boss' 9th grader code anything else after that.)
If you thought it was over, you thought wrong. It still gets better....
It turns out they have been working on a replacement solution. "It is a web service based solution." How bad can you screw that up?
Fairly badly, apparently. Basically, the process involved the HTTP POSTing of a variety of XML messages to process a transaction. Instead of returning XML in response (since, you know, you don't tend to have people type a bunch of XML into a text box and hit the submit button.....no, you generally have some sort of code that forms the stuff and sends it) it returned plain old HTML where you had to parse the return data out of the body. The body consisted of free-form text. Grrrr...
So, out of spite, I put about 2MB worth of garbage in one of the fields and posted it to their test system. The test system was unreachable the rest of the day.
A few days later, I get an official communication letter from the third party processor. It is something to the effect of, "We are no longer supporting the COM SDK, use the web interfaces."
That project, to this day, is the fastest project I've ever completed. It might have something to do with the project ending before getting to the part where I code anything, but still.....
-
@ooblek said:
This third party processor is a rather large company, how in the HELL did they let this shit get out?
The larger the company, the bigger the shit usually. I feel sorry for that asian guy.
-
@ooblek said:
I vote to move this to the frontpage right the fuck now.So, out of spite, I put about 2MB worth of garbage in one of the fields and posted it to their test system. The test system was unreachable the rest of the day.
-
@SQB said:
@ooblek said:
Seconded
I vote to move this to the frontpage right the fuck now.So, out of spite, I put about 2MB worth of garbage in one of the fields and posted it to their test system. The test system was unreachable the rest of the day.
I burst to laughter several times, but this one hits it really.
-
-
@ooblek said:
someone that should not have been allowed to even own a programming book
Something tells me you probably needn't worry about that.
-
Definitely front page material. Nice!
-
An actual WTF in my Sidebar?
-
@morbiuswilters said:
An actual WTF in my Sidebar?
It's more likely than you think.
To the OP: If I didn't know better I'd swear you worked with my company. One of my first projects when I was starting was to fix a similar 100% CPU problem while waiting on a modem, because the original developer used a "for" loop in that exact way, not to mention never fully killing the thread (so the first time the modem dialed out, the CPU jumped to 100% and stayed that way, and God help you if you needed to dial out again).
-
@ooblek said:
Ah, the famous last words before the system goes down in flames. I wonder if "crashing spectacularly" was part of the design, then."It is working as designed."
Good thing that project didn't go through ... I wouldn't trust my transactions to a third-party like this one!
-
This is what you get when you hire the cheapest programmers you can find.
-
+1 for frontpage; this is hysterical.
-
@merreborn said:
This is what you get when you hire the cheapest programmers you can find.
No, this is what happens when you hire programmers that happen to be the boss' old high school buddy, who "knows a bit about computers".
-
The scariest part about this post is that I believe it completely... have seen many projects that have gone that route. Not quite as bad as that, at least not all crammed into 1 project, but I always get a feeling of dread when I'm sent a third party API to integrate with.
-
Totally frontpage +1 vote etc. n-ded.
-
@dhromed said:
Totally frontpage +1 vote etc. n-ded.
+1 on the +1 etc.
Excellent story, and well written!
-
@PhillS said:
+1 indeed.@dhromed said:
Totally frontpage +1 vote etc. n-ded.
+1 on the +1 etc.
Excellent story, and well written!
Something that uses Windows registry keys to save temporary data, which has to be thread-safe ... man. I just can't count all the WTFs here.
