The WSO₂ Has Come To Pass.
I don't need to know shit about WSO2 to do my current job and I (crosses self) never will.
I just had to go open my mouth, didn't I?
Ok, so backstory part A: I almost didn't apply for my current job, because the only information on the job listing was that they wanted a Java developer, preferably one with WSO2 experience. I don't have any, so I figured I was headed straight for the ‘no’ pile. Turns out they just didn't know what they wanted; they've had some kind of a legal thing that has cooled their enthusiasm for the main product, and they've got all kinds of plans to make the next big thing. That's what I was hired on for. Luckily, they got me to take the test at betterprogrammer.com before I even got an interview, and I had some code on github they could look at, so we didn't need to waste a bunch of time at the interview determining if I even know how to program, and I know all of my coworkers are competent, at the very least (all the recent hires, anyway).
Backstory part B: We're a small company, but we've got some big clients. Several years ago, someone came in with big plans about setting up an Enterprise Service Bus, the better to service our Enterprise clients with. WSO2 was the product they chose (don't google it; they will take over every banner, sidebar, youtube overlay you see for months, until the phrase “IBM too expensive? Try WSO2!” is burned into your every retina) to do it with. This guy got about 80% of the way through a couple of projects before calling it quits. His replacement managed to get the job done, but deeply regretted it: the time it took to get that last bit of the way is longer than the entire thing should have taken. The couple of weeks between me arriving and him leaving was not really enough time to teach me the ins and outs of the whole byzantine apparatus.
For instance, here's how we get some data out of the database and into a csv: A scheduled task on the ESB server calls a proxy service that triggers an endpoint (also on the esb server) that calls an endpoint on the DSS (Data Something Something) server that runs a sql script (specified inline in the DSS server configuration, naturally) and transforms it into xml. On receiving that xml, the esb server passes it to an Application Server for some kind of processing (passing one column of data through a java library: of course we need a soap endpoint for that), receives xml back from the AS, then finally transforms it into csv using xslt and places the result in an ftp folder for the client to pick up and import into their own system.
So, we're running three whole servers for this, and in the end there's only this one process, returning about 30 rows from the database each morning, and a couple other less important ones on there. But the whole thing is so complex we're terrified to dismantle it, and do you think we could ever find the original project requirements anyway? The way it failed this week was entirely predictable: The initial call from the esb to the dss to get the thirty rows of db data typically takes about a minute. My predecessor explained to me how much of a thorn in his side the fact that the esb seems to have a hardcoded limit of 120 seconds before it will time out, and it disables the endpoint on the esb that points to the dss when it does. He adjusted every setting he could find, and there were many, that seemed even remotely related to timeout limit, and nothing worked. So when an ‘unrelated’ fuckup in some other area dumped a bunch of extra rows in the db, causing the query to take longer than that, everything went to the dogs. I just needed to get a small amount of known data through wso2 for processing, but I couldn't, because the query for that day always failed to complete in time.
I ended up doing a SELECT INTO temporary on production and changing the query to do SELECT * FROM temporary, just to get missing data; took all day just to figure out a way that I could do it, and I'm not even sure if it's gonna work over the weekend—I think I put everything back the way it was, and that the root cause of the problem won't affect future queries, but only god knows that for sure. I don't even understand how half of the things that are happening even happen (I'm wondering if we actually have a whole extra esb server somewhere that nobody told me about), and it seems like the process is creating the csv files twice every time it gets called, which hasn't caused a problem yet (I understand our client's process can't handle duplicate data: it would assume each item got created twice), because they both have the same filename, but if the files ever get created in different seconds...
I was always going to propose some kind of game where the idea was to transfer some data collection through as many things as possible. Maybe from a database to a web server, over http to another server that converts to XML then hands it off to a process that use Perl to convert it to a flat file, which is then read by PHP...blah blah blah you get the idea.
But you already have that, which takes all the fun out of it.
Data Scrying Satellite
Now I understand why some people mentally translate SOA to Rube Goldberg Machine.