No REST for the wicked
-
Someone I know (and have collaborated with for years) is trying to use a RESTful webservice. To use it, you send an XML document to it that describes the sort of search you want, and it sends back (as plain text) the response. That's rather clunky, but sometimes that's how it goes.
But that's not the
.The WTF about http://www.rcsb.org/pdb/rest/search (yes, I'm naming and shaming) is that for it to work, you have to send that XML input document with its content-type set to
application/x-www-form-urlencoded, and it sends back its response without a content-type header at all. It just states that the compression algorithm used istext/plaininstead.Double
time.My friend wants my help getting his code working with that site. I'm really tempted to tell him that the only help he'll get from me is with carrying the napalm for the flamethrower.
Anyone else got some good stories about deeply misimplemented services? I need reassuring that not everyone is a complete fuckwit.
-
Anyone else got some good stories about deeply misimplemented services?
GitHub - discourse/discourse: A platform for community discussion. Free, open, simple.
A platform for community discussion. Free, open, simple. - discourse/discourse
-
I thought it was mostly in the UI that things were truly messy with Discourse. Because that's what Jeff sees…
-
No story yet, since still implementing, but the XSD for this is terrible:
-
I thought it was mostly in the UI that things were truly messy with Discourse. Because that's what Jeff sees…
http://what.thedailywtf.com/t/the-unofficial-discourse-api-docs/8908/15?u=maciejasjmj
And possibly numerous others.
Okay, why does the first link onebox and the other two don't? And why does it eat my newlines?
-
We use a SOAP service provided by a service provider we're contractually required to use by our single largest customer. The service provider knows what SOAP is. It's XML over HTTP. The WSDL contract for each method has one argument: A string which is an XML document that contains the message we wish to send. Gone are all the advantages of a formally specified contract, easy marshalling and demarshalling of objects in XML, automatic validation provided by the framework.
I work for a government owned company, the service provider is a large publically traded company.
-
Were they forced to implement it via legislation?
Welcome to passive-aggressiveness.
-
No, they tendered for the contract, and providing this service was part of the contract.
They have demonstrated a shortage of competence at many stages of the implementation, not just of this service, but of other stuff we have to do with them also.
-
In enterpriseland, this makes sense. Changing a 'file format' or 'data layout' is MUCH more poltically viable than changing an 'API'.
Sauce: I deal with this kind of idiocy all day long. I even design this idiocy from time to time (though my preferred mechanism is to avoid ever letting anyone political get wind of what the API is. This often requires not documenting the API outside of conversations and emails, unfortunately)
-
We are the only consumer of the service. Changing the data layout is just as hard as changing the contract in the WSDL, which is to say it needs to be coordinated with us but it's not particularly difficult.
-
Not really services in my latest story, but a communications channel.
One of our clients decided that they wanted to have us send some data to them through a third party company, specific messages to be EDI format, transmitted over a secure method selected by us and the third party. I suggested web services set up on each side to be called as needed, since the communication needed to be two way. Third party said no, they wanted pure FTP. I shot that down as we didn't have anything set up to handle FTP, we didn't have a server we felt comfortable putting FTP on, and we had no space for a new VM.
I finally put my foot down and said we were going to use AS2, it's secure, it's standardized, there's plenty of tools to handle the communications, and the spec requires a checksum verification of the received message. The client backs me on this plan, so third party just has to suck it up.
But then they can't implement it. They refuse to provide the receipts, so we have no way of knowing if they actually received our messages. ntoo not to mention the fact that it took them two weeks just to get their side working.
Then they couldn't build or process messages correctly. After a month of waiting on them to get things working right, the client cans the project, says they'll be starting over.
Two months later, client comes back and revives the project, with the same third party. We have until next Friday to get it working in production.
-
The worst one I've seen is still this: http://what.thedailywtf.com/t/incompetent-contractors-idea-of-an-api/47870
My boss also loves him some using FTP as method of communication. Eg, service A dumps stuff from its database into the FTP "inbox" of service B, which then sends a POST request to services C, D and E, one of which then dumps a file to FTP F...
Good luck debugging this stuff.
-
What about the XML format for reporting tax by our government, that use UTF-8 but insist every non-Lower ASCII range of characters (I can't remember how exactly they spelt the rule) in text field should be counted as 2 characters?
-
I Once had to integrate with an API that was implemented in the same way.
Fortunately, but only marginally better, they provided another web method that took the data as around 20 Separate parameters instead. Using this seemed like the less of 2 evils, just.
As a bonus WTF, the first time they deployed the service it was on their public website domain, but in an area hidden by forms authentication, so any request returned the markup of their login page.
Amateurs.
-
Eg, service A dumps stuff from its database into the FTP "inbox" of service B, which then sends a POST request to services C, D and E, one of which then dumps a file to FTP F...
Does it tell you about errors by FTPing a document saying that something went wrong? (But not what it was. Oh no.) I've seen that before.
-
Does it tell you about errors by FTPing a document saying that something went wrong? (But not what it was. Oh no.) I've seen that before.
It gets copied to "failed" folder. Then you dig through logs (if you have them) and try to figure out what went wrong. Or just copy the file back to "inbox" and see what happens, who cares.
The best part is that one of these abominations, that was supposed to be a super simple uploader, over time sprouted an API, and then, on top of it, a custom made SPA-like site (no router, so you better not refresh the page!). And everything gets loaded and dumped into a single 10 MB json file, that serves as its "database". Ugh.
-
Sounds like they weaponised their incompetence well.
-
Exactly. But all the bullshit approval committees of nontechnical people view an API change as the end of the universe, needing to be carefully documented and reasoned in nauseating detail only to reject it out of hand for 'safety' reasons, whereas a 'data change' is likely negotiable over the phone between producer and consumer and documented internally to the teams.
-
So, they've implemented a strategy that guarantees that all changes will be done in the least safe manner possible... in the name of "safety". Good job.
-
Enterprise!
-
Sadly, all sorts of people still FTP.
You guys that are stuck there are lucky. Pity the poor schlubs who are stuck working with companies that want to do data exchange over email.
-
My boss also loves him some using FTP as method of communication
I once wrote a system that used email for IPC. The client commissioning the software had very restrictive firewall policies between the different segments of their LAN and kept saying no to methods like TCP WCF. Then they suggested that they had an Exchange server and IMAP was allowed internally so I should just use that. No amount of objecting changed their minds once I was forced to admit it would probably work.
So, all the nodes communicated via a single mailbox on an Exchange server. The email subjects became a source/destination/type/TTL header so each client could poll the mailbox to look for messages that concerned them without having to get the full body of the emails.
The worst thing was it worked really well, one deployment even had hundreds of nodes. I just felt so dirty...
data exchange over email
Hah, didn't see that before posting. The second worst bit was people asking why my software required a mailbox when they were doing deployments, their look after explaining was always
-
The worst thing was it worked really well, one deployment even had hundreds of nodes. I just felt so dirty...
I'm disgusted and impressed. Dispressed?
What did the system do? (as in type of work)
-
It mainly controlled a hardware device attached to the workstations and some deep behaviours of the OS itself as well. The interface to the user was WPF and I was genuinely proud of that, it looked great and was my first proper attempt at doing UX.
The IPC was for stats tracking and settings updates mainly. Some nodes also had groupings and should do some behaviours more-or-less in sync with their neighbours.
-
-
The worst thing was it worked really well, one deployment even had hundreds of nodes. I just felt so dirty...
Did you know that communication via email is one of the documented and profiled ways of doing SOAP? That there's been in the past lots of people doing this, enough that there are standards specifically dealing with it? (I think there's a similar spec also for sending it over XMPP, giving what's little more than SOAP over IRC, and that that's one of the better ways of doing it…)
At least you were actually using the low-level protocols right. It's when they don't bother with that that I get really upset.
-
Sounds like they weaponised their incompetence well.
To make it worse, based on past experience with the customer, if there are any problems with this communication, blame will fall first on us. The burden will be on us to prove it was the third party's fault.
-
any request returned the markup of their login page.
This is a hanging offence. Returning HTML for a web service request is bad. I bet it was served with a 200 response code too.
-
data exchange over email
Did I not mention that we still do that for some processes? Some parts of the world move pretty slowly.
-
Returning HTML for a web service request is bad.
Unless it was specifically requested (e.g., by content-type negotiation). But yes. And sometimes that's all you've got.
I bet it was served with a 200 response code too.
Of course. You got a webpage back, what more do you want?
-
Accept: application/json
Content-Type: text/plain
['BAD CSRF']
-
sending it over XMPP, giving what's little more than SOAP over IRC
XMPP is almost reasonable for the purpose. It is simply a generic n-way message and pub/sub protocol sending around XML payloads.
It has one serious issue though; it does not have¹ acknowledge, so you have to implement your own ack/nack protocol on top if you need any reliability. Given the main advantage of having a broker server is communicating with devices with intermittent connection, it's really pity.
-
i hate that response from discourse......
what causes it to do that?
-
i hate that response from discourse
Not as much as I do at the moment; I'm getting it on @riking's forum too, not just here.
@accalia said:what causes it to do that?
Best guess, it's somehow failing to match the CSRF in the HTTP headers to the one served bycsrf.json; whether that's because it doesn't save the value, or whether it does but saves it immediately invalidated… whatever the reason, I know adding a short delay betweenGETtingcsrf.jsonandPOSTing the login attempt doesn't help.
-
-
Incompetent developers.
That's a cause of so many things. When it isn't incompetent management, incompetent salesmen, etc…
-
@accalia said:
what causes it to do that?
Incompetent developers.
In this instance, those who work on the Node.jsrequestmodule
-
This post is deleted!
-
Yeah -- a few folks use ISO-message-based formats instead of pure flat files, which are somewhat entertaining to deal with offline (hello, IPM :D ) and there's one major one that sends and wants EBCDIC (BASE II,
)I was shocked when I entered the industry -- you'd think our payment systems would at least be somewhat modern... but they are not. Many credit card platforms were built in the 60s-70s in COBOL and are still used today.
Yeah, there are but a few intrepid souls working on a notion of modern credit-card-processing systems...Filed under I interned for some of them...
-
Because that's what Jeff sees…
I imagine Jeff to be like the Pyro from Team Fortress 2. He probably sees everything he creates as beautiful rainbows.
-
and there's one major one that sends and wants EBCDIC (BASE II,
)
EBCDIC actually comes from IBM Mainframes
-
... which were used to run the BASE I and BASE II networks VISA used to reconcile payments. They probably no longer use mainframes, but
compatibility .
-
They probably still use IBM Mini, because compatibility
-
I thought it would be interesting to note that MOST financial data that comes out of credit card core systems is handled in flat files via FTP. When your bank orders you a debit card, for example, that data is FTP'd to the card vendor in fixed-length record in a flat file. This isn't an anomaly -- it is the standard mode of operation.
But, that's not where it ends. INTERNALLY, within the card production system, data routinely follows the "Just throw it in X directory" flow. Need to prepare data for the magnetic stripe -- throw the file in x directory and the MagStrpSrvc will prep the data and send it back.
I was shocked when I entered the industry -- you'd think our payment systems would at least be somewhat modern... but they are not. Many credit card platforms were built in the 60s-70s in COBOL and are still used today.
I actually don't find that too surprising. They built those systems back when such technology was cutting edge. They can't just upgrade wholesale because it would bring most of the world's transactions to a standstill.
The card companies could do a gradual update, running both systems in tandem until the old system has been fully replaced. But we're talking a massive overhaul of a system that currently works. Businesses – especially big businesses – aren't in the habit of spending large sums of money to replace something that works.
Would it save them money in the long run? Probably. Are they running the risk that they will have an unsupportable system in the not to distant future? Almost definitely. Would fixing that cut into short term profits? For certain. What do you think they care about more: short term profits, or long term sustainability?
-
Sadly, all sorts of people still FTP.
You guys that are stuck there are lucky. Pity the poor schlubs who are stuck working with companies that want to do data exchange over email.
E-mail might not be pretty but FTP is just the worst of the worst.E-mail handles binary transfer decently, can use TLS more easily, and it declares file types.
Oh, and did I mention that you don't need to poke holes in the firewall in both ways? The whole "one connection one way, another in the other direction" is just mental.
-
What do you think they care about more: short term profits, or long term sustainability?
Whatever most pleases the shareholders.
-
Whatever most pleases the shareholders.
Unfortunately, many shareholders are idiots who only look at profits. Not sure what percentage of total shareholders they constitute, though.
-
Profits are becoming irrelevant these days too; paying dividends is so passe. Market speculation is becoming the be-all and end-all of shareholding...
-
To make it worse, based on past experience with the customer, if there are any problems with this communication, blame will fall first on us. The burden will be on us to prove it was the third party's fault.
I assume they're bigger than you and your company needs their money.
Sounds as though you have crossed over into the world of savagery.
-
@blakeyrat said:
Incompetent developers.
That's a cause of so many things. When it isn't incompetent management, incompetent salesmen, etc…
Discourse has all of these - most of them are the same person.
