Mocking-Up A 3rd Party API



  • Ok so.

    We have a new push at our company to increase test coverage. The product I'm currently working on already has basically 100% coverage for all of the tasks it actually does on its own, but the problem is the primary purpose of the product is to push our data to a 3rd party API we do not in any way control in the slightest. So even if all the current unit tests pass with flying colors, it's no guarantee the tool actually does what it's supposed to do.

    My boss argued that we need to mock-up the 3rd party API so we can properly write unit tests for the large portion of the product that makes use of it. I told him that it's impossible to unit test a black-box 3rd party API, writing the mock-up would take a significant portion of the development time, and instead of doing that, we should write integration tests for it. He argued back that we don't currently have a framework that can run those API tests in an automated way* and until we do the mock-up API is the best we can do. I told him that so far 95% of our bugs have been due to them changing their API without notifying us and the mock-up API is utterly useless for discovering those, so creating it feels like pointless busywork and not software development.

    As a compromise, he said that if I can come up with a better way to test the functionality of the tool without creating the mock-up API, he'd consider it. But frankly I have no ideas.

    So in short: is it possible to write useful unit test based on a mock-up of a 3rd party's API? Or is it literally just busy-work that makes our test coverage numbers look better while providing no actual benefit to the project?

    I'd especially appreciate @Yamikuronue 's opinion on this one. Or anybody else who spends a lot of their time in testing.


    * Which is true-- they are a huge PITA to set up, requiring the installation of certificates and such. We currently have a set of integration tests, but they require being manually-run from a machine that's been specifically set up to run them with the correct SSL cert installed.



  • @blakeyrat said in Mocking-Up A 3rd Party API:

    So in short: is it possible to write useful unit test based on a mock-up of a 3rd party's API? Or is it literally just busy-work that makes our test coverage numbers look better while providing no actual benefit to the project?

    I cannot speak to how valuable you will find the advantage I can think of, but this is it: You get the guarantee that your part of the product is correct. That's pretty much it. You may also have an easier way to find exactly where a breakage occurs, but not terribly much.


  • ♿ (Parody)

    @blakeyrat said in Mocking-Up A 3rd Party API:

    So in short: is it possible to write useful unit test based on a mock-up of a 3rd party's API?

    Obviously it's only as good as the mock-up. But if you have good coverage of all the ways you use it, you'd be able to see the effects of their changes once you learn of them and update your mock-up.

    Depending on how frequently stuff changes, probably very front loaded but with increasing benefits later on. That said, I'd probably still prefer integration tests, if you can get them automated.


  • Grade A Premium Asshole

    @blakeyrat I have nothing to add, but I will be following this one because it is a really good question about something I never considered.


  • Garbage Person

    WtfFramework has a testing mock for a black box system. The mockup is arguably a better implementation than the real thing.



  • @Weng said in Mocking-Up A 3rd Party API:

    The mockup is arguably a better implementation than the real thing.

    But that's a huge problem, right?

    Like I said, the vast majority of breakages of the older version of this tool were due to the 3rd party's changes. Since my mock couldn't possibly track their changes, that's why this feels like a complete waste of time to implement. (If mocking it were easy, that'd be one thing-- just do it, it's harmless. But the mock is likely to be 25% or so of the total project code.)


  • Garbage Person

    @blakeyrat Yeah. It wasn't done for engineering reasons. It was done for politics, pure and simple. Basically, because we have a "known good" reference point, we can hold the other party responsible for unexpected changes.



  • @Weng Gotcha. Personally, I see that bit of the problem as "handled", because we do have the integration tests that can be manually-run.


  • Trolleybus Mechanic

    What is it that they want to test?

    That data from MockApp into your product works?

    That data leaving your API into MockApp works?


  • I survived the hour long Uno hand

    @blakeyrat said in Mocking-Up A 3rd Party API:

    the mock is likely to be 25% or so of the total project code.)

    This seems to be the problem. I'm not sure technically how the integration is working (I assume you're writing in .NET? And this is a server-to-server call, rather than an AJAX call or an EDI or something?); in my line of work, I'd probably grab Sinon.js and have it run a mock server to communicate with.

    What you can get out of doing the mocking is having documentation of what you're expecting, so you can verify when they change their API what you used to have and exactly what changed; furthermore, you can then update the mock and see every place the code breaks, rather than just whatever one alerted you to teh change. I'm not sure if that's useful or not; it heavily depends on how much work it is to mock out and how much benefit you'd get out of it, but that's the big one I'm seeing.

    I was at this talk while you were posting this thread: https://speakerdeck.com/jimholmes/automated-testing-beyond-the-basics

    (Jim Holmes is a great guy for testing expertise, by the way, but his slides are pretty rough without the accompanying video, and I don't think this talk was recorded)

    One of the main goals for this kind of test is increasing trust. If it would help your boss feel more confident in the application, maybe it's worth doing? But if it's just busywork that nobody will ever really care about, maybe it's not? We talked about not testing stupid stuff, like not bothering to open a gmail client to try and verify that you received email sent from an application; this might well fall under the vein of "stupid stuff". He argued in favor of using manual testers for those last 10% of things that are a PITA but which are important, like verifying that the UI looks correct, emails are sent, et cetera. This might be a case where you use manual tests to verify that the full integration works as expected, from your special SSL machine.

    I'm pretty on the fence on this issue, it's a call that depends a lot on stuff only you can really answer.



  • @Yamikuronue said in Mocking-Up A 3rd Party API:

    This seems to be the problem.

    There's a back and forth involved. I could just hard-code responses I suppose, but that strikes me as even more pointless.

    @Yamikuronue said in Mocking-Up A 3rd Party API:

    I'm not sure technically how the integration is working (I assume you're writing in .NET? And this is a server-to-server call, rather than an AJAX call or an EDI or something?);

    WCF, generated from a WSDL.

    @Yamikuronue said in Mocking-Up A 3rd Party API:

    And this is a server-to-server call, rather than an AJAX call or an EDI or something?);

    It's a "whatever WCF does" call. Probably either REST or SOAP with XML content, but that's all abstracted away from me.

    @Yamikuronue said in Mocking-Up A 3rd Party API:

    in my line of work, I'd probably grab Sinon.js and have it run a mock server to communicate with.

    That doesn't really change anything, does it? Either way, you're mocking-up a 3rd party's black box based on nothing but reverse-engineering.

    @Yamikuronue said in Mocking-Up A 3rd Party API:

    I'm not sure if that's useful or not; it heavily depends on how much work it is to mock out and how much benefit you'd get out of it, but that's the big one I'm seeing.

    It sounds like you're just confirming that the mocked API is completely useless for determining whether bugs exist in the program.

    @Yamikuronue said in Mocking-Up A 3rd Party API:

    He argued in favor of using manual testers for those last 10% of things that are a PITA but which are important, like verifying that the UI looks correct, emails are sent, et cetera.

    HOLY SHIT I DISAGREE.

    If your application has a UI at all, it's going to be way more than 10% of the testing burden, and it's all manual work. How the UI behaves is like 50% of your app, and it's like 100% of the important to test part of your app. Maybe a lot of people following this guy's advice is why UIs are so shitty in the industry right now, haha.


  • I survived the hour long Uno hand

    @blakeyrat said in Mocking-Up A 3rd Party API:

    If your application has a UI at all, it's going to be way more than 10% of the testing burden, and it's all manual work.

    I meant 10% of the codebase that makes up the UI. It's crucial to have actual eyes on that. People keep trying to invent automated GUI tests and they're all balls because you need actual people to make decisions based on how things are perceived by actual human brains.



  • @Yamikuronue I think anybody sane realizes that automated UI testing is shit. Even functional testing, and that's the least important part of it.

    But anyway, off-topic. I don't know what to tell my boss.


  • I survived the hour long Uno hand

    @blakeyrat What does he want to get out of it? I mean, high coverage is nice, but is that all? Or is there something specific he's worried about?



  • @Yamikuronue said in Mocking-Up A 3rd Party API:

    @blakeyrat What does he want to get out of it? I mean, high coverage is nice, but is that all? Or is there something specific he's worried about?

    Honestly we had a bad release this week (because our QA department is horribly under-staffed right now) and management is now all about "we need to increase our automated test coverage" even though, IMO, that has absolutely nothing to do with the problem. The problem is the QA department hasn't made any hires in like 6 months, and due to the way our project is deployed we end up doing 6-8 full regressions a year. Plus one of those testers is working more-or-less full time on billing issues.

    At our last retrospective, one of the front end developers pointed out that for all practical purposes, we have about 5 developers for each tester, that those testers are always busy doing full regressions anyway, and that by our own agile definition of "done" we shouldn't have shipped any of the new features we did in the last 3 months.

    Anyway, long story short, our QA manager sucks. She's ok as a person, but her department is in dire straits and has been for ages.

    But this also all means maybe it's a problem I can sit on for a couple weeks and it'll go away. I won't have all the non-API bits of the app done for at least another week anyway, and those are all easily tested.


  • Discourse touched me in a no-no place

    @blakeyrat said in Mocking-Up A 3rd Party API:

    How the UI behaves is like 50% of your app, and it's like 100% of the important to test part of your app.

    I respectfully disagree. It's important to some parts of the app (and holy balls it needs much effort to test thoroughly) but many apps have priorities other than being nice to users. Compliance is a big one, very often, and most users don't pay it any attention at all despite it being something that can totally kill the company. Test? Yes. Test UIs? Yes! Assume that they're the main thing that needs testing? No. :(


  • area_pol

    @blakeyrat
    If the API changes and you need to respond quickly, you will probably not have time to update the mock-up before fixing the main program, so you will update the mock-up later to match the existing fix in the main program.

    Do you have access to the external API for testing purposes before release?


  • I survived the hour long Uno hand

    @blakeyrat Yeah, it sounds like he's after extra confidence. Which is one of the main goals of testing, but also, your testers should be re-aligning their activities to provide it, and they don't seem to be able to. I'm not sure a good developer can fully compensate for a bad QA department :/



  • @Adynathos said in Mocking-Up A 3rd Party API:

    Do you have access to the external API for testing purposes before release?

    Yes, they've given us 3 environments to use. So we've slated one for dev, one QA and one Prod. Still require the certificate annoyance in the dev one, though. If they didn't, I'd just write "unit tests" that exercise the API and let our Bamboo build servers chug through them.

    One possibility my boss brought up is somehow temporarily installing the cert in code, which I know you can do if you have access to the low-level web shit in .net but I'm not sure you can do with a WCF service.



  • @Yamikuronue Now that I think about it, part of the bigger problem is since our company moved its offices to bumblefuck we've made about 2 hires and we have like 10-12 open reqs. (Which is a ton considering the entire company is maybe 60 people right now.) It doesn't help that both our recruiters left in the last 3 months, heh.

    That office move was a fucking terrible idea.

    Anyway, our testers are great, there's just not enough of them.


  • I survived the hour long Uno hand

    @blakeyrat said in Mocking-Up A 3rd Party API:

    Still require the certificate annoyance in the dev one, though.

    I wrote some Java tests that present the appropriate certificate before they run, it's possible to do in an automated fashion, just kinda sucks. Or, depending on your build system, you could set up an agent and make the tests run there? In Bamboo, I can designate a "Capability" called "Has SSL cert" and assign it to only those agents with the cert installed, so it'll only run build jobs that require that capability on those agents.


  • area_pol

    @blakeyrat said in Mocking-Up A 3rd Party API:

    Still require the certificate annoyance in the dev one, though. If they didn't, I'd just write "unit tests" that exercise the API and let our Bamboo build servers chug through them.

    If you have a test instance of the real API, testing against it seems infinitely more useful than against the mock-up.
    If your boss wants to spend time of improving testing, maybe use it to try automate the certificate installation and testing against real API.



  • @Adynathos Yah, I think I'm just going to sit on it for a couple weeks and wait until there's something else the office is panicking over. If that doesn't work, I'll put some effort into seeing if you can install a cert in WCF before it does its shit.



  • Is this the same service you generated all those wrappers for? Seems like 999% of the work would already be done then. If I remember right, each method on your wrapper object would do some logging, and then call the corresponding method on the actual service reference and return the result. Mocking the service should be the same, but instead of calling the actual service, just return whatever mock data is appropriate for the current test.

    You don't need to model the 3rd party service's complex internal behaviour for unit tests, and you don't need to test the WCF-generated code (do you?), you just need to be able to set your tests up so that a) you can ensure that the unit under test called the appropriate methods b) you return appropriate data from the method call to test the unit's behavior.



  • @blakeyrat said in Mocking-Up A 3rd Party API:

    @Adynathos Yah, I think I'm just going to sit on it for a couple weeks and wait until there's something else the office is panicking over. If that doesn't work, I'll put some effort into seeing if you can install a cert in WCF before it does its shit.

    Load an X509Certificate2 from wherever you store your cert, make a new ChannelFactory<TChannel>, then set factory.Credentials.ClientCertificate.Certificate to that cert before using the ChannelFactory to instantiate your Channel.


Log in to reply