Smart Unit Tests



  • So, today a coworker linked me [url=http://channel9.msdn.com/Events/Visual-Studio/Connect-event-2014/215]a video[/url] about a new feature in VS2015.

    Halfway through the video, my mind went where most of the comments went: "Wait, hold on, these tests will just exercise all the code and say what it does, not identify problems!"

    But, like the few sane comments, I can see the positives. Especially when dealing with a massive legacy codebase. If nothing else, you become able to safely refactor. Overall, it's pretty cool. You just need to know when to use it.


  • I survived the hour long Uno hand

    That actually looks really cool for a starting place to retrofit old code. I'm tempted to write some version of that for javascript files :D



  • Yeah. I figure they're using Roslyn for this, and Roslyn is only at the very beginning of it's lifespan. I'm interested in seeing what else they do.



  • You might look at https://github.com/SRA-SiliconValley/jalangi#concolic-testing; I haven't tried it, and it's probably research-prototype quality, but if it does what it says on the tin then it might be what you are looking for. The relevant term (well, maybe the most common term for this) is "concolic execuiton."

    (Actually there are a few different ways people have used to automatically generate test cases like this; my impression is that concolic is the most successful, and I'd give good odds that serves as the basis for smart unit tests, but it's not the only possibility.)



  • @Magus said:

    Especially when dealing with a massive legacy codebase.

    Sounds like what my company needs. We have a massive codebase that would have looked stale in 2005.


  • I survived the hour long Uno hand

    So instead I spent the afternoon writing a node app that uses esprima to parse a file and creates the top half of a decision tree to indicate how many cases this file will need per function.



  • What I see it does is guess test cases to get 100% branch coverage. So you don't have to think hard about how to set up the tests and which cases you need to get the coverage. That definitely saves some work especially when you need to add tests to existing code where you don't know or remember what the corner cases may be.

    What I miss in the demo is the step where you decide whether the results are actually correct. Because the tool can't do that. The tool doesn't know.

    What I saw as wrong in the demo (and is related to the missing bit) is how he “fixed” the null case. If a function gets null in a required parameter, I usually expect it to throw upNullPointerException¹, so the correct “fix” should have been to tell the tests to consider that as success in that case.

    And of course then there are all the cases where two inputs don't fall in different branches but one still produces incorrect result and the other does not. Perhaps because to produce correct result they should have fallen to different branches and one of those branches is missing. Automatic analysis can't discover these cases and neither can checking coverage.

    ¹ Edit: It would be formally more correct to throw ArgumentNullException, but for exceptions that just detect internal coding bug I don't consider that distinction worth adding an explicit check in each and every method when the runtime is going to check anyway.



  • @Magus said:

    Especially when dealing with a massive legacy codebase. If nothing else, you become able to safely refactor.

    Yeah, I think that's the point. You know that the code works now, and want to ensure you don't accidentally screw something up.

    You can't really automate writing unit tests on your new code anyway - the compiler can't read your mind to find out what the intent was. If it tried, you'd probably be cursing it all over the place for not taking special cases into account.


  • Discourse touched me in a no-no place

    @Maciejasjmj said:

    the compiler can't read your mind to find out what the intent was

    It reads your mind by reading your code. 😦



  • @dkf said:

    It reads your mind by reading your code.

    You'd need some fairly advanced AI for that. And you do not put AI near your code, or it's gonna be Skynet all over again.


  • Discourse touched me in a no-no place

    @Maciejasjmj said:

    And you do not put AI near your code, or it's gonna be Skynet all over again.

    Either that or it will be SAP all over again…


Log in to reply