Code Coverage



  • At the Friday morning WTF-is-gong-on-around-here weekly meeting, it was announced that MegaCorp has decreed that the new metric of the day is Code Coverage. Naturally, the question is asked: what percentage of your application is covered in JUnit tests?

    I answer honestly with about 3-4%, coupled with the explanation that tests are being added for each new feature as part of its development, and that I'm slowly working on the backlog of old, untested features as time permits. They sort of expected this, and said to keep at it.

    A peer team proudly announces that they have 90+% coverage. Even these managers know better than to buy into that, so they shoot me a look.

    After the meeting, I dive into the peer project code and run a Code Coverage tool on it. Sure enough, when run through the tests, it appears that the coverage is above 90%. Then I run the application through a profiler, and only about 50% of the code is executing, even though I'm running a dataset that should give a pretty wide array of customer cases. Now I gotta figure out if our data set is AFU or figure out how a nearly full-coverage data set is only executing half of this application's code.

    Sure enough, I notice that there are several whole subtrees that don't even show up in the profiler. Hmmm.

    Investigation shows that these are unreachable. Completely unreachable. As in they could not possibly execute. Ever. And source control shows that it's been this way for more than a year. Yet most of the JUnit tests in their source tree are explcitily for classes in this unreachable code, and have been frequently modified within the past year. It turns out that if you subtract off the JUnit tests for the unreachable code, their code coverage drops to slightly more than 2%. That's right, they've been spending most of their time writing and updating JUnit tests for unreachable code.

    I report up the tree.

    Wait a second, are you telling me that these folks haven't been able to deliver new features because they've been busy writing unit tests for code that can not possibly ever execute?

    Yep!

    I deleted more than 200K LOC and countless JUnit tests from their source tree, rebuilt their application and ran the relevant suite of tests; everything worked. I ran MY full-swath test case and everything worked.

    It's Friday!

     



  • Maybe I made an oversight, but how can code coverage drop from 90% to 2% when 50% of the code is deleted? shouldn't at least 38% of the coverage be in reachable code?



  • @dtech said:

    Maybe I made an oversight, but how can code coverage drop from 90% to 2% when 50% of the code is deleted? shouldn't at least 38% of the coverage be in reachable code?
    You didn't. I'm just having a really spastic day. I was going for 40+% but just mangled it (you have no idea how many times I cycled between compose and preview).



  • snoofle will you have my babies?



  • @Sutherlands said:

    snoofle will you have my babies?
     

    What makes you think he time to spare for babysitting duties?



  • @snoofle said:

    Wait a second, are you telling me that these folks haven't been able to deliver new features because they've been busy writing unit tests for code that can not possibly ever execute?
    So who catches the blame for this? The team manager? The team lead? No-one at all?



  • @OzPeter said:

    who catches the blame for this?
    The team manager was directing folks to build more and maintain the existing JUnit tests per the recommendation of the team lead. The manager tried to deflect the blame straight back at the team lead. The senior managers, clueless though they may be on some things, picked right up on this and threw it back in the manager's face as he was in charge and is supposed to know what's going on. Both were reamed, and instructed to stop wasting time (now that the code is gone, it should be somewhat easier) and get on the to-do list.

    The manager later thanked me for pointing out their screwup. I pointed out that I was ASKED to do it by the senior managers because nobody believed the BS that HE was spewing forth at the meeting.

     



  • @Cassidy said:

    @Sutherlands said:

    snoofle will you have my babies?
     

    What makes you think he time to spare for babysitting duties?

    I don't think Sutherlands was asking about babysitting, but so long as you matched the salary he is currently paid for babysitting these idiots I'm sure he could find time.



  • @snoofle said:

    The manager later thanked me for pointing out their screwup. I pointed out that I was ASKED to do it by the senior managers because nobody believed the BS that HE was spewing forth at the meeting.
     

    "thanks for highlighting just how useless I am and confirming to senior management that I really am full of shit and can't manage my own te-- WAITAMINNIT!"



  • @Cassidy said:

    @snoofle said:
    The manager later thanked me

    Facetiously....



  • @snoofle said:

    Facetiously....
     

    Yeah, I gathered... "thanked".

    As an aside... any reason to believe your new task could be the new overlords trying to identify deadwood for a potential pruning operation?


  • BINNED

    That would be like identifying water in the middle of the Atlantic.



  •  Here's a way to improve your coverage figures:

    1. Pull in a huge collection of libraries, together with their complete unit tests. Make sure the libraries are 10 times the size of your code.

    2. Link your code to them (this step is optional).

    3. Try and conceal it from busybodies like Snoofle.

    Voila! Instant 90%+ coverage of your code base! 



  • @PedanticCurmudgeon said:

    That would be like detecting paedophiles employed by the BBC.
     

    Sorry, I think I'm presently Britishing at "like 7".

    I'll go read some youtube comments to reintegrate.



  • @snoofle said:

    Friday morning WTF-is-gong-on-around-here weekly meeting
    New idiomatic expression unlocked!

    @snoofle said:

    I deleted more than 200K LOC and countless JUnit tests from their source tree, rebuilt their application and ran the relevant suite of tests; everything worked.
    It's been said before: you should be paid by line of code removed.

    @snoofle said:

    I ran MY full-swath test case and everything worked.
    So all the bugs are either uncovered by the tests (less likely, since they're your tests) or cancelling each other in some way?



  • @Zecc said:

    It's been said before: you should be paid by line of code removed.
     

    "I seem to have truncated our CVS codebase. My bill for ${millions} will be in the post later."

    Job done.



  • @Cassidy said:

    @Zecc said:

    It's been said before: you should be paid by line of code removed.
     

    "I seem to have truncated our CVS codebase. My bill for ${millions} will be in the post later."

    Job done.

     

    Dunno where this quote came from, but here it is:

     "if Java had TRUE garbage collection, most programs would delete themselves on execution."



  •  Protip: You can use Reflection to improve code coverage numbers, by simply calling every function in the binary with random parameters while ignoring results and eating exceptions.


  • Discourse touched me in a no-no place

    @bgodot said:

     Protip: You can use Reflection to improve code coverage numbers, by simply calling every function in the binary with random parameters while ignoring results and eating exceptions.

    That's actually a thing! Fuzz testing.



  • When I was doing testing for Microsoft Games they had a whole server rack full of xboxes doing fuzz testing with random controller inputs. It was pretty epic.



  • @blakeyrat said:

    When I was doing testing for Microsoft Games they had a whole server rack full of xboxes doing fuzz testing with random controller inputs. It was pretty epic.

    I agree, properly done, Fuzz testing can reveal issues that are almost certain to escape "planned"/"deterministic" testing.



  • @blakeyrat said:

    When I was doing testing for Microsoft Games they had a whole server rack full of xboxes doing fuzz testing with random controller inputs. It was pretty epic.

    Bonus points if you had actual police officers pushing the buttons and joysticks at random.

     



  • @Zecc said:

    Bonus points if you had actual police officers pushing the buttons and joysticks at random.

    No, that would be copper testing.



  • @Qwerty said:

    @Zecc said:
    Bonus points if you had actual police officers pushing the buttons and joysticks at random.

    No, that would be copper testing.

    Unless they're battery-powered, then it's copper topping.



  • @blakeyrat said:

    @Qwerty said:
    @Zecc said:
    Bonus points if you had actual police officers pushing the buttons and joysticks at random.

    No, that would be copper testing.

    Unless they're battery-powered, then it's copper topping.

     

     

    fuzz testing on my group involves randomly generated unicode strings (numeric generated, not generated on code page).  not my tests.. but they can blow things up wonderfully if you slightly mishandle a string.



  • @blakeyrat said:

    @Qwerty said:
    @Zecc said:
    Bonus points if you had actual police officers pushing the buttons and joysticks at random.

    No, that would be copper testing.

    Unless they're battery-powered, then it's copper topping.

    The police prefer to avoid battery, though.


  • BINNED

    That would depend on who's being battered.



  • Fish jokes are in the OTHER thread, thankUvewwyMuch.

     



  • You can also batter things like bacon before deep-frying, so it still works (kinda).



  • @locallunatic said:

    You can also batter things like bacon before deep-frying, so it still works (kinda).

    Only in America!



  •  I've been wondering, is there any food that is baters, deep friend, then battered again, and deep fried again?

    I was thinking of doing that to bacon strips, lightly brushing them with maple syrup, then rolling them in bacon bits.



  • @bgodot said:

    I've been wondering, is there any food that is baters, deep friend, then battered again, and deep fried again?

    Your ideas are intriguing to me, and I wish to subscribe to your newsletter.


  • Discourse touched me in a no-no place

    @blakeyrat said:

    @Qwerty said:
    @Zecc said:
    Bonus points if you had actual police officers pushing the buttons and joysticks at random.

    No, that would be copper testing.

    Unless they're battery-powered, then it's copper topping.

    That sounds kinky somehow.



  • @bgodot said:

    is there any food that is baters, deep friend, then battered again, and deep fried again?
     

    UK have two responses: 

    • Deep Fried Mars Bar - something is covered in melted chocolate and allowed to cool, then dripped in batter and deep-fried. It's as strange as it sounds: I managed a third of one before my arteries threatened to physically separate and storm off in protest.
    • Deep Fried Pizza - some variants batter the pizza before deep-frying it.
    It's as bad as it sounds. I've never tried the latter, but it strikes me as turning the cheapest and most nutritionally weak pizza into an even more unhealthy option, if that were possible.

  • Considered Harmful

    @blakeyrat said:

    @bgodot said:
    I've been wondering, is there any food that is baters, deep friend, then battered again, and deep fried again?

    Your ideas are intriguing to me, and I wish to subscribe to your newsletter.


Log in to reply