Test: Decision coverage



  • Anybody here knows anything about testing and decision coverage?

    I am a contracter/freelancer/self employed IT guy (been coding since 1978) working mostly as a programmer or project manager.

    Recently I decided to get certified within software testing (ISTQB) and am currently participating in a training course.

    This weekend I tried out a test exam, and one of the questions was as follows:

    Giving this pseudo code:

    Input Number_of_coins
    Total = 0
    While Number_of_coins > 0
      Input Coin_value
      Total = Total + Coin_value
      Number_of_coins = Number_of_coins - 1
    End loop
    Print "Your coins has a total value of" & Total

    What is the least number of test cases required to guarantee 100% decision coverage?

    I answered 2 as the WHILE could be considered an IF as well, but the correct answer is 1. When confronting the teacher, he agrees with the test.

    As I see it, you could e.g. input the value zero. In this case, you don't enter the loop at all.

    Decision coverage is about ensuring all decisions are covered (duh), and there's a formal definition as well:

    Decision coverage = Number of decision outcomes exercised / Total number of decision outcomes x 100%

    In short, for each decision you should always follow both the true and false path.

    Am I right or am I right? :-)


  • Discourse touched me in a no-no place

    @ochrist said:

    As I see it, you could e.g. input the value zero. In this case, you don't enter the loop at all.
    Alternatively, entering 0 doesn't actually execute any code that a positive integer would not cover.



    @ochrist said:
    What is the least number of test cases required to guarantee 100% decision coverage?
    Zero and non-zero still goes past that decision, the result of that decision seems irrelevant for the question at hand. This is not to say using 0 as a test case, in general, is a bad idea - it just doesn't appear to contribute to this particular question.



  • @PJH said:

    Alternatively, entering 0 doesn't actually execute any code that a positive integer would not cover.

    That's code coverage, not decision coverage. It's possible that ochrist misread the question multiple times, but that's certainly not the only possibility.

    There is an additional issue, which is that the number of branch points in the source code doesn't necessarily correspond to the number in the object code. Unless you know your compiler back to front, there's a case to be made for including three tests: 0 times through the loop, once through the loop, and more than once through the loop.



  • @pjt33 said:

    That's code coverage, not decision coverage. It's possible that ochrist misread the question multiple times, but that's certainly not the only possibility.

    The official term is 'statement coverage', but you are right - and no: I didn't misread the question :-) 

    @pjt33 said:

    There is an additional issue, which is that the number of branch points in the source code doesn't necessarily correspond to the number in the object code. Unless you know your compiler back to front, there's a case to be made for including three tests: 0 times through the loop, once through the loop, and more than once through the loop.

    Very interesting. And I'm pretty sure you can't get 100% decision coverage anyway - except in very simple systems. And what happens to the source code if you compile it with or without optimiser?



  • @ochrist said:

    Recently I decided to get certified within software testing (ISTQB) and am currently participating in a training course.

    I am certified, so will know this one.

    @ochrist said:

    I answered 2 as the WHILE could be considered an IF as well, but the correct answer is 1. When confronting the teacher, he agrees with the test.

    As do I. You can execute every branch with only one test.

    @ochrist said:
    In short, for each decision you should always follow both the true and false path.

    Am I right or am I right? :-)

    You are correct, but that's not what the question asked: what is the LEAST number of test cases to follow the path. One test (Number_of_coins = 2) can do all decision paths.

    @ochrist said:

    As I see it, you could e.g. input the value zero. In this case, you don't enter the loop at all.

    So what you're saying is that a poor choice of input value may mean the one test you perform may not exercise all the branches sufficiently. That's not a fault of the test case, it's a flaw in the test value used in the test case.

    Two tips I'll give here:

    1. Draw it as a flowchart and then you can trace a spiral path out of the WHILE loop
    2. The "min number" is how much effort is required, not how much more effort you want to do above that. You can run five tests if you wish, but thequestion asked how many at minimum is required.
    HTH!


  • @Cassidy said:

    I am certified, so will know this one.

    As am I - now :-)

    @Cassidy said:

    Two tips I'll give here:

    1. Draw it as a flowchart and then you can trace a spiral path out of the WHILE loop
    2. The "min number" is how much effort is required, not how much more effort you want to do above that. You can run five tests if you wish, but thequestion asked how many at minimum is required.

    HTH!

    That's exactly my point. If you draw it as a diagram, depending on how you do it, you can actually end up having a path you don't cover.

    Basically I think this is a bad exam question, as it requires developer knowledge, and the answer is disputable.

    But now I've passed the exam, so why should I give a damn? :-)



  • @ochrist said:

    As am I - now :-)

    Congrats! 2011 syllabus, I take it?

    @ochrist said:

    That's exactly my point. If you draw it as a diagram, depending on how you do it, you can actually end up having a path you don't cover.

    You can end up with a path you don't cover... or you can end up with one complete path that covers everything. What you're arguing here is that there may be several ways of drawing that diagram, but one way shows a complete singular path through all branches, ergo the minimum number of tests required = 1. You seem to fixated on the notion that it's possible to do it using more than 1 pass, so the minimum must be greater. It's not.

    @ochrist said:

    Basically I think this is a bad exam question, as it requires developer knowledge, and the answer is disputable.

    No, it requires knowledge of recognising loop constructs - there aren't that many ways of writing a while/for/until/do loop.

    It's a classic K4 question: recognise the processing that a code fragment performs, and scope out the number of tests required for the test plan. Next step is to create that number of test cases that exercise the branches. This is a frequently-performed activity in the test design process... why would you think an examination question assessing your knowledge of those concepts is a bad question?

    I don't see the answer is disputable: your lecturer, the examination board and I can do the test in one pass but you can do it in two. Which is the smaller number?

    @ochrist said:

    But now I've passed the exam, so why should I give a damn? :-)

    Because it may come up in your faily life as a tester, and if you answered "2" for that question, then you'll look at creating two tests when one would have done - expend unnecessary effort and increasing the overall test costs.



  • @ochrist said:

    Am I right or am I right? :-)
    You are wrong, because entering the value 1 will do the loop part and then go back to the decision point with a value of 0. So you do exercise both paths, and there's no need to test it separately with 0.

    This is the essence of what Cassidy is saying, but as you don't seem to be understanding him I've tried to phrase it in accordance with the OP in the hope that it will be clearer that way.


  • ♿ (Parody)

    This particular code would be fine with a single test case. In general, I would prefer to have two, to make sure that the code correctly handles the case where nothing inside the loop happens. In this case, the only effect is to change the value of Total, which is just printed out, so it's not a big deal. In more complex code, especially with other side effects, you should really test that, too.

    I'm not certified (possibly certifiable), but I guess this doesn't change the decision coverage as defined, but skipping over the loop certainly needs to be considered as a part of some sort of coverage. That's the sort of test that can find (e.g.) null pointers for you when the code changes in the future. I suspect that's what the OP is thinking, in which case he's wrong in the narrow sense of this question, but correct in terms of building a better test.



  • @Cassidy said:

    Congrats! 2011 syllabus, I take it?

    No, the previous version, I'm afraid. The 2011 syllabus was not ready in my language yet at the time of start (my first language is not English).

    @Cassidy said:

    No, it requires knowledge of recognising loop constructs - there aren't that many ways of writing a while/for/until/do loop.

    I understand what you and others are saying, but remember this:

    1. The definition of decision coverage asks you to follow both paths each time you ask a true/false question.

    2. You (and others) are assuming that the code inside the loop works as expected, but this is what a test should prove (or not prove).

    So it all boils down to how much of the code you can assume is working as expected when planning your tests. In this particular case we can of course for all practical reasons guarantee 100% decision coverage with only one test case - but if the code is just slightly more complicated we don't know it - and shouldn't assume too much.

    In the real exam, there were no questions like this at all. No while constructs (only if/switch type constructs). So I guess somebody else is having second thoughts about this as an exam question (where you have 90 seconds to answer each question).



  • @ochrist said:

    @Cassidy said:

    Congrats! 2011 syllabus, I take it?

    No, the previous version, I'm afraid. The 2011 syllabus was not ready in my language yet at the time of start (my first language is not English).

    Oh, okay. I've just finished writing our course notes, based on the 2011 syllabus. There's very little difference - part 1 just contains something about the culture of the tester and the code of ethics. That's.. about all. (what IS your first language, if you don't mind me asking?)

    @ochrist said:

    1. The definition of decision coverage asks you to follow both paths each time you ask a true/false question.

    No. The definition of decision coverage is that all branches must be exercised, whether it's a true/false condition, a case statement or a loop... i.e. any point at which processing forks.  However this isn't a true/false question, it's a loop - and in drawing a diagram of a loop you'll find that one path leads back to the original decision point:


    so it is possible to traverse both paths with one choice of value, hence the min number of test cases is 1.

    @ochrist said:

    2. You (and others) are assuming that the code inside the loop works as expected, but this is what a test should prove (or not prove).

    I'm not assuming that at all. I'm assuming that if you trace the path through the code for multiple values there will be identical path routes for a range of values, and it is possible that some values navigate through all decision points in one pass, others won't (thus requiring more than one).

    And, in establishing that the min number of cases I need is 1, I then use BVA to discover the domain ranges that would give me a single test case.

    @ochrist said:

    So it all boils down to how much of the code you can assume is working as expected when planning your tests. In this particular case we can of course for all practical reasons guarantee 100% decision coverage with only one test case - but if the code is just slightly more complicated we don't know it - and shouldn't assume too much.

    Again, I don't assume anything when testing, other than what has been given to me is "assumed correct" as perceived by the author (i.e. they're not giving me something that requires unnecessary testing). I do adopt the mindset that testing doesn't prove success, it exposes failure; the more failures exposed, the more certainty that bugs which require fixing still exist, and the greater the overall quality of the product when these bugs are fixed (subject to regression/confirmation testing).

    If the code is "just slightly more complicated" then it's different code, and I'll analyse it accordingly then report my findings of min number of tests based upon that analysis. However, for the code snippet provided, I will still arrive at the same conclusion that your tutor, the ISTQB and numerous other testers found: that the min number of tests required for decision coverage is still 1 (as you're stating above).

    @ochrist said:

    In the real exam, there were no questions like this at all. No while constructs (only if/switch type constructs). So I guess somebody else is having second thoughts about this as an exam question (where you have 90 seconds to answer each question).

    Perhaps they do, but I won't make that assumption: when I come to teach it, I'll show them loops because it's part of the syllabus and I can safely assume that it will be examined upon. Possibly not in their exam, but will be at some point, and continue to do so until it's removed from the syllabus.



  • @Cassidy said:

    (what IS your first language, if you don't mind me asking?)

    I don't mind you asking - as long as you don't mind that I don't answer :-)

    @Cassidy said:

     

    No. The definition of decision coverage is that all branches must be exercised, whether it's a true/false condition, a case statement or a loop... i.e. any point at which processing forks.  However this isn't a true/false question, it's a loop - and in drawing a diagram of a loop you'll find that one path leads back to the original decision point:


    so it is possible to traverse both paths with one choice of value, hence the min number of test cases is 1.

     

    In your diagram, how will you traverse the red path?

    Btw. this is very interesing, but I'm running out of time due to other matters, so can't guarantee that I'll be answering anything right away.

     



  • @ochrist said:

    I don't mind you asking - as long as you don't mind that I don't answer :-)

    Not at all - just wanted to say that your English is pretty good.

    @ochrist said:

    In your diagram, how will you traverse the red path?

    You do a test case that traverses the black route. In doing so, everything in the red route is covered.


  • ♿ (Parody)

    @Cassidy said:

    @ochrist said:
    In your diagram, how will you traverse the red path?

    You do a test case that traverses the black route. In doing so, everything in the red route is covered.

    Which is a great illustration of why simple coverage metrics aren't sufficient.



  • Sufficient for what?

    Metrics are simply that: numbers. It's interpretation of those values that puts them into context and provides meaning.


  • ♿ (Parody)

    @Cassidy said:

    Sufficient for what?

    Metrics are simply that: numbers. It's interpretation of those values that puts them into context and provides meaning.

    The apparent definition of "Test decision coverage" leaves out the case of skipping everything in the loop entirely. Read my previous post for more detail. Basically, you're considering exiting the loop to be the same as skipping over it. So, either the definition of "Test decision coverage" isn't as useful as it seems like it should be, or the OP is right.



  • @boomzilla said:

    The apparent definition of "Test decision coverage" leaves out the case of skipping everything in the loop entirely.

    Aha, I see what you mean.

    Decision/branch coverage only exercises different paths in the code; you may be thinking of statement coverage, in which those situations (statements being covered/omitted) can be analysed.

    I agree that picking poor values for test cases means edge cases could be missed.


  • ♿ (Parody)

    @Cassidy said:

    @boomzilla said:
    The apparent definition of "Test decision coverage" leaves out the case of skipping everything in the loop entirely.

    Aha, I see what you mean.

    Decision/branch coverage only exercises different paths in the code; you may be thinking of statement coverage, in which those situations (statements being covered/omitted) can be analysed.

    I agree that picking poor values for test cases means edge cases could be missed.

    No, I'm definitely not thinking of statement coverage. I'm just going by what you've said. I agree with the OP's answer, but then, I've never read a formal definition of Test Decision Coverage. So if you're right and he's wrong, then I'm thinking of a different definition of Test Decision Coverage.

    It all comes back to that red line in your chart. You claim that the black line tests the red path, and I say it doesn't. But then, I may be thinking of something that might be better described as "Flow Coverage," because the black line most certainly does not test the case of skipping the stuff in the loop. But I would argue that the decision to skip over the loop is a different decision than to exit out of a loop after an iteration. Likewise, the decision to enter the loop is different than deciding to execute another iteration of it.



  • @boomzilla said:

    I agree with the OP's answer, but then, I've never read a formal definition of Test Decision Coverage. So if you're right and he's wrong, then I'm thinking of a different definition of Test Decision Coverage.

    A definition (one of several) here.

    In terms of right and wrong... his tutor and I go along with what ISTQB claim since that's the answer they were looking for, given that question. The topic of how many more tests, or what values would cause paths to be exercised didn't feature into the question - it was about how small a number of tests would allow all branches to be traversed.

    @boomzilla said:

    It all comes back to that red line in your chart. You claim that the black line tests the red path, and I say it doesn't. But then, I may be thinking of something that might be better described as "Flow Coverage," because the black line most certainly does not test the case of skipping the stuff in the loop. But I would argue that the decision to skip over the loop is a different decision than to exit out of a loop after an iteration. Likewise, the decision to enter the loop is different than deciding to execute another iteration of it.

    Again, I can see where both of you are coming from... Perhaps if  we focus on teh green or blue lines, that could make things clearer (I hope).

    In a nutshell: you want to walk all possible branches.

    - pick one value, and you follow the red route. This takes you down to some processing P.

    - pick another value, and you follow the black route back to the decision box, but on the second iteration you exit the box and follow the red route and perform P.

    - pick a third value, and you can circle around the black loop a few times (green, blue) and eventually exit, covering red again, hitting P.

    Now, the question isn't about how many different ways you can get there, the question is about which  route you can take that traverses the lines. The black line doesn't "test the red path" in the sense of picking a value that takes you directly down the red route, it's about covering the branches. 

    Oddly enough, I fell into this trap when I first encountered it, until I realised I was concentrating on how many different combinations of routes there could be, rather than "painting all branches with as few brush strokes as possible". The tutor illustrated the point by putting processing boxes in at each branch and asked how many times I need to run the program to get all the values output (one at the start of the loop, one outside, one inside). I initially thought "three" but then after looking at the diagram, realised you could actually print out all three values with only ONE run.


  • ♿ (Parody)

    @Cassidy said:

    A definition (one of several) here.

    In terms of right and wrong... his tutor and I go along with what ISTQB claim since that's the answer they were looking for, given that question. The topic of how many more tests, or what values would cause paths to be exercised didn't feature into the question - it was about how small a number of tests would allow all branches to be traversed.

    Well, based on that page, decision testing applies to if-then style constructs. There's another section there talking about loop coverage(Has every possible loop been executed zero times, once, and more than once?), which was where I was going, the point being that loops are not as simple as and if-then or case construct.

    @Cassidy said:

    Oddly enough, I fell into this trap when I first encountered it, until I realised I was concentrating on how many different combinations of routes there could be, rather than "painting all branches with as few brush strokes as possible".

    Except (as I said before) that you and the instructor are assuming that skipping the loop is the same as deciding not to execute an additional iteration, and that starting the loop is the same decision as as deciding to execute an additional iteration. Which is the same as ensuring a test case with zero, one and multiple iterations of a loop. So, either the definition is incomplete or the question was applying the concept inappropriately.



  • @boomzilla said:

    Well, based on that page, decision testing applies to if-then style constructs. There's another section there talking about loop coverage(Has every possible loop been executed zero times, once, and more than once?), which was where I was going, the point being that loops are not as simple as and if-then or case construct.

    Right. Why not change the example a bit and see what then happens:

     

    Input Number_of_coins
    Total_number_of_coins = Number_of_coins
    Total = 0
    While Number_of_coins > 0
      Input Coin_value
      Total = Total + Coin_value
      Number_of_coins = Number_of_coins - 1
    End loop
    Print "Your coins has a total value of" & Total
    Print "Your coins have an average value of" & Total / Total_number_of_coins

Log in to reply