Another team in our department has this fairly hefty data crunching application that's been in production about 2 years, with short upgrade cycles about every 10 days. They recently got a new manager who believes in automated testing. We were asked to help them create and implement JUnit tests for it. All in all, there are countless different permutations that the data can take. After some analysis of the data, I determine that there are ~200 fields that can each have its own small enumerated set of values, about 50 sets of fields that can take a variety of values, depending on the business context, and ~150 fields that are simply scalars (dates, numbers, etc).
I recommend that we do edge-case tests for each of the scalar fields (eg: date >= [earliest business date]), brute force tests for each of the enumerated fields (one for each value), and spend most of the effort on the business-context cases.
The other manager decided that his code is too important to merely be tested with edge cases. Oh no. He wanted us to come up with a case for every single permutation of ALL of the data. But that would be one test case for every record in the DB. No matter, the decision had been made, the order has been given: do it!
A few of their offshore guys started writing individual tests. 20 hours a day. 7 days a week.
I took a snapshot of the data from what the manager described as their busiest day ever. I spent a day writing a short routine that loaded each record in turn, and would, via reflection, grab all the relevant fields, and generate the source that would comprise an entire test case. I then wrote another short routine that would keep track of contextual business function so the tests could be logically grouped in nested test cases. Then I kicked it off, verified that it worked properly, and took the next two weeks as previously scheduled vacation, with the intention that I would start working on it when I returned.
Upon my return, the offshore folks were burned out from feverishly cranking out test cases, but still going at it.
I log in, and to my delight, my hard disk was filled with source code for millions of test cases. I took a deep breath and kicked off the compile - it worked. I ran a couple of the test cases at random, just to be sure - they worked as expected. I then announced to the world what I had done, and that I was ready to test the entire application. Everyone was excited. The other manager and mine were standing over my shoulder. I fired it off.
Test # 1...completed Test # 2...completed Test # 3...completed ... Test # 10000...completed ... Test # 50000...completed ...
As this was scrolling by, they asked how long it would take to perform all the tests. I eyeballed it with my watch, and multipled by the total number of test cases. It would take 13 days to run all the test cases. "But we have a 10 day release cycle, and some of that is for coding, and some is for QA". "Well, that's why I said not to do brute force testing like this!"
He finally decided that they would kick off the test suite after coding ceases and while QA is doing their thing. If QA finds a bug, the test suite is restarted after the code is patched. When the test-suite gets lapped by the a subsequent release cycle, they will just jump over the one in the pipe and run the test suite on the newest code.
I'm debating how long I should wait before telling him about the concept of segmenting the data and test cases, and running them on multiple machines in parallel.
Thoughts?I think pizza is delicious, and that the other manager is insane.
why are you generating source code? why not just make a single test that runs all test cases? also, how is that testing every combination? Surely there are possible combinations that are not in your database. Considering your fields I would say you would have at least:
2^32^150 * 5^200
You want to make sure you tell him before somebody else figures it out. Doesn't sound like that's likely, though. I say mention it when the manager's at his most worried, and come out looking like a hero.
why are you generating source code?
The code I'm generating is the source for each JUnit test case. And yes, there were suites of tests, hierarchically organized. You simply run the top level test, and it recursively runs everything else. I should have been more clear: we weren't testing the data so much as the application that crunched it (e.g.: create a transaction that looks like xxx and make sure that the results coming out the other end looks as expected).
While I agree that there can always be another permutation, since they wouldn't allow intelligent business-context testing, and insisted on brute force, and couldn't supply a list of cases to test (I floored him with that one), he agreed that their busiest day would be the best approximation that they could make.
IIRC, there were several billion test cases in all. I figure that we could have gotten away with a few thousand in all if they had let us do it intelligently.
...before somebody else figures it out
Good idea... thanks!
Great! Now you just need to generate tests to verify that the code that generates your test cases is doing so correctly.