I want to give another example why we can’t actually test everything, only prioritize based on our needs, risk and time.

Let’s consider the software application to test is just a set of functions.

In a software product, functions interact with each-other. The test for a function can pass (like validating an input), but problems could arise   in integration (the export of a report from a web app works, but in Windows the exported file cannot be saved because of some chars present in the filename, which only work on Linux).

So if we test “all”, like some suggest, we definitely have to try to test every combination of function interaction (not really in my opinion, just being sarcastic at those who try this approach)

Let’s say we have N functions (by function not considering necessarily the code definition, but by it let’s say we define “login”,”registration”,”export”,”customize report” etc…)

If we test all, we must test every group of functions so we need to try (registration, login, export) not only (registration,login) (registration,export) and (login,export) pairs. That is because the bugs can occur only in a special case where all the three functions were used (username too long and the export is not possible because of that) (and of course we need to test “ALL” )

Applying a simple formula:

Its the sum of the test cases that are necessarily for testing every situation where we take each function, then we take each group of two functions, then we take each group of three functions etc..

You might say that if a group of three functions works for example, then each of the functions will be working implicitly so we don’t need both type of testing (group and individually). Well I think there are bugs in the application where everything works ok for our basic tests for a group of functions, but the functions individually have flaws. That is because the developers also use the final output as reference for his/her implementation and some time two bugs neutralize each-other for some situations(but it’s still a bug when a function is used in conjunction with a third different one).

For testing “all” and “systematically” we need then also those situations, so we have 2^N test cases now, where N is our number of identifiable functions in the application. If we have 100 functions then we need 1267650600228229401496703205376 test cases!

So that is what you need to test “All”. If you do execute 1000 Test Cases you did basically 0. 000000000(lots of 0′s) 1 % of the necessarily coverage.

Of course you will say that not all combinations are really necessarily, that something is not risky and doesn’t matter, that is not enough time, that you can’t even create automated scripts. YES, because you already started to PRIORITIZE and you NEVER actually tested “ALL” and “SYSTEMATICALLY”!

ShareThis

1. @fabrice31 says:

   October 15, 2012 at 8:23 am

   You can also add a part about the ordre of the test.

   My team found recently a bug on a webpage, existing only of you came from only one other page.

   ALL is important when you test,k but you can test ALL : you have to make choice

   Reply to this Comment
   • Eusebiu Blindu says:

     October 15, 2012 at 8:42 am

     Hi Fabrice,

     Thank you for your comment. I am still confused “but you can test ALL”. Is that a typo?

     Of course integration with external products, different environments, long term duration tests etc… will make the number of TC unable to be represented in the Test Tracking tool

     Reply to this Comment
     • @fabrice31 says:

       October 15, 2012 at 9:02 am

       It is a typo. Even for a little comment, I need more test. Monday effect.

       I wanted to say :
       ALL is important when you test, but you can’t test ALL : you have to make choice.

       Reply to this Comment