Seven principles of software testing

1.  Testing shows the presence, not the absence of error states

Testing can show that fault conditions are present, but it cannot prove that fault conditions do not exist. Testing reduces the probability that undiscovered errors are still present in the software, but even if no error conditions are found, testing is not proof of correctness.

This is a very important principle, also for stakeholders of testing, in order not to pursue false expectations of testing.

2. Complete testing is impossible

Completeness is a nice goal. But with all possible input values and their combinations, taking into account all the different preconditions, the goal is not achievable - with the exception of very trivial test objects. Instead of trying to test completely, risk analysis, test procedures and prioritisation should be used to focus the testing effort.

The need for thoughtful test planning becomes clear here, where test efforts are planned based on identified and assessed risks (keyword risk-based testing), among other things.

3. Early testing saves time and money

To find error conditions early, both static [e.g., document reviews] and dynamic testing activities [e.g., unit tests or acceptance tests] should be started as early as possible in the software development lifecycle. Early testing is often referred to as shift left. Early testing in the software development lifecycle helps to reduce or even completely avoid costly changes.

Early testing (and testing in general) helps a company to be more economically successful!

4. Error states occur frequently

A small number of modules are often responsible for most of the error effects in the operation of a software or contain most of the error conditions discovered during testing in the pre-commissioning phase. Feared accumulations of fault conditions and the actual observed accumulations of fault conditions in test or in operation are an important input to the risk analysis used to focus the testing effort (as mentioned in Principle 2).

Thus, one task for test control is to measure defect densities and include them in risk-based testing (i.e., testing based on feared defects and defect consequences).

5. Tests wear out (former pesticide paradox)

If the same tests are repeated many times, they become increasingly ineffective at detecting new defects. To overcome this effect, existing tests and test data may need to be modified and new tests written. If the same paths are followed again and again, it is almost impossible to discover anything new.

In some cases, however, repeating the same tests can lead to a positive result, e.g. in automated regression tests.

So, activities to modify existing tests and test data should be scheduled well in advance in test planning, including test effort estimation.

6. Testing is context-dependent

Depending on the application and context, testing must be adapted. For example, safety-critical industrial control software is tested differently than a mobile e-commerce application. Another example is testing in agile projects, which is performed differently than testing in a project with a sequential software development lifecycle.

The context has an influence on software development lifecycle models, risk assessments, test efforts, test procedures and test environments, among other things.

7. Fallacy: "No errors" means a usable system

Some companies expect testers to be able to run every conceivable test and find every conceivable error condition. But Principles 1 and 2 teach us that this is impossible. Moreover, it is a fallacy that simply finding and fixing many bugs [as part of verification] leads to a successful system.

This is because, despite thorough testing of all specified requirements [verification] and fixing the errors found, a system may be created that is of lower quality than comparable systems or that fails to meet the user's needs and expectations. It would thus fail validation.

Here it becomes clear that it is not sufficient to merely verify a product. If, at the end of product development, a product is to emerge that sufficiently fulfills the needs and expectations of the user, validation is also required.

Source: ISTQB® Certified Tester Foundation Level Syllabus V 4.0