Automation test case coverage

Basics of Automation test coverage

• Test coverage by feature
• Test coverage by GUI icon
• Test coverage by instrumentation
• Test coverage by structure
• Test coverage by scenario
• Test coverage by transition
• Test coverage by web script

1. Test coverage by feature The specification says the system has the following n features plus start-up and shut-down. Do we have a test (set) for every feature plus start-up and shut-down?

2. Test coverage by GUI icon. The user interface has a number of screens, buttons, pull-downs, tabs, menus, etc. Do we have them all listed, and do we have tests which execute every one?

3. Test coverage by instrumentation Use a code instrumentation tool to instrument a build, and then test that build using the system tests already prepared. The tool output should be able to indicate how much coverage in code terms the system had. Note that this need not occur for every build once sufficient code coverage is assured.

4. Test coverage by structure When unit testing you need to be sure you have exercised some minimum part of the code. Testing should include Statement coverage, Decision (branch) coverage, Condition coverage.

5. Test coverage by scenario Users have a number of goals which they want to achieve. They achieve them using a number of (parts of) features.

6. Test coverage by transition Typically on web applications, but also in more conventional applications, there is a number of “paths” a user may take to achieve a goal. These paths need to be identified possibly in the form of a state transition diagram (typically from URL to URL in the case of a web test) such that a minimum number of paths can be identified and traversed.

7. Test coverage by web script web page, application, and component. Having identified the risk level of the website, you can then decide the level of coverage necessary to mitigate that risk by selecting the test types.