CSE403 Software Engineering Autumn 2001More Testing
Gary Kimura
Lecture #10
October 22, 2001
Today
•Continue on with testing
•Understand the distinction between white-box and black-box testing
•Understand high-level basics of black-box
•Understand coverage issues in white-box
Discreteness
•It’s also important to remember that testing software isdifferent from testing physical widgets
–In general, physical widgets can be analyzed in terms ofcontinuous mathematics
–Software is based on discrete mathematics
•Why does this matter?
•In continuous math, a small change in an input correspondsto a small change in the output
–This allows safety factors to be built in
•In discrete math, a small change in an input can correspondto a huge change in the output
Ken Butler
•“The only unit of change in a program is the wholeprogram.”
•That is, if you change anything in a program, you shouldtreat it as if you changed everything
–Of course, this isn’t practical in all cases
–But it’s a darned good model that captures the discrete nature ofsoftware
General observation I
•A goal of picking a testcase is that it becharacteristic of a class ofother tests
•That is, one case buildsconfidence in how othercases will perform
General observation II
•The overall objective is tocover as much of thebehavior space as possible
–It’s generally infinite
•In general, it’s useful todistinguish the notions ofcommon vs. unusual casesfor testing
Black box testing
•Treat the unit (program, procedure, etc.) as a black box
–You can hypothesize about the way it is built, but you can’t see it
•Depend on a specification, formal or informal, fordetermining whether it behaves properly
•How to pick cases that cover the space of behaviors for theunit?
–Use heuristics
Equivalence partitioning
•Based on input conditions
–If input conditions are specified as a range, you have one validclass (in the range) and two invalid classes (outside the range oneach side)
–If specified as a set, then you can be valid (in the set) or invalid(outside the set)
–Etc.
Boundary values
•Problems tend to arise on the boundaries of input domainsthan in the middle
•So, extending equivalence partitioning, make sure to pickadded test cases that exercise inputs near the boundaries ofvalid and invalid ranges
Others include
•Cause-effect graphing
•Data validation testing
•Syntax-direct testing
•…
White box testing
•In this approach, the tester has access to the actual software
•They needn’t guess at the structure of the code, since theycan see it
•In this approach, the focus often shifts from how the codebehaves to what parts of the code are exercised
White box coverage
•In black box, the tests areusually intended to coverthe space of behavior
•In white box, the tests areusually intended to coverthe space of parts of theprogram
Statement coverage
•One approach is to cover all statements
–Develop a test suite that exercises all of a program’s statements
•What’s a statement?
max = (x > y) ? x : b;
if x > y then
max := x
else
max :=y
endif
Weakness
•Coverage may miss some obvious issues
•In this example (due to Ghezzi et al.) a single test (anynegative number for x) covers all statements
–But it’s not satisfying with respect to input condition coverage, forexample
if x < 0 then
x := -x;
endif;
z := x;
Edge coverage
•Another approach is to usea control flow graph(CFG) representation of aprogram
–Essentially, a flowchart
•Then ensure that the suitecovers all edges in theCFG
Condition coverage
•Complex conditions can confound edge coverage
if (p != NULL) and (p->left < p->right) …
•Is this a single conditional statement in the CFG?
•How are short-circuit conditionals handled?
–andthen, orelse
Path coverage
•Edge coverage is in some sense very static
•Edges can be covered without covering paths (sequencesof edges)
–These better model the actual execution
Example
Path coverage and loops
•In general, we can’t boundthe number of times a loopexecutes
•So there are an unboundednumber of paths in general
–We resort to heuristics likethose from black boxtesting to exercise theseloops
Usefulness?
•Even with a CFG and statement coverage as testingtechniques there is still a lot to be desired
–How does one pick the proper input that will actually exercise acertain statement or path?
–What about complex interactions with other system events andcomponents?
–Etc.
Some more practical aspects
•Who tests the tests, especially a large complicated test?
–For example, if your triangle test program generates random data,who confirms the results?
–Another example is testing trig functions.
•Testing the error cases can be a wider set of inputs. Youhave two problems
–Making sure you have proper test coverage and
–Making sure the results are correct.
•Fault injection is another way of testing systems.
–For example, injecting I/O failures in a disk controller can test theerror cases for the disk driver and file system.
–Another example is injecting memory allocation errors, to see howprograms behave when they run out of memory.
Final note on testing
•It’s unsound
•It’s heuristic
–Heuristic doesn’t mean undisciplined
•It’s extremely useful and important
•Good testing requires a special mindset
–“I’m going to make that sucker fail!”
•Good coding requires a special mindset
–“Nobody’s going to break my code!”
Next time
•Wednesday
–Group Project Work Day
•Friday
–Bugs