Conformance TestExperiments forDistributed Real-Time Systems
Rachel Cardell-Oliver
Complex Systems Group
Department of Computer Science & Software Engineering
The University of Western Australia
July 2002
Talk Overview
Research Goal:to build correctdistributed real-time systems
1. Distributed Real-Time Systems
2. Correctness: Formal Methods & Testing
3. Experiments: A New Test Method
1. Distributed Real-TimeSystems
Real Time Reactions
Distributed
System Characteristics
React or interact with their environment
Must respond to events within fixed time
Distributed over two or more processors
Fixed network topology
Each processor runs a set of tasks
Processors embedded in other systems
Built with limited HW & SW resources
Testing Issues for these Systems
Many sources of non-determinism
2 or more processors with independent clocks
Set of tasks scheduled on each processor
Independent but concurrent subsystems
Inputs from uncontrolled environment e.g. people
Limited resources affects test control e.g. speed
Our goal: to develop robust test specificationand execution methods
2. Correctness: Formal Methods & Testing
Goal: Building Correct Systems
Design
(intended behaviour)
Implementation
behaves like this?
Software Tests
are experiments designed to answer the question“does this implementation behave as intended?”
Defect tests are tests which try to try to forcethe implementation NOT to behave as intended
Our focus is to specify and execute robustdefect tests
Related Work on Test Case Generation
Chow TSE 1978 deterministic Mealy FSM
Clarke & Lee 1997 timed requirements graphs
Neilsen TACAS 2000 event recording automata
Cardell-Oliver FACJ 2000 Uppaal timed automata
Specific experiments are described by a test case:a timed sequence of inputs and outputs
Non-determinism is not handled well (if at all)
Not robust enough for our purposes
3. Experiments:A New Test Method
Our Method for Defect Testing
1.Identify types of behaviour which are likely touncover implementation defects (e.g. extremecases)
2.Describe these behaviours using a formalspecification language
3.Translate the formal test specification into atest program to run on a test driver
4.Connect the test driver to the system under testand execute the test program
5.Analyse test results (on-the-fly or off-line)
Example System to Test
Step 1 –Identify interesting behaviours
Usually extreme behaviours such as
Inputs at the maximum allowable rate
Maximum response time to events
Timely scheduling of tasks
Example Property to Test
Whenever the light level changes from low tohigh
then the valve starts to open
within 60cs
assuming the light level alternates betweenhigh and low every 100cs
Step 2 – choose a formalspecification language
which is able to model
real-time clocks
persistent data
concurrency and communication
use Uppaal Timed Automata (UTA)
Example UTA for timely response
m:=0
m:=0
Writing Robust Tests with UTA
Test cases specify all valid test inputs
no need to test outside these bounds
Test cases specify all expected testoutputs
if an output doesn’t match then it’s wrong
No need to model the implementationexplicitly
Test cases may be concurrent programs
Test cases are executed multiple times
Step 3. Translate Spec to Exec
UTA specs are already program-like
Identify test inputs and how they will becontrolled by the driver
Identify test outputs and how they will beobserved by the driver
then straightforward translation into NQC(not quite C) programs
Example NQC for timely response
task dolightinput() {
while (i<=MAXRUNS) {
Wait(100);
setlighthigh(OUT_C);setlighthigh(OUT_A);record(FastTimer(0),HIGH-LIGHT);
i++;
Wait(100);
setlightlow(OUT_C);setlightlow(OUT_A);record(FastTimer(0),LOW-LIGHT);
i++;
}// end while
}// end task
task monitormessages() {
while (i<=MAXRUNS) {
monitor (EVENT_MASK(1)) {
Wait(LONGINTERVAL);
}
catch (EVENT_MASK(1)) {
record(FastTimer(0), Message());
i++;
ClearMessage();
}
} // end while
} // end task
Step 4 –test driver
Step 4 - connect tester and execute tests
Step 5: Analyse Results
Scheduling Deadlines Test Results
Concluding Observations
Defect testing requires active test driversable to control extreme inputs andobserve relevant outputs
Test generation methods must take intoaccount the constraints of executing testcases
Robust to non-determinism in the SUT
Measure what can be measured
Engineers must design for testability
Results 1:Observation Issues
Things you can’t see
Probe effect
Clock skew
Tester speed
Things you can’t see
Motor outputs can’t be observed directlybecause of power drain
so we used IR messages to signal motorchanges
But we can observe
touch & light sensors via piggybacked wires
broadcast IR messages
The probe effect
We can instrument program code toobserve program variables
but the time taken to record results disturbsthe timing of the system under test
Solutions
observe only externally visible outputs
design for testability: allow for probe effects
Clock Skew
Clocks may differ for local results fromtwo or more processors
Solutions:
user observations timed only by the tester
including tester events gives a partial order
Tester speed
Tester must be sufficiently fast to observeand record all interesting events
Beware
scheduling and monitoring overheads
execution time variability
Solution: use NQC parallel tasks andoff-line analysis for speed
Results 2:Input Control Issues
Input value control
Input timing control
Input Values can be Controlled
Touch sensor input (0..1)
good by piggybacked wire
Light sensor input (0..100)
OK by piggybacked wire
Broadcast IR Messages
good from tester
Also use inputs directly from the env.
natural light or button pushed by hand
Input Timing Control is hard tocontrol
Can’t control input timing precisely
e.g. offered just before SUT task is called
Solution: Run tests multiple times andanalyse average and spread of results
Can’t predict all system timings for a fullyaccurate model
c.f. WCET research, but our problem is harder
Conclusions from Experiments
Defect testing requires active test driversable to control extreme inputs andobserve relevant outputs
Test generation methods must take intoaccount the constraints of executing testcases
Robust to non-determinism in the SUT
Measure what can be measured
Engineers must design for testability