Robustness of Software
Seminar Software-Development (programming styles),
WS2002/2003
Johannes Kepler University Linz,
System Software Group
Christian Zeilinger
Summary
This paper gives
an idea, how we can improve the robustness of our software. Starting with a
short motivation which shows that we should program defensively, main
principles like “Design by Contract”, “Resource-Management” and
“Self-Describing Data” are treated. Finally, the paper deals with different
“Testing”-strategies. All these points should lead to more stability in our
computer’s daily life which should in addition prevent us from frustrating and
annoying situations.
Table of Contents
2.2 Contracts and Software Development
5.1 Testing in the Software-Life-Cycle
5.2 Black-Box vs. White-Box-Testing
1
Motivation
You Can’t Write Perfect
Software! [HuTh00]
Unfortunately, this is true. There are always some
tricky errors which are very hard to find - even with our best testing
techniques. Therefore, they often stay in our software. A suddenly full hard
disk, a nearly impossible user-behaviour our just the so called
cleaner-syndrome are some of those errors which always make troubles.
What should we do now? Should we accept this, or
should we fall into despair. The answer is:
Design your Software as robust as possible!
At this point, I want to mention some of the worst
software-bugs which ever occurred:
·
1985: People died
because of too high radiation in an X-ray apparatus caused by a software bug.
·
1996: The
spacecraft Ariane 5 explode 36 seconds after the
start. The problem was the software which was also used in its predecessor, Ariane 4. It contained an error which didn’t cause a fault
in the old version, because it was smaller than the new one. The effect of this
error was a change in direction which was too big for the new larger version of
the spacecraft which broke due to aerodynamic forces.
·
1999: The Mars
Climate Orbiter burned up in the Martian atmosphere
because data that was in expressed in English units of measurement was entered
into a computer program that was designed to use metric units.
·
The original
software in the F-16 fighter plane would have turned the plane upside-down when
it crossed the equator. Fortunately, this problem was detected early enough
with the help of different simulations.
Improve the robustness of your software –
Program Defensively!
This means that you should design your software in
respect of robustness. Not only should your program work in a correct way in
each situation, it should also resist against any possible user-misbehaviour or
error in its surroundings. This implies that you always should check the
assumptions you made during the development process. Such assumptions could be:
·
“This value should normally be positive here”.
·
“The pointer is not allowed to be NULL in this
situation”.
Check
these assumptions!
Definition: Robustness is a system’s ability to handle abnormal
situations [Eif02]
2
Design
by Contract
2.1
Contracts
in General
Nowadays, contracts already exist in nearly every kind
of business. Every time when a company instructs another to produce some kind
of hard-/software or just requests any kind of service, there will always be a
certain contract which regulates the main points that have to be done. This
contract is influenced by user-requirements on the one hand and by
producer-abilities on the other hand.
Example:
parcel service
Requirements
by packet deliverer:
·
Maximum parcel size and weight
·
Payment before delivery
Requirements
by client:
·
Arrival within a certain period of time
·
Careful handling during the transport
2.2
Contracts
and Software Development
This principle can also be applied to the software
design process. There are on the one side always condition which must be
fulfilled to make the software module working. In response, the module has to
guarantee other conditions after its end. A module could be, for example, a
function, a class or a complex software component. These conditions can be
split into three groups:
·
Preconditions:
These must be fulfilled before a module can start to work and therefore the
caller is responsible for them.
·
Postconditions: Conditions which
are (/should be) guaranteed after the end of a certain module. That could be
requirements to a return-value of a function or the fact that a list is sorted
after the call of a sorting-method, etc.
·
Invariants: Constraints
which have to be true all the time from the caller’s point of view. They may be
false during the execution of the module but must be true again afterwards.
Loop Invariants and Class Invariants are two examples.
/*
class invariant:
* array is a field of values which can be set (-> value
>= 0)
* count represents the amount of values which are set
*/
public class IntegerArray {
…………………
public int Max {
get {
int max = -1;
/* loop invariant:
* before
each loop the condition max == max(array[0:i-1]) is fulfilled */
for(int i=0; i
< array.Length; i++) {
if(array[i] > max) max = array[i];
}
return
max;
}
}
public int this[int index] {
get { return
array[index]; }
// Precondition: 0 <= index < size
set {
array[index]
= value; //after this statement is executed, the class invariant is false
count++; //class invariant is true again
}
//
Postcondition: value is set
}
……………………
}
2.3
Contract
Verification
As we want to design robust software, we have to
verify all these conditions. In order to do that, we can use different
techniques:
2.3.1
Assertions
These are calls of a special method with the condition
as parameter (e.g. assert(x>0)). Whenever the condition is fulfilled, the
method will do nothing. In the other case, the method normally prints and error
message and stops the program. This immediate termination (Early Crash) is not
mandatory but suggested because otherwise the program will continue his
execution with incorrect data. This could cause more errors at a later point of
time which eventually results in terrible effects. Furthermore, the detection
of the real error-source would be very hard, if you don’t stop immediately.
using System.Diagnostics.Debug;
public class IntegerArray {
…………………
public int this[int index] {
get {
Debug.Assert(index >0
&& index < array.Length); //check precondition
return
array[index];
}
set {
Debug.Assert(index
>0 && index < array.Length); //check precondition
array[index]
= value;
count++;
}
}
public IntegerArray(int size) {
Debug.Assert(size
> 0); //check precondition
array = new int[size];
for(int i=0; i<size;
i++) array[i] = -1;
}
}
Postconditions and invariants
can be checked in the same way.
Because of the class Debug, you may assume that assertion are just used during the
development process and are switched off in the final version of the software.
In fact, many programming environments support this. However, you should use
this switch very carefully, because you will deactivate all of your data
verification which originally was implemented to increase your software
robustness. Of course, the performance will suffer from such assertions, and
you may think that it is nonsense to leave them in the software, even after
hundreds of tests. But on the other hand, you are normally not able to test
every situation that can occur during the execution of your software. Therefore,
there always can be invalid input data which you never had before. If
performance is a real problem, you should just deactivate those assertions
which are within the critical path and thus a real brake.
2.3.2
Exceptions
Some programming languages support the so called
exception-handling model. This model provides an elegant way to inform the
caller (of a method) about errors or illegal data values. Furthermore,
collective error-handling can be implemented very easily.
The core of this concept forms the so called “exception”
which can be thrown whenever an error occurs (throws statement). This means
that the procedure that throws the exception is terminated and a special block
in the caller-procedure will be executed (try-catch-statement). If this special
block doesn’t exist, the caller will also be terminated and the exception will
be forwarded to the caller of the caller, and so on – until it reaches an
appropriate catch-block or the end of the caller hierarchy which leads to the
termination of the program.
Without
Exception-Handling:
#include
<stdio.h> //without
Exceptions
int main() {
int xMin, xMax, yMin,
yMax, error = 0;
FILE
*file = fopen("file.cfg", "rb");
if (f != NULL) {
error = 1;
}
else if(xMin = fgetc(file) == EOF) {
error = 1;
}
else if(xMax = fgetc(file) == EOF) {
error = 1;
}
else if(yMin = fgetc(file) == EOF) {
error = 1;
}
else if(yMax = fgetc(file) == EOF) {
error = 1;
return -1;
}
if (error == 0) {
printf("%d,%d,%d,%d", xMin,xMax,yMin,yMax);
} else {
printf("Error reading file.cfg");
}
fclose(file);
}
With
Exception-Handling:
using System; //using Exceptions
using System.IO;
class Test {
static void
int xMin, xMax, yMin, yMax;
try {
FileStream
str = new FileStream("file.cfg", FileMode.Open);
xMin = str.ReadByte();
xMax = str.ReadByte();
yMin = str.ReadByte();
yMax = str.ReadByte();
Console.WriteLine("{0},{1},{2},{3}",
xMin,xMax,yMin,yMax);
} catch(IOException) {
Console.WriteLine("Error reading file.cfg");
}
finally {
/*
this code will be executed in every case
(exception
thrown and caught, or normal execution) */
str.Close();
}
}
}
When to use exceptions or
assertions?
The answer is: “in exceptional cases”. For example, an
attempt to open a file which doesn’t exist shouldn’t lead to an exception – normally.
However, there could be the case that the file must exist because otherwise
there would be something wrong with e.g. your operating system – think about
configuration files in Microsoft Windows which exist certainly, and which are
for sure always at the same place. In this case, an exception would be
appropriate.
If you are not sure, if you should use exception or
not, you can ask yourself the following question: “Would my code work, if I remove
the whole exception-handling?” If the answer is “yes”, then everything is okay
and your exceptions are appropriate. Otherwise, you should use another kind of
error-handling (e.g. if-statements, etc.).
Note that you should never use assertion or exceptions
to check user-input data, because it always can be wrong and therefore invalid
data is not exceptional but “normal”. Those verifications should therefore be made
with standard if-statements.
2.3.3
Software-Tools
Another possibility to check preconditions, postconditions and invariants are software-tools. An
example is iContract which is designed for Java. In
order to tell the tool which conditions should be checked, you have to put them
in your comments starting with @pre, @post or @invariant. Then you can use
constructs like forall, exists or implies.
Example: Unique, ordered list:
/*
@invariant forall Node n in elements()
n.prev() != null
implies
n.value().compareTo(n.prev().value()) > 0
*/
public class
OrderedList {
/* @pre contains(aNode) == false
@post contains(aNode) == true
*/
public void insertNode(final Node aNode) {
………
}
}
3
Resource-Balancing
There are many different kinds of resources: memory,
files, threads, timer and so on. All of them are limited. Therefore, it is very
essential to handle them with care. In some cases, your programming language supports
resource management, in other cases you have to do that by yourself. Normally,
you allocate a resource, use it, and give it free afterwards. This can be done
in many different ways. Here is one example:
using System.IO;
class Budget {
FileStream fileStr = null;
void ReadBudget(string fileName, out int budget) {
fileStr = new FileStream(fileName,
FileMode.Open, FileAccess.ReadWrite);
budget = (new BinaryReader(fileStr)).ReadInt32();
}
void WriteBudget(int budget) {
fileStr.Seek(0, SeekOrigin.Begin);
(new BinaryWriter(fileStr)).Write(budget);
fileStr.Close();
}
public virtual int Update(string fileName, int newBudget)
{
int
oldBudget;
ReadBudget(fileName, out oldBudget); //read old budget
WriteBudget(newBudget); //write
new one
return newBudget;
}
}
The file is opened in the method ReadBudget, and written in the
method WriteBudget.
Furthermore, we assume that the file won’t be used after it was written.
Therefore, we close it in WriteBudget. Finally, the procedure Update calls both methods to provide the whole functionality of
updating the budget. – Everything seems do be okay, doesn’t it?
As long as there are no further changes, the program
will work correctly. Now, the company that uses this class to manage their
budget decides that it shouldn’t be possible to write any negative value into
the file. As the programmer knows about sub-classing, he implements the
following solution:
class NewBudget
: Budget {
public override int Update(string fileName, int newBudget)
{
int
oldBudget;
ReadBudget(fileName, out oldBudget);
if (newBudget
>= 0) {
//
just write positive budget
WriteBudget(newBudget);
return newBudget;
}
return oldBudget;
}
}
What is going on now? Whenever somebody wants to write
a negative budget, WriteBudget
is not called because the value shouldn’t be written. Therefore, the file is
left open. For a long time, the software will work correctly – but after this
time it will suddenly crash printing the message “Error: Too many open files”.
Good luck while finding the error!
The main problem is that ReadBudget and WriteBudget share
the file resource via the global variable fileStr, and it is not mandatory
that both methods are called. In order to get rid of this problem, the
following principle does help:
Finish What You Start!
This means that every function which allocates a
resource is responsible for the release. Consequently, the program should be
changed to the following:
using System.IO;
class Budget {
protected void ReadBudget(FileStream fileStr, out int budget) {
budget = (new BinaryReader(fileStr)).ReadInt32();
}
protected void WriteBudget(FileStream fileStr, int budget) {
fileStr.Seek(0, SeekOrigin.Begin);
(new BinaryWriter(fileStr)).Write(budget);
}
public virtual int Update(string fileName, int newBudget) {
int
budget;
FileStream fileStr
= new FileStream(fileName, FileMode.Open,
FileAccess.ReadWrite);
ReadBudget(fileStr, out budget);
//read old budget
if (newBudget > 0) {
WriteBudget(fileStr, newBudget); //write new one
budget = newBudget;
}
fileStr.Close();
return budget;
}
}
Now, only the function Update is responsible for the resource management and the error
from before cannot occur anymore.
This principle is applicable as long as you don’t use
dynamic data-structures. In this case, there are many functions which can add/remove
some elements to/from the structure. Therefore, this principle cannot be used
directly. However, there should be a class which provides the key functionality
of adding and removing elements and which is also responsible for removing all
of them before the program is finished. And this is also a kind of
finish-what-you-start-principle.
Nowadays, garbage collectors help us keeping our
memory free, but there are still many areas where we have to think about a
correct resource management not at least to construct robust software.
4
Self-Describing
Data
Everybody knows the situation to sit in front of an
uncountable amount of data trying to find out what they are representing. Many
of the so called data-files are often designed in respect of readability but their
exact meaning is often not mentioned. Here is an example of such a file:
File „clients.dat“:
Chris Darker,
Cliff Mayers,
Ronald Petterson,
Logically, the first datum represents a name followed
by probably the birthday. Now, it becomes difficult. We could assume that the next
datum is the day since the person is client of our company and that he (Chris
Darker) gets a discount of 20% since 1.12.2002 – but better, we try to contact
the person who was responsible for the development of the data structure and
hope that he remembers which part of the data-stream carries which information.
Another example of a slightly confusing data in a
programmer’s daily life is the Linux directory information:
Output using
„dir“:
total 44
-rw-r--r-- 1 pr17
pr
4810 Dec 4 14:37
ETRC
-rw-r--r-- 1 pr17
pr
4810 Dec 4 14:36
ETRC.BAK
drwxr-xr-x 2 root root
512 Oct 24 2001 TT_DB
-rw-r--r-- 1 pr17
pr
0 Dec
4 17:02 out.txt
drwxr-xr-x 3 pr17
pr
512 Dec 4
14:19 round
drwxr-xr-x 2 pr17
pr
512 Nov 7
15:12 test
drwxr-xr-x 2 pr17
pr
512 Dec 4
14:25 traces1
drwxr-xr-x 2 pr17
pr
512 Dec 4
14:29 traces2
drwxr-xr-x 2 pr17
pr
512 Dec 4
14:33 traces3
drwxr-xr-x 2 pr17
pr
512 Nov 12 17:35
ueb13
drwxr-xr-x 2 pr17
pr
512 Nov 20 16:51
ueb24
drwxr-xr-x 2 pr17
pr
512 Dec 3
15:35 ueb32
drwxr-xr-x 2 pr17
pr
512 Dec 3
17:43 ueb33
drwxr-xr-x 2 pr17
pr
512 Dec 4
12:30 ueb42
drwxr-xr-x 3 pr17
pr
512 Dec 4
12:17 ueb43
What do all of these attributes mean? Which
information is behind the strange number afterwards? Questions, questions and
questions ……
As you can see, it is very important to keep the
relation between data and their meaning. Self-describing data is a very good
idea to do that. This means that you save the data combined with sufficient
information about its representation. The key technique is the concept of
name-value pairs. A possible solution for our first example would be:
File „clients.dat“:
%name „Hans Maier“
%birthday „
%firstTransaction „
%discountStartDate
„
%discountPercent 20
%name „Cliff Mayers“
%birthday „
%firstTransaction
„
%discountStartDate
„
%discountPercent
10
%name „Ronald
Petterson“
%birthday „
%firstTransaction
„
%discountStartDate
„
%discountPercent 5
Now, it is easy to discover the meaning of each
dataset. However, an obvious problem is the length of this new file which needs
much more disc space now. Therefore, we should use an optimized version:
File „clients.dat“:
naHans
Maier|bi03/04/1976|ft02/07/2001|ds01/12/2002|di20
naChristoph Huber| .........
Additional Information in a data dictionary:
ABBREVIATION NAME UNIT
na name text
bi birthday date
ft firstTransaction date
ds discountStartDate date
di discount percent
A further technique, and nowadays the most common one,
is the usage of XML-data-representation. This also relies on name-value-pairs but
adds hierarchical aspects. Because of the wideness of this topic, I don’t want
to work it out in more detail at this point and refer to several papers which
can easily be found on the internet.
5
The
Art of Testing
One of the most important points to improve the
robustness of our software is to test it. Not only do we have to check the
runtime- and input-output-behaviour, also side-effects should be discovered.
Testing is a determined, systematic attempt to break a program that you
think is working [KePi99].
In other words: Our test succeeds if it breaks the
program.
Testing can demonstrate the presence of bugs, but NOT their absence
[KePi99].
The reason is the enormous, nearly infinite amount of
different system states, that all have to be tested,
if we really want to show that a program works 100% correctly.
5.1
Testing
during the Software-Life-Cycle
·
Specification
test: A Specification is the basis for further development and therefore it is
very essential to test it in respect of clearness, completeness and
consistency.
·
Module test: As
soon as a module, like a class or method, is written, it has to be tested.
·
Integration test: When
putting several modules together in order to construct a subsystem, new errors can
come into being. Consequently, further tests are required.
·
System test: The
same occurs when you assemble different subsystems -> more tests are
necessary.
·
Acceptance test:
Finally, you have to show your client that the system works without errors.
The earlier you find a bug, the easier it can be removed.
5.2
Black-Box
vs. White-Box-Testing
Basically,
you can distinguish two different views of testing:
·
Black-Box testing:
Here, we don’t know the exact
content of the module we want to test. Consequently, we focus on the input-output
behaviour which is (/should be) fixed by the specification. This can be done by any third person who
hasn’t developed the module and can also be automated.
·
White-Box testing:
With further knowledge about
the module, we can test in more detail. The input data can be chosen in a way
that, for example, every statement is executed at least once. It is harder to
do this, because you have to know the program-code very well. However, more
errors can be found.
5.3
Measuring
the Progress
In order to get a feeling how much of our program is
already tested, we can “measure” the coverage during executing. This can be
done on different abstract levels:
1.
Statement Coverage: Every statement was executed at
least once.
2.
Decision Coverage: Every condition in a control
statement, like if or while, was at least once true and once false.
3.
Condition Coverage: Every sub-condition was at least
once true and once false.
4.
Path Coverage: Every possible path in every function was
followed (a path is a unique sequence of branches from the method entry to the
exit).
All of these measurements can only be done during
white-box testing where we know enough about the code behind the module.
Black-box testing needs a different strategy. The following example explains a
systematic approach for black-box testing.
- Identify the test-object:
String ToUpperCase(String str, int startIndex,
boolean unicode);
A method which
converts characters to capital letters beginning from a certain position. If
there are also Unicode characters, the boolean-variable
unicode must be set to true. Otherwise, special
Unicode symbols will not be converted, but it is then faster.
- Choose the inputs: As there are always inputs
which lead probably to the same behaviour, we can classify them.
- Divide inputs into equivalence classes (valid
and invalid classes):
str: null / string in ascii-code
/ string in Unicode
startIndex: < 0 /
between 0 and N-1 / >=N
unicode: false / true
- Choose input values at the classes’ borders
startIndex: -1, 0, 1, N-2, N-1, N;
str: null, 1 character, 2 character, N character
- Combine values to construct input data
“Test”, 0, false
“Test”, 3, false
“”, 1, false
All of the valid
class combinations have to be tested. In order to test invalid classes, choose
only one parameter to be invalid.
- Delete useless combinations of input data
str.length == 0 combined with unicode
== false
str.length == 0 combined with unicode
== true
Just one of these
two cases has to be checked.
- Fix the expected output for each input data
combination:
“Test”, 0, false
-> “TEST”
“Test”, 3, false
-> “Test”
Input data together
with the expected output forms the so called test-case.
- Execute test
- Compare output with the expected one
5.4
Test
Automation
As testing is a
systematic process, it is possible to automate its execution. You can always
find similar patterns in the software development process or in the product
itself, and therefore there is a big desire for common test suites. Furthermore,
each test case has to be repeated until it executes without errors. This also
calls for automation. Here are some approaches which provide automated testing:
- Automatic Code Reviews
The code is tested
in respect of programming guidelines like syntactic conventions (i.e. write
each if-block in brackets) or requirements for variable and method
declarations.
- Generation of Input Data
As the example
above shows, many different inputs are needed in order to perform reasonable
testing. These data can be generated automatically. Fundamentally, you can
distinguish two cases:
-
Generic Data
The data is
generated without knowledge about its usage. This means that for example a
string parameter is always tested with a null-string, a string with one
character and one with ten characters. So, it is sufficient to construct one
data generator for every situation where a string is used.
-
Intelligent Data
Here we consider
the usage of the data. If for example a string represents a filename, we
generate the test set: null-string, name of an existing file and name of a non-existing
file. This seems to be the better solution, and in fact it is, but on the other
hand it is connected with more work, because we have to write an extra data
generator for every different case.
- Stress Testing
This strategy
tries to detect errors like buffer overflows, problems with full discs, run
outs of resources, etc. It feeds our program with an enormous amount of data,
which not necessarily has to be different.
- Regression Testing
Each change in
software should lead to a further execution of all test cases. This can be done
automatically. Not only are the test cases saved, also their results are stored
and after a new execution compared to the old ones. If they are equal, the
program has probably no new errors – but it still can contain bugs!
5.5
Test
Termination
Another
question is: When can I stop my test process?
Here are two fine
ideas:
- Stop, if you have found enough bugs!
This
is based on a study which shows that within 10 to 25 statements there is an
average of one error. With this information, you are able to calculate the
approximate amount of errors in your program.
- Stop, if the error detection rate decreases!
Note
that this is only reasonable if you continue in adding new test cases.
6
Bibliography
[Amb00] Scott W. Ambler: "Accessors increase robustness of Java code", http://www-106.ibm.com/developerworks/library/tip-accessors.html,
Oktober 2000
[Bent00] Jon Bentley: "Programming Pearls,
2nd edition" - Kapitel 4, 5. Addison Wesley 2000
[Bent88] Jon Bentley: "More Programming
Pearls" - Kapitel 4. Addison Wesley, 1988
[Dav98] Thomas E. Davis: "Improve the
robustness and performance of your ObjectPool", http://www.javaworld.com/javaworld/jw-08-1998/jw-08-object-pool.html,
August 1998
[Drap02] Anke Drappa: "Checkliste Robustheit", http://www.informatik.uni-stuttgart.de/ifi/se/service/checklists/download/Robustheit-1.html,
2002
[Eck02] David J. Eck: "Introduction to
Programming Using Java" - Kapitel 9, http://math.hws.edu/javanotes/c9/ ,Juli 2002
[Eif02] Eiffel Software: "Building
bug-free O-O software: An introduction to Design by ContractTM",
http://archive.eiffel.com/doc/manuals/technology/contract/,
2002
[Hen02] Michael Hendrix: "Software-Qualität", http://www.wi-bw.tfh-wildau.de/~hendrix/grundstudium/cpp/skript/swqualitaet.html,
2002
[Hum02] Joe Hummel: "Defensive Programming
in .NET", http://www.lfc.edu/~hummel/talks/2002/defensive-programming_files/frame.htm,
2002
[HuTh00] Andrew Hunt, David Thomas: "The
Pragmatic Programmer" - Kapitel 21-25. Addison
Wesley, 2000
[KePi99] Brian W. Kernighan, Rob Pike: "The
Practice of Programming" - Kapitel 6, Addison
Wesley, 1999
[LoRaWr97] C. Low,
J. Randell, M. Wray: "Self-Describing Data
Representation (SDR)", http://www.globecom.net/ietf/draft/draft-low-sdr-00.html,
Oktober 1997
[Sch00] Uwe Schmidt:
"Software-Technik mit
Java: Robustheit", http://www.fh-wedel.de/~si/vortraege/IUG/Sprache6.html,
März 2000
[ScHi02] M. Schmid, F.
Hill: "Data Generation Techniques for Automated Software Robustness Testing",
http://www.moasoftware.co.kr/web/software/28_ictcsfinal.pdf,
2002
[Ste02] Ch. Steindl:
"Skriptum zur Lehrveranstaltung Testen von Softwaresystemen" 2002