Validation: Testing the Model

The basic idea of validating a model is to compare the behavior of the model with the measured behavior. In the example we have a time line from 1991 until 2005 and for some of these time points (cf. 3.2) we have data of the amount of population at that time. The model has been build to compute the amount of population at every time point given a starting time point. Thus a first simple question would be whether the values of the measured data and the simulated data are different. And, if so, what can one say about this difference?

Figure 3.12: Simple validation scenario: comparing measured data with simulated data

Before the start of a first simulation one has to keep in mind that the Change Rate (ChR) is not directly measured but indirectly computed from the measured data. And as one can see is the change rate from year to year not constant! For the model an approximation of $ChR$ has been computed by taking the arithmetic mean. Thus here is some potential of error in the simulation of upcoming values.

Figure 3.13: OMS IMO window for 2D graphical simulation management

Figure 3.13 shows the graphical user interface of OKSIMO in the case a user wants to organize a 2D-Output a simulation. Every of these windows can be separately be handled. Left upper there is a window for the parameter values. Right upper there is a window to manipulate which parameters shall be shown, with which kinds of X- and Y-axes; left below one can manage the kind of output (graphic or text console, how many runs); right below one can see the graphical output in a small format.

For the above example we will give a starting value for $POP$ in the year 1961 with '7839.0' (in thousand) and for $ChR$ with '0.0084' (because we approximated a value of '0.84%'). We want to have the parameter $POP$ in the graphical output, we decide to have '29' runs (1961 - 1990) and we the values for each time point in the format of Bar graphs. This results in the following graphical output (cf. 3.14):

Figure 3.14: OMS IMO 2D Output for run with model f1c

In this picture one has a nicely graphical representation but it is difficult to identify the correct numerical values. It is possible to show the numbers directly in the graph, but then the drawings have to be not so dense, otherwise the numbers are hiding each other. In principal one can decrement the density of drawings by calling a value not at every time point but e.g. only at every second. This results in the diagram 3.15.

Figure 3.15: OMS IMO 2D Output for run with model f1c, less time points, with numbers

This allows the inspection of the concrete numbers in the graph, but one is missing some numbers. Another option is not to show graphical output, but text output in a console window. This looks like the following print out (The values of the ChR-parameter have been let out intentiously to improve readability for the POP-parameter values).

 !!!Die Werte von ChR gelöscht !!!

Wed Mar 11 18:09:16 CET 2009 Editor: 0.8.3 Simulator: 
ForSYnt V 0.4.2d beta - FCL Message: 
Start Simulation with id: f1c_1
Simulator version: ForSYnt V 0.4.2d beta
 Start Time: Mittwoch, 11. Maerz 2009 18:09:16

*** ForSYnt V 0.4.2d beta

$POP_IN1,$CHR_IN2,$POP_OUT1
7839.0,0.0084,
 1 Output:  POP, contains 1 FLOAT: 7.904848e+03 
 2 Input :  POP, contains 1 FLOAT: 7.904848e+03 
 
 2 Output:  POP, contains 1 FLOAT: 7.971249e+03 
 3 Input :  POP, contains 1 FLOAT: 7.971249e+03 
 
 3 Output:  POP, contains 1 FLOAT: 8.038207e+03 
 4 Input :  POP, contains 1 FLOAT: 8.038207e+03 
 
 4 Output:  POP, contains 1 FLOAT: 8.105728e+03 
 5 Input :  POP, contains 1 FLOAT: 8.105728e+03 
 
 5 Output:  POP, contains 1 FLOAT: 8.173816e+03 
 6 Input :  POP, contains 1 FLOAT: 8.173816e+03 
 
 6 Output:  POP, contains 1 FLOAT: 8.242476e+03 
 7 Input :  POP, contains 1 FLOAT: 8.242476e+03 
 
 7 Output:  POP, contains 1 FLOAT: 8.311712e+03 
 8 Input :  POP, contains 1 FLOAT: 8.311712e+03 
 
 8 Output:  POP, contains 1 FLOAT: 8.381530e+03 
 9 Input :  POP, contains 1 FLOAT: 8.381530e+03 
 
 9 Output:  POP, contains 1 FLOAT: 8.451936e+03 
 10 Input :  POP, contains 1 FLOAT: 8.451936e+03 
 
 10 Output:  POP, contains 1 FLOAT: 8.522932e+03 
 11 Input :  POP, contains 1 FLOAT: 8.522932e+03 
 
 11 Output:  POP, contains 1 FLOAT: 8.594524e+03 
 12 Input :  POP, contains 1 FLOAT: 8.594524e+03 
 
 12 Output:  POP, contains 1 FLOAT: 8.666719e+03 
 13 Input :  POP, contains 1 FLOAT: 8.666719e+03 
 
 13 Output:  POP, contains 1 FLOAT: 8.739520e+03 
 14 Input :  POP, contains 1 FLOAT: 8.739520e+03 
 
 14 Output:  POP, contains 1 FLOAT: 8.812932e+03 
 15 Input :  POP, contains 1 FLOAT: 8.812932e+03 
 
 15 Output:  POP, contains 1 FLOAT: 8.886960e+03 
 16 Input :  POP, contains 1 FLOAT: 8.886960e+03 
 
 16 Output:  POP, contains 1 FLOAT: 8.961610e+03 
 17 Input :  POP, contains 1 FLOAT: 8.961610e+03 
 
 17 Output:  POP, contains 1 FLOAT: 9.036888e+03 
 18 Input :  POP, contains 1 FLOAT: 9.036888e+03 
 
 18 Output:  POP, contains 1 FLOAT: 9.112798e+03 
 19 Input :  POP, contains 1 FLOAT: 9.112798e+03 
 
 19 Output:  POP, contains 1 FLOAT: 9.189346e+03 
 20 Input :  POP, contains 1 FLOAT: 9.189346e+03 
 
 20 Output:  POP, contains 1 FLOAT: 9.266536e+03 
 21 Input :  POP, contains 1 FLOAT: 9.266536e+03 
 
 21 Output:  POP, contains 1 FLOAT: 9.344375e+03 
 22 Input :  POP, contains 1 FLOAT: 9.344375e+03 
 
 22 Output:  POP, contains 1 FLOAT: 9.422868e+03 
 23 Input :  POP, contains 1 FLOAT: 9.422868e+03 
 
 23 Output:  POP, contains 1 FLOAT: 9.502021e+03 
 24 Input :  POP, contains 1 FLOAT: 9.502021e+03 
 
 24 Output:  POP, contains 1 FLOAT: 9.581838e+03 
 25 Input :  POP, contains 1 FLOAT: 9.581838e+03 
 
 25 Output:  POP, contains 1 FLOAT: 9.662325e+03 
 26 Input :  POP, contains 1 FLOAT: 9.662325e+03 
 
 26 Output:  POP, contains 1 FLOAT: 9.743488e+03 
 27 Input :  POP, contains 1 FLOAT: 9.743488e+03 
 
 27 Output:  POP, contains 1 FLOAT: 9.825334e+03 
 28 Input :  POP, contains 1 FLOAT: 9.825334e+03 
 
 28 Output:  POP, contains 1 FLOAT: 9.907867e+03 
 29 Input :  POP, contains 1 FLOAT: 9.907867e+03 
 
 29 Output:  POP, contains 1 FLOAT: 9.991094e+03 
 30 Input :  POP, contains 1 FLOAT: 9.991094e+03 
 
..................... done

Wed Mar 11 18:09:16 CET 2009 Editor: 0.8.3 Simulator: ForSYnt V 0.4.2d beta - FCL Message: 
Simulation with id: f1c_1 was finished.
Simulation Time: 0 h 0 m 0 s 59 ms

The interesting question now is how compare the real values with the simulated values? If one compares the run with $ChR = 0.84$ (cf. diagram 3.16) then one can see that the model generates in the first 20 years more population members than measured in the real world and then it generates less members in the following years. But the differences are not constant.

If one manipulates the value of $ChR$ e.g. to the value $0.83$ and repeats the run one detects, that the differences are smaller. Concrete the arithmetic mean of the differences in the run with $ChR = 0.84$ is '-0-41%' and in the run with $ChR = 0.83$ the arithmetic mean of the differences is '-0.17%' ^3.3.

Figure 3.16: Comparing two runs: one with ChR = 0.84 and one with ChR = 0.83

The actual behavior of the modified model $f1c$ looks quite promising. If one starts now a new run for 44 cycles, which means from 1961 until 2005 then one can observe (cf. 3.17) that the model generates increasingly more members of the population than in the real world. This shows that the change rate $ChR$ must be different in the real world than in the model. Whereas the change rate in the model is constant one has to deduce that the change rate in the real world is changing, indeed the change rate decrements. This raises the question: Why? And such a question could be the starting point for a further more fine grained analysis.

Figure 3.17: Forecasting the years 1995 - 2005 with ChR = 0.83