All that we see or seem
Is but a dream within a dream.
Testing and Validation
Literature search and definition of your scientific question is due today.
random number generators
bubble sort
quick sort
Timing tests:
you do not have to use the exact number of points I specified
Use the "limit" command to extend your CPU time on the science cluster.
Simulations and Models
Simulation - an attempt to imitate the behavior of a system
Model - the set of dynamical equations used to create a simulation
or visa versa
Not scientific domain
Not computer algorithm or data structure
Not numerical or mathematical methods
What does it all mean?
Is the hypothesis true or false?
What predictions can you make based on the simulation?
Define and write your conclusions, both technical and scientific!
Remember: We are not just computer geeks, we are also scientists!
Quantitative Comparision
numerical comparison of data and model
how do the values compare?
Qualitative Comparision
less rigourous comparison between data and models
how does the morphology and structure compare?
What physical phenomena governs the evolution of this system?
Hydrostatic Equilibrium
Mass Continuity
Radiative Transfer
Energy Generation
Equation of State
1) Mathematical Model
geometry
equations
2) Discrete Algebratic Approximation
3) Simulation Algorithm
4) Technical Plan
5) Code Validation
6) Timing and memory tests
7) Simulation
8) Interpretation and visualization
9) Analysis and Conclusions
Computer simulations are NOT just writing some code.
Don't forget the science.
Know thine code.
Remember: We are not just computer geeks, we are also scientists!
technical plan - part 1
1. Code will be written in ANSI C
2. First targeted platform will be a Sparc 10
3. Simple programming standards will be maintained
4. The code will have internal documentation.
5. The user interface and internal diagnostics will be specified
6. A plan for validation will be created
How do I know this works?
syntax errors
does the darn thing compile?
Fix minor typographical errors
logic errors
are there problems with the coding?
algorithm errors
does the code solve the mathematical model?
model errors
is the mathematical model correct?
symmetry
does geometry affect the results
convergence
do the algorithms actually converge
conservation
are conserved quantities conserved?
consistency
is it consistent with previously published results or analytic solutions
It must successfully model a scientific phenonemina.
Errors must be well understood, predictable, and monitored.
It must run in less than a "Hubble time".
It must have sufficient resolution for the problem under study.
1) Errors are subtle, and difficult to detect.
2) It is more cost effective to plan software than to create it without planning.
3) Your reputation is on the line.
"Publish and perish"
4) The consequences of having an invalid code are too terrible to contemplate.
test the entire program at once
Does it successfully model the desired phenonemia?
Used heavily by NASA during HST design.
Component testing
test each subroutine separately
determine the correctness of each component before integration
Will not find errors in routine integration.
You need to understand the results before you make conclusions.
Careful designing a set of basic tests can save you lots of time.
Determining how to analyze the results is also important.
Do not under estimate the importance of graphics.
Do the results change when I switch
orientation in my system?
Left propagating vs right propagating waves. Are they the same?
Do symmetric phenonema have
symmetric results?
Are forces equal and opposite in a Newtonian system?
It is almost always possible to reduce the problem to some form which has an analytic solution.
Imagine a code which simulates the motion of a particle through a force field which has the form:
F = -k r-a
Where k and a are constants, and r is the distance from the origin.
What analytic solutions could you use to test the code?
F = -k r-a
If a = -1, you have a simple spring and simple harmonic motion.
This can be applied in one-dimension
and solved directly.
If a = 2, you have Newtonian gravity. The solution is found in Kepler's laws.
Asymptotic Approximations
what do the equations do as time gets very large?
Perturbation Theory
what happens to the solution when you change the equation by a tiny amount?
Both have lots of formal theory associated with them.
Both can be applied to validation.
What behavior can we expect in an ecological system?
3 components
grass - grow from nothing, but is eaten by bunnies
rabbits - eat grass, reproduce, and are eaten by dingos
dingos - eat rabbits and reproduce
k0 = munch rate of rabbits
k1 = growth rate of grass
k2 = munch rate of dingos
k3 = birth rate of rabbits
k4 = natural death rate of rabbits
k5 = birth rate of dingos
k6 = natural death rate of dingos
What simplifications can be made in this system?
k0 = munch rate of rabbits
k1 = growth rate of grass
k2 = munch rate of dingos
k3 = birth rate of rabbits
k4 = natural death rate of rabbits
k5 = birth rate of dingos
k6 = natural death rate of dingos
k1 = 0
Primary food source is not renewed.
rabbit population increases
dingo population increases
both populations starve and crash
Case 2: how green was my simulation
k0 = 0
No grass is consumed.
Total amount of grass increases linearly with time.
Case 3: death of a dingo
k5 = 0
Dingos population never increases.
Dingos die off exponentially.
k2 = k4 = 0
Rabbits never die. Rabbit population expands - like rabbits.
Case 5: land of the eternal dingo
k6 = 0
Dingos never die. The dingo population expands.
Case 6: no new rabbits
k3 = 0
No rabbits are born.
Dingo and rabbit populations crash.
The point:
Complex systems often have simple test cases.
What happens when your step size decreases?
Integrate a system with a step size of h. Repeat with a step size of h/2.
Does the answer get closer to the true value?
Is the integration routine stable?
A simple bad example
An explicit routine.
Only uses the last values to solve for the next values.
growing errors
An implicit solver.
You need to invert a tridiagonal matrix to solve this set of equations.
decreasing errors
PDE and ODE solvers have criteria for convergence and for stability.
Not all routines are "unconditionally stable".
"Unconditionally stable" routines should not be used on every problem.
You need to use the correct numerical solution for the problem at hand.
Some physical quantities are conserved in scientific simulations.
They include:
total energy
enthalpy
momentum
center of mass
mass
The conservation of these quantities is usually not explicitly contined in the equations.
Testing for conservation is often a very useful check of validity.
Is kinetic + potential + thermal energy conserved?
You need to calculate each term from the system variables.
In most simulations, energy is NOT explicitly conserved. However, it can almost always be monitored.
Case a: energy is increasing - probably in an unbounded way. System is unstable.
Case b: energy varies randomly -
system is probably stable.
Case c: Energy is decreasing - probably over damped solution.
a - smaller time step - better conservation of Energy
b - larger time step - worse conservation of Energy
Energy allows one way to set the step size.
Can you reproduce Dr. X's results?
Obviously, your code is better than Dr. X's old, moldy, smelly code. Can you strip the elegant features of your code down to Dr. X's level?
Make sure the basics work before moving to the super new features.
symmetry
does geometry affect the results
convergence
do the algorithms actually converge
conservation
are conserved quantities conserved?
consistency
is it consistent with previously published results or analytic solutions
What is the biggest and bestest simulation you can do? Is this sufficient?
How does the time scale as the problem increases in complexity?
What is the slowest part of the code? Can it be improved?
Is there a memory limitation in the simulation?
simple time start and stop time
profile the code
profilers determine how much time is spent in each subroutine
Example:
main1.3 secs
read data3.7 secs
rksolv45.3 secs
output 65.7 secs
main 1.6 sec
read data 5.6 sec
rksolv 123.4 sec
output 68.4 sec
The rate of plant growth depends on how much light reaches there leaves.
The amount of light reaching a plant's leaves depends on the amount of leaves above it.
Each plant has a maximum size and dies at a specific rate.
What tests can be used to validate a program with these equations?
test your codes!
Check for conservation!
Check for convergence!
Compare with analytic approximations!
Learn about
nslookup ping* netstat*
ftp telnet
kermit *sx sy sz
tar