Experimental Design and Statistical Analysis Workshop
PORTLAND, OREGON
THURSDAY, 8 JULY. 14:00-18:00 |
|
|
|
|
|
|
| Thomas Bartz-Beielstein |
Cologne University of Applied Science |
| Steffen Christensen |
Policy Research Initiative, Ottawa |
| Mike Preuss |
TU Dortmund |
Mark Wineberg
|
University of Guelph |
|
This workshop is designed to be a hands-on companion to the Experimental Design and Statistics tutorials:
- Statistical analysis for evolutionary computation (EC): Introduction (Mark Wineberg,Steffen Christensen)
- Tuning and Experimental Analysis in EC: What We Still Have Wrong (Thomas Bartz-Beielstein, Mike Preuss)
This workshop will give you a chance to try out the various techniques
learned in the tutorials in a controlled environment so you could then
apply them to your own problems when you get back to your lab. Our
field is becoming more and more rigorous in the application of
appropriate experimental design and statistical analysis for any
experiment being submitted the major conferences in the field. Make
sure that you or your student’s have the experience you need to be able
to perform a proper analysis. You supply your laptop; we will supply
the tools and techniques. Parts of this workshop use “R”, a freely
available language and environment for statistical computing and
graphics. R provides a wide variety of statistical and graphical
techniques. The software can be downloaded from http://cran.r-project.org.
The workshop will be presented at an introductory level and is meant
for any EC researchers who want to compare their newly designed EC
system with established EC systems to see if there is an improvement,
or who want to determine which EC system performs better on their
chosen problem; i.e., nearly everyone. It is vital, if our community is
to be taken seriously, for us to continue to educate ourselves, and
especially new Graduate students, on the statistical and experimental
design techniques that are considered rigor for experiments performed
in any field that is stochastic in nature, as is EC.
Topics Covered
- Experimental Design: Factorial and space-filling designs, experimental setup and scope etc.
- Statistical Analysis: Hypotheses testing, regression, analysis of variance (ANOVA), Bonferonni Post-hoc correction.
- Sequential Parameter Optimization: Introduction to automatic and
interactive tuning of algorithms. The sequential parameter optimization
toolbox (SPOT) package for R is a toolbox for tuning and understanding
simulation and optimization algorithms. SPOT includes methods for
tuning based on classical regression and analysis of variance
techniques; tree-based models such as CART and random forest; Gaussian
process models (Kriging) and combinations of different meta- modeling
approaches. This workshop exemplifies how SPOT can be used for
automatic and interactive tuning.
Schedule
- Intro Talk – orientation
- Download relevant material. Installing an R system and the SPOT package on your computer
- Design first experiment
- Run first experiment
- Analyze first experiment
- Comparison: Interactive and automatic Tuning
- Discussion, Q&A
FAQ
Q-1. How can R be installed on my local computer?
Precompiled binary distributions of the base R system and contributed packages are available from the following resources:
The R FAQ provides further help.
Q-2. How can SPOT be installed on my local computer?
You need to have a running R system. Follow the instructions from Q-1 first.
- Execute the following command from your running R session:
> install.packages(“SPOT”)
- Choose a CRAN mirror. We recommend "Austria"
- Now SPOT files are available on your local hard disk.
- SPOT is a package, which has to be loaded into R's workspace. Execute the following command from your running R session
> library(SPOT)
- Now you can start a demo to see if SPOT was installed correctly:
> demo(spotDemo07RandomForestSann,ask=F)
Note, the option “ask=F” is necessary to avoid user interaction each time the output window is redrawn.
Workshop Organizers
- Thomas Bartz-Beielstein is a professor for Applied Mathematics at Cologne University of Applied Sciences.
- Steffen Christensen is a Senior Policy Researcher at the Policy Research Initiative.
- Mike Preuss is Research Associate at the Computer Science Department, University of Dortmund, Germany.
- Mark Wineberg is Assistant Professor at the Department of
Computing and Information Science, University of Guelph, Guelph,
Ontario, Canada.
References