In the last few years, researchers have identified disturbing flaws in the way that experiments are performed in computer science. For example, in the area of performance evaluation of computer systems, our measurements on one system are rarely reproducible on another. As hardware and software grow more complex, this problem just gets worse.

Bad evaluations misdirect research and curtail creativity. A poorly performed but successfully published evaluation can encourage fruitless investigation of a flawed idea, while publication of flawed  observation can discourage further exploration of an important area of research.

In this mini-lecture we distill not only our own experience in performance evaluation but also the insight gained through our involvement in the "Evaluate Collaboratory" (<link (http://evaluate.inf.usi.ch>http://evaluate.inf.usi.ch</link>), an open group of researchers with an interest in improving the state of practice in experimental evaluation of software and systems.

This mini-lecture covers topics including the selection of benchmarks, performance metrics, measurement bias and accuracy, statistics, reproducibility, negative results, and observational studies. The lecture includes discussions of recent research results on these topics. Besides the above topics, the lecture can include discussions of topics and issues proposed by the participants. The goal of this mini-lecture is to enable the participants to learn from past mistakes of others and to avoid common pitfalls in their own future performance evaluations.

This lecture series is particularly devoted to PhD students and advanced master students.

Date & Venue

• 25.05.2011, 9-11 h, Bibliothek E185.1
• 26.05.2011, 8-10 h, Seminarraum Argentinierstr.
• 27.05.2011, 9-11 h, Seminarraum Argentinierstr.

Short CV of Prof. Dr. Matthias Hauswirth (University of Lugano, Switzerland):

Matthias Hauswirth is an assistant professor at the Faculty of Informatics at the University of Lugano (Switzerland), where he leads the Software and Programmer Efficiency (Sape) research group. He is interested in performance measurement, understanding, and optimization. Matthias received his PhD from the University of Colorado at Boulder. (<link www.inf.usi.ch/faculty/hauswirth _blank>http://www.inf.usi.ch/faculty/hauswirth</link>, <link sape.inf.usi.ch/&gt;http://sape.inf.usi.ch/</link>)

The Sape Research Group at the University of Lugano is recruiting PhD students for the ParaBoost project (<link sape.inf.usi.ch/paraboost&gt;http://sape.inf.usi.ch/paraboost</link>) for research on concurrent speculative execution in virtual machines.