Testbeds for experimental computer science

Only a handful of testbeds for large-scale computer science experiments exist worldwide. Last November representatives of three prominent testbeds - Future Grid and Open Cirrus in the United States, and Grid'5000 in France - gathered for a workshop, co-located with SuperComputing 2011, to discuss the unique challenges associated with using and operating these testbeds.

The workshop organizers and attendees recently released a report based on the workshop; the report highlights some critical issues the field of computer science faces:

"...the experimental methodology for computer science so far has seen relatively little attention and, until recently, little investment."

According to the report, Grid'5000 and Open Cirrus were the first testbeds of their kind. To develop the community of testbed users and their knowledge of experimental methodologies, both testbeds have held meetings where users could share their insights.  

Two main points seemed to arise in the report's conclusion:

1. "...the quality of research in computer science needs more attention, more structure, and more guidelines for computer science practitioners."
2. "Support for experimental science does not come cheaply: new custom capabilities and software must be developed, a higher than expected amount of effort must be devoted to supporting users with specialized requirements, and high levels of low utilization rates must be tolerated because of reconfiguration and availability needs. All these make support for experimental computer science more expensive than support for production science."

Sound interesting? You can download the report here.