Is XSEDE a wise use of public money?

At the XSEDE15 conference, the award-winning technical paper considered the return on investment (ROI) for XSEDE. The paper was co-authored by Craig Stewart of Indiana University’s Pervasive Technology Institute (PTI), Timothy Cockerill of the Texas Advanced Computing Center (TACC), Richard Knepper of Indiana University's Pervasive Technology Institute (PTI), Richard Moore of the San Diego Supercomputer Center (SDSC), Ralph Roskies of the Pittsburgh Supercomputing Center (PSC), and Justin Whitt of the National Institute for Computational Sciences (NISC). The full paper is available here.

Smart Money?

Is money invested in the eXtreme Science and Engineering Discovery Environment (XSEDE) well spent? A team of researchers found good news for taxpayers when weighing service provider reports, cost avoidance, and value added to scientific research.

Over the last 30 years, cyberinfrastructure (CI) funding by the US National Science Foundation (NSF) has shifted from independent centers to a single organizing and supporting entity. In 2011, XSEDE became the central body for delivering NSF-funded CI services in the US.

“In the 1980s, the supercomputing center program provided a small group of researchers with access to extreme-scale computing," says Craig Stewart, lead author of the report. “Now, the mission for NSF CI facilities serves thousands of researchers, accelerating research and helping develop a diverse, technically savvy 21st-century workforce. It’s much more difficult to assess the value of NSF CI facilities today.”

The researchers argue that XSEDE is not only more relevant to the broad scientific community, but also adds value as an integrating point for computing resources, services, and standards. XSEDE’s scale enhances this value by fostering collaboration around the world and across the disciplines, thus extending broad technical expertise and support (whether a user needs to learn basics like how to run scripts or advanced techniques like massively parallel programming).

By employing the accounting metric of cost avoidance, the team set out to better quantify XSEDE's value – in particular, how the XSEDE model fared against the minimal costs of models of independent centers serving national audiences. Looking at the functions required to maintain a national CI center, Stewart et al. projected that XSEDE avoided $18,418,250 compared to four independent CI centers, and $3,428,250 per year compared to two independent centers. That, Stewart indicated, demonstrates that XSEDE’s operations are relatively cost effective as compared to other alternatives.

The cost exercise, however, still leaves open the question of whether XSEDE is a good investment of taxpayer monies. XSEDE provides services directly to Service Providers (SPs) – the organizations operating the advanced CI systems supported by XSEDE – and end users. XSEDE polled SPs to ask them the value of services provided by XSEDE. The SPs reported an impressive annual value: Over $11,000,000 (based on XSEDE budgeted staff costs).

ROI is a ratio relating generated income to resources used to produce that income. A ROI over 1.0 indicates return greater than investment. Ultimately, the research team's accumulated evidence shows the value US taxpayers get from investing in XSEDE services is greater than the $26 million invested annually.

Since the research investments generate yields over years or decades, measuring the value of research investments is never simple.

For instance, what value can be attributed to a Nobel Prize for science enabled by high-performance computing? Can detailed seismic simulations be adequately assessed in terms of dollars? How much does a stock market crash cost? With the aid of the XSEDE Novel and Innovative Projects program, the work of Professor Mao Ye of the University of Illinois College of Business resulted in regulation changes increasing the security of the market against crashes.

Stewart pointed out that “this is very much a first attempt at measuring the ROI. Still, as far as we can tell, it is the first time anyone has ever published a ROI analysis on an entire CI center or organization.” While the research team will continue conducting surveys to provide a more precise ROI for XSEDE, their initial conclusions suggest US investment in science is a smart bet.

--Lance Farrell

--iSGTW is becoming the Science Node. Watch for our new branding and website this September.