Simulating IceCube data using GPUs

The IceCube laboratory at the Amundsen-Scott South Pole Station, in Antarctica, hosts the computers collecting raw data. Due to satellite bandwidth allocations, the first level of reconstruction and event filtering happens in near real-time in this lab. Only events selected as interesting for physics studies are sent to UW-Madison. Image courtesy the US National Science Foundation and IceCube Collaboration.

IceCube, the world’s largest neutrino detector, uses 5,160 light sensors to detect the tiny light flashes that result when neutrinos interact with the ice at the South Pole. The detector is sometimes referred to as a telescope because its sensors can take precise energy and directional measurements.

IceCube encompasses a cubic kilometer of ice. Construction started in 2004 and finished in 2011. The collaboration now includes 300 people from 44 institutions in 12 countries.

The sensors start at a depth of 1.5 kilometers and go to a depth of 2.5 kilometers. They generate about one terabyte of data every day, 365 days a year — too much for the project’s satellite hookup, which allows about 100 gigabytes per day. Fifty on-site servers filter out the most interesting 100 gigabytes of data, which is then sent to the University of Wisconsin–Madison, US, (UW–Madison) via the satellite connection.

Data processing and analysis begins without much lag. The raw data is archived at the pole, and the project team physically retrieves it once a year for long-term archiving. During analysis and production of results, data can expand to three- or four-times the original, filtered size. “Many of our analyses rely on simulations which we then compare to the actual data,” says Gonzalo Merino, IceCube computing facilities manager at UW-Madison.

“The Monte Carlo simulation data piles up to more than a half petabyte every year. IceCube, in terms of core computing resources, includes the main data center, Tier-0, here at UW-Madison, where the data is received from the South Pole. We currently warehouse about 3.5 petabytes at the Tier-0 — all the accumulated data since the beginning of the detector. There is also a Tier-1 center in Germany, DESY-Zeuthen, where the processed data products are replicated.”

UW-Madison also provides data access and processing services for collaborators all over the world. Most are in the US and Europe, and some are in Asia and Australia.

“The detector looks simple,” adds Merino. “But we need lots of computing power to model the very complex physics processes that take place when IceCube detects cosmic particles. We have a lot in common with the Large Hadron Collider and other particle physics research in that respect. What we do is very similar to those large experiments except that our project is smaller, so we have fewer people and fewer resources. That is also a challenge. Being well connected to other collaborations is very important because we can learn from one another.”

Using HTCondor, the team submits a job that will run on UW-Madison resources first but then will automatically run on other Open Science Grid (OSG) resources if more computing power is needed. This happens transparently for users.

Merino says that one thing they are doing in IceCube that could be brought to bear elsewhere is using graphics processing units (GPUs) alongside central processing units (CPUs). “For some workloads, GPUs are critically important,” he says. “We need good modeling of how millions of photons propagate through the ice and then to be able to understand the data coming from the sensors.”

Tracking photons in the ice is one of the big computing challenges. To exploit the detector's full potential, they needed higher resolution simulations of the way light propagates in the ice. “GPUs turned out to be a perfect fit,” notes Merino. “In the end, light propagation is pretty much what is done in video games, so we started using GPUs, and it worked. GPUs are doing a good job in photon simulation — up to 300 times faster than CPUs."

Using GPUs has since become a core part of their process. "We had to learn how to run a large cluster of them, a pretty steep learning curve in terms of system administration. We also have developed middleware that orchestrates all these workflows.” Part of the workload requires GPUs; other parts use CPUs. “Right now, all of the GPUs used for simulation belong to or are allocated to IceCube,” notes Merino. “We are looking for ways to use opportunistic GPU resources on the OSG.”

Last year, the project reported on 28 neutrinos that likely came from a source outside our galaxy. In October, they presented observations from three years' worth of data in a new paper.

A version of this article appeared first in the Open Science Grid newsletter.