(Clockwise from top): Nural Akchurin, Sung-Won Lee, Alan Sill and Vanalet Rusuriye examine data transfer and local cluster performance for the Tier-3 center at Texas Tech University while remotely monitoring parameters of the CMS experiment. The mini-Remote Operations Center at TTU keeps the group in close contact with the CMS operations at CERN. Image courtesy Alan Sill, TTU

The Large Hadron Collider will generate a torrential flood of nearly half a Gigabyte of data each second.

It’s too much data to simply record for later contemplation. It would fill your 160 GB iPod in about five minutes, and your 500 GB laptop in about 15 minutes. Instead, physicists will have to filter it, monitor it and analyze it, day in and day out.

That will take the efforts of more than 7500 scientists scattered around the world. Researchers found a way for the physicists who are not at CERN to assist in filtering, monitoring and analyzing the data remotely, in real time.

“That amount of work, it’s hard to imagine,” said Dan Green. Green is chair of the collaboration board for one of the larger LHC experiments, called the Compact Muon Solenoid, or CMS. “it’s going to increase very fast and so we have to engage everybody.”

The amount of data they expect to receive motivated CMS collaborators around the world to create remote operations centers, or ROCs. (Not to be confused with EGEE Regional Operations Centers.)

“Very soon, when the data comes, we’ll have to have learned how to work remotely and collaboratively across many time zones,” said Green. “We have to learn how to do this or we won’t be able to make the transition to simultaneously be running 24/7, upgrading, analyzing and doing data operations.”

Today, several ROCs for the CMS experiment exist. Their capabilities vary, but generally include videoconferencing, data analysis and data operations. The most fully equipped require high bandwidth lines, a professional computing department, local physicists who were involved in the design of accelerator or detector components, and a critical mass of scientists who have experience in operating detectors.

Fermi National Accelerator Laboratory, or Fermilab, inaugurated the first ROC in 2007. It resides on the ground floor of Fermilab’s Wilson Hall, where tourists visiting the lab can see it in action.

The Fermilab ROC is a fishbowl — a large room with a glass wall that looks out on the Wilson Hall atrium. Inside, there is a partial circle of consoles with a small round conference table in the center of the room. Researchers sit scattered throughout the room, engrossed in their work; their screens display computer code, graphs, and other technical materials crucial to data analysis and data operations. During data shifts, large screens visible to everyone display information about the events CMS is detecting.

In short, it looks like what it is: a control room. The only difference is that the experiment it is linked to is thousands of miles away.

One large screen, however, stands out from everything else. It is split into several different sections. One is labeled DESY, for a German particle physics laboratory that hosts another ROC. A second is labeled “CERN,” where a remote operations center was set up because of the distance between the main CERN site and the CMS detector — it’s about a twenty minute drive. The largest window on the monitor is labeled “P5 Control Room.” This is the main control room for the CMS experiment, and it is located near the actual detector.

The windows show the video feeds from the indicated locations. A camera that sits atop the large-screen monitor returns the favor, sending the Fermilab ROC’s live image out to the other operations centers. The video feeds from these four locations never stop transmitting, and are always on display.

Because of its status as the main control room for CMS, P5 does not mute its microphone. Turn the volume up at a ROC, and you will hear the hustle and bustle of people who are working at P5. The ROCs, on the other hand, only transmit audio when someone presses the talk button.

“Some people were concerned about being on camera, but the video feed is bidirectional, so it is more like talking to someone face-to-face in an office environment,” said Erik Gottschalk, who was the project leader for the Fermilab ROC.

In other words, the constant video link is no different from working in a large, open office space. You can hear your coworkers. You can look over and see them, wave to catch their attention, and get to know them by sight. And just as in a normal office or control room, someone could overhear you discussing a difficult problem and offer you assistance, said Gottschalk. They're just offering that unexpected assistance from thousands of miles away.

–Miriam Boon, iSGTW