Feature - Pegasus invites new communities to saddle up

This mosaic image of the Orion Nebula was created by Montage using Pegasus and DAGMan tools. Workflow softare Pegasus chains dependent tasks together, so that multiple tasks can be automatically completed in the correct order.  Image courtesy of Pegasus

Grid workflow system Pegasus recently earned new wings with a share in a US$1.7 million grant from the U.S. Office of CyberInfrastructure, awarded over three years to make further improvements to Pegasus’ usability and breadth of support.

Pegasus’ principal investigator, Ewa Deelman from the University of Southern California’s Information Sciences Institute, says the grant has come at the perfect time.

“We’re now in a position to work with more communities and more users, including those using Open Science Grid and TeraGrid, to bring Pegasus technologies to their computing work.”

Pegasus allows researchers to translate complex computational tasks into workflows that link and manage ensembles of dependent tasks and related data files. 

Deelman says the OCI grant will allow her team to introduce Pegasus with a host of new capacities.

“We’re planning to improve ease of deployment, introduce new debug capabilities, support richer workflow representations and enhance monitoring. We really want to work with new communities; we’re keen to get them started and help them benefit from this technology.”

Comparison of actual and simulated earthquake waves used by the Southern California Earthquake Center CyberShake workflows Image courtesy of Robert Graves, SCEC

Efficient, hands-free workflow management

Pegasus takes a modular approach to workflow systems, working in partnership with workflow execution engine DAGMan, which was developed by the Condor team at the University of Wisconsin and shares in this month’s OCI grant.

The Pegasus/DAGman combination is already used in a variety of scientific applications, including gravitational-wave physics project LIGO, astronomy software Montage and applications at the Southern California Earthquake Center, and can scale to manage large numbers of tasks.

“Some workflows consume terabytes of data and can also produce terabytes of data,” says Deelman. “Some of the SCEC workflows have in the order of 100,000 tasks.”

Another bonus is that Pegasus cleans as it works.

“It minimizes its workflow storage requirements by cleaning up the data produced by the workflow as the computation progresses; it removes data that has already been processed to keep the workflow footprint as small as possible, which helps improve efficiency.”

“Pegasus can also cluster tasks in the workflow,” she adds. “This reduces waiting times, which can add up when you need to manage lots of smaller tasks. This can significantly improve efficiency when you’re dealing with workflows with hundreds of thousands of nodes.”

Deelman says she hopes to continue to leverage collaborations with science communities to advance the state of the art in workflow technology. 

Pegasus and DAGMan were first developed under the National Science Foundation GriPhyN project.

- Cristy Burne, iSGTW