Feature - Pegasus invites new communities to saddle up
Grid workflow system Pegasus recently earned new wings with a share in a US$1.7 million grant from the U.S. Office of CyberInfrastructure, awarded over three years to make further improvements to Pegasus’ usability and breadth of support.
Pegasus allows researchers to translate complex computational tasks into workflows that link and manage ensembles of dependent tasks and related data files.
“Pegasus automatically chains dependent tasks together, so that a single scientist can complete complex computations that once required many different people,” explains Deelman. “It compiles not only software and data, but also the expertise of the people who developed the software.”
Deelman says the OCI grant will allow her team to introduce Pegasus with a host of new capacities.
“We’re planning to improve ease of deployment, introduce new debug capabilities, support richer workflow representations and enhance monitoring. We really want to work with new communities; we’re keen to get them started and help them benefit from this technology.”
Efficient, hands-free workflow management
Pegasus takes a modular approach to workflow systems, working in partnership with workflow execution engine DAGMan, which was developed by the Condor team at the University of Wisconsin and shares in this month’s OCI grant.
The Pegasus/DAGman combination is already used in a variety of scientific applications, including gravitational-wave physics project LIGO, astronomy software Montage and applications at the Southern California Earthquake Center, and can scale to manage large numbers of tasks.
“Some workflows consume terabytes of data and can also produce terabytes of data,” says Deelman. “Some of the SCEC workflows have in the order of 100,000 tasks.”
Another bonus is that Pegasus cleans as it works.
“It minimizes its workflow storage requirements by cleaning up the data produced by the workflow as the computation progresses; it removes data that has already been processed to keep the workflow footprint as small as possible, which helps improve efficiency.”
“Pegasus can also cluster tasks in the workflow,” she adds. “This reduces waiting times, which can add up when you need to manage lots of smaller tasks. This can significantly improve efficiency when you’re dealing with workflows with hundreds of thousands of nodes.”
Deelman says she hopes to continue to leverage collaborations with science communities to advance the state of the art in workflow technology.
Pegasus and DAGMan were first developed under the National Science Foundation GriPhyN project.
- Cristy Burne, iSGTW