Feature - DILIGENT: from digital libraries to virtual research environments

Fires in Greece over August 2007 killed 64 people and destroyed half a million acres of vegetation, leaving a scarred battlefield. Thanks to continual collection of earth observation data, scientists have a detailed record of this event. Image courtesy of European Space Agency

In August 2007, Greece saw more wildfire activity than any other European country in the last decade.

The fires, which involved scores of separate blazes, killed 64 people and destroyed half a million acres of vegetation. Thanks to data collected by organizations such as the European Space Agency, we have a detailed record of this event.

A drop in the proverbial data bucket

Such earth observation data is collected all the time, all around us. For instance, to continue the wildfire example, two European Space Agency satellites—ERS-2 and Envisat—are continually monitoring the Earth’s surface, sensing hot areas: spots exceeding 308 Kelvin at night are classified as “burning” fires. Current and archived maps are online as part of the ESA’s World Fire Atlas. 
 
Such data is invaluable to the efforts of agencies like the European Environment Agency, European Union and the United Nations; however, these data quantities are just a drop in the proverbial data bucket when it comes to ESA’s earth observation.

Over the next 15 years, the ESA alone expect to collect and process around 12 petabytes of data. To manage these mammoth amounts of data, the ESA is turning to grids, and to the DILIGENT project in particular.

Software produced by grid-powered archiving project DILIGENT can make it much easier to sift and search through the many databases of information being collected by agencies all over the world. DILIGENT can help compile, analyse, process and connect this data to other distributed data libraries. Images courtesy of DILIGENT

Virtual research environments, on demand

DILIGENT, which stands for “A testbed DIgital Library Infrastructure on Grid ENabled Technology” is now testing new software that helps to create dynamic, on-demand virtual research environments.

Called gCube, DILIGENT’s software allows users to search for and collect data from distributed and dissimilar resources, and to then process or analyze that data using the requested tools and services. The digital library or research environment thus created can be saved for later use or shared with others.

The ESA are participants in DILIGENT’s testbed phase and welcome the new software, which will help them to more efficiently access huge amounts of space-based data and information, complete distributed calculations, collaborate across distances and, especially welcome, prepare technical reports.

“ESA data and information on earth observation contribute to many regular national and international reports on the status of the environment,” says Veronica Guidetti, ESA’s work package coordinator for DILIGENT. “These are huge, complex documents; they can be referred to as ‘living documents’ because of their need for periodic updating.”

Guidetti says DILIGENT’s technology has the potential to save many hours of time. “For instance,” she says, “if I had to contribute to a yearly technical report on Europe’s environment, with updated statistics and maps using the same calculations as previous years, but with updated input, I would like to submit fast re-processing of entire archives of data with just one click of a button.”

DILIGENT is currently rolling out the first full release of its gCube software, which aims to help organizations rapidly adapt to new requirements, implement new workflows and reprocess data archives. 

Grid for computing; grid for content; grid for services

And since gCube and DILIGENT are grid-enabled, making use of the shared resources of Enabling Grids for E-sciencE, organizations can also access a large amount of computing power for a fraction of the cost.

“This reduces the cost of ownership and increases the availability of exploitable data information, while still preserving user integrity and confidentiality,” says George Kakaletris, DILIGENT validation manager and research associate at the University of Athens. “We’re hoping our findings will become fundamental to the future of the grid.”

DILIGENT celebrates its project sequel as DILIGENT for Science or D4Science starting January 1, 2008. DILIGENT services will be available to the grid-interested public once the system is in production stage.

- Danielle Venton, EGEE