Feature - Danish grid project aims to learn from the past

Backbone of the Danish national research network. Image courtesy of Forskningsnet

(The following is an informal summary of a 14-page report commissioned by the Danish Center for Scientific Computing. Both the summary and the report were written by project leader Frederik Orellana of the Niels Bohr Institute at the University of Copenhagen. Full text is available.)

Last summer, the Danish Center for Scientific Computing granted funds to a new Danish, academic, cross-disciplinary grid project, operating under the working title of "grid.dk"

This project will pool computing resources from five universities for the benefit of multiple branches of science. Staffing is in place, and we are getting off the ground.

In my opinion, it is a good moment for a new project such as ours to look at history and extract some lessons from projects of the past. Grid computing is now old enough — more than ten years — that we have the luxury of doing so.

In 2008, one of the leaders in the international grid computing environment, Wolfgang Gentzsch, wrote in an Opinion piece in August's iSGTW (Editor's note: What clouds and grids can learn from each other): "During the past 10 years, we have seen hundreds of grid projects come and go, passing away after government funding ran dry . . .  Often, the only asset left after the project's end was the hands-on grid expertise of the project partners, which certainly is highly valuable but in and of itself does not justify all the effort and funding."

Provoked by the seeming discrepancy between the negative vein of these statements and the many success stories reported by EGEE, Open Science Grid, NorduGrid and other major grid projects, we decided to take a serious look at what technology is actually available to help us build our envisioned national scientific computing infrastructure

New technology is available in the form of cloud-provisioning systems from Eucalyptus, Nimbus and OpenNebula. In principle, such technology could allow on-demand software provisioning for compute jobs. We find that while this prospect makes the technology extremely interesting, it is still too early to consider basing a national computing infrastructure on it.

Popularity of grid computing versus cloud computing according to GoogleTrends. Blue: grid computing. Red:cloud computing..  Image copyright GoogleTrends.

Consider the grid?

Instead, we turn our attention to the established grid technology. One of grid computing's main selling points is on-the-fly collaborations, i.e. virtual organizations — which are precisely what we are interested in. What we in particular would like to have is a user-driven and user-administered computing ecosystem, in which:

• small collaborations can be formed on the fly,

• individuals can easily assign rights and share disk space, computing power and software, and

• resource owners can bill users and organizations.

After analyzing past efforts, we find that there's still some way to go before grid systems will provide such functionality in a seamless way. This is inherently related to the heterogeneity of both the computing resources and user needs. In particular, billing, both of hardware
and software usage, is a tricky issue. Therefore we conclude that for us, the most viable path is to start by establishing a national grid system that is dead-easy to use, and be clear about what we offer to the users and system administrators. The more advanced functionality will
then have to wait for future developments — that may well come from the merging of grid and cloud technologies that we are starting to see.

This system will build on existing expertise and infrastructure. We will use traditional grid systems as opposed to renting systems such as clouds. In the Nordic countries this involves deploying NorduGrid/ARC technology in tight collaboration with the Nordic Data Grid Facility, or NDGF. We conclude that the high degree of user-friendliness we aim for will involve a good deal of development work on our part — again harnessing existing experience and expertise in our group. For all this we propose a very simple measure of success, namely the total CPU occupancy time of the involved resources.

—Frederik Orellana, Niels Bohr Institute, University of Copenhagen