At the Uppsala Gala Dinner, Bob Jones of EGEE handed over to Steven Newhouse of EGI his most prized possession — a crown made from all the name tags he collected from conferences in the past six years. Image courtesy GridTalk

After six years, on 1 May, EGEE will hand over responsibility for the world’s largest grid infrastructure to a new organization dedicated to its coordination and development (EGI.eu), and its newly elected director, Steven Newhouse.

During its lifetime, EGEE — Enabling Grids for E-SciencE — assembled a world-wide infrastructure of CPU cores, hosted by computing centers around the world. Each month, about 13 million jobs are executed on the EGEE Grid.

This massive multi-disciplinary production infrastructure was led until now by Bob Jones who initially, like Steven, held the position of technical director at EGEE, and quickly advanced to project director.

During the 5th and last EGEE User Forum in Uppsala, Sweden, Rüdiger Berlich of Karlsruhe Institute of Technology discussed with Bob and Steven topics ranging from the need for sustainability to the relationship between grids and clouds. Here are their comments, in Question-and-Answer format.

Rüdiger Berlich: How would you, in a few words, define today’s grid infrastructures?

Steven Newhouse: Grids are effectively a mechanism for bringing together computing  resources located in different administrative domains for secure accounted for access.

Bob Jones: In terms of grid infrastructure deployment, it is at a global level which has reached production operation. However, there is still a lot of work to be done to make grids easier to use and cheaper to operate.

Berlich: Where does EGEE fit into this?

Jones: EGEE has done a lot of work to push forward production grid deployment and acted as a good showcase of what is possible with production grids.

Newhouse: EGEE has been working for 6 years — and for 3 years in the European Data Grid before this — on these secure, accounted-for access mechanisms for services needed to support high-throughput data analysis.

Berlich: Can you describe the organization's major achievements ?

Jones: Putting in place the largest collaborative production grid infrastructure in the world for e-science. We have demonstrated that such a production infrastructure can be used by a wide range of research disciplines. It has produced scientific results in these disciplines and allowed us to do things which would not have been possible without this infrastructure.

Thus, through EGEE, scientists were able to do more science and on a larger scale, and get results in a shorter time frame. EGEE has formed collaborations within Europe and allowed Europe to collaborate as a whole with other regions. This will last.

Berlich:  Can you give us a few numbers regarding the total investment in grid infrastructures over the course of EDG's and EGEE(I-III)'s lifetime?

Jones: The European Commission has contributed about 70 million euros. The total budget was in the range of 150-200 million euros, depending on how the partners’ contributions are counted.

Berlich:  Can you describe the infrastructure that has been created ?

Jones: EGEE is present in 50 countries around the world. 300 sites contribute to its infrastructure, comprising some 150,000 CPU cores. There are more than 15,000 users. The majority of these users will continue to have access to resources via National Grid Initiatives (NGIs) — these are organizations such as the D-Grid alliance, which manage aspects of national grid deployment — inside of EGI.

Berlich: What is the relationship between EGEE and the Worldwide LHC Computing Grid (WLCG)?

Jones: There are several layers. First of all, there is a continuous exchange of ideas, people and technology. While EGEE has been a multi-disciplinary infrastructure from day one, WLCG has specifically been created for the needs of the LHC experiments and collaborations. LCG makes use of several grid infrastructures, of which EGEE is the largest. Other infrastructures include OSG in the US and NDGF in the Nordic countries.

Berlich:  Where do you see major differences between grids and clouds, and where do they overlap?

Jones: They share the same heart. Amazon-style clouds provide users with a simpler interface than EGEE. By the same token, their middleware is also simpler. Clouds have a far more understandable and obvious business model. The e-science grid world is all about collaboration — bringing together resources that partners had anyway. Clouds cannot satisfy all the needs of grid users today. In particular, aspects of collaboration and result-sharing in virtual organizations are not well covered by clouds today, but will probably come in the future.  Many of the more complex data management aspects are not there either at this time.

Berlich:  Will grids and clouds converge ?

Jones: Aspects of clouds are picked-up and must be implemented in grids. In particular, the interface must get simpler. Virtualization is already present in both and will become even more present in grids, as time goes by.

I strongly believe we will see links being developed between commercial cloud offerings and collaborative grids.

Steven Newhouse of EGI toasting the new era. Image courtesy GridTalk

Berlich:  How will EGEE's mission be carried on after the end of EGEE ?

Newhouse: EGI.eu will continue the coordination of European resources for international collaborations.

Jones: We are happy that EGI.eu is there and see it as a culmination of a lot of what has been done in EGEE over the last few years. I am very pleased to see that there is a sustainable structure in place in Europe.

Berlich: What will happen to EGEE's technical developments, such as gLite and the physical infrastructure after the end of EGEE?

Newhouse: gLite — middleware for grid computing — will continue to be supported and developed by the gLite Open Collaboration which will participate in the EMI project. The physical infrastructure will continue being coordinated by EGI.eu on behalf of its NGI & EIRO stakeholders. EGI.eu will look to deploy software from any external software provider that delivers software of high quality that is needed by our user community.

Berlich: With respect to gLite, from the perspective of “academic software development,” what worked and what didn’t ?

Jones: We managed to assemble a software suite picking the best components available and developing our own to fill in the gaps. I also think we did a good job in the certification and testing part of gLite. I am, however, not so happy with the overall interfaces between the components and the ability to install only subsets of it. This could have been better layered.

Newhouse: It brought a community together that was able to collaboratively develop software to meet the requirements of a user community. The processes for assessing the success of the software (and to stop developing failed prototypes) and to focus on what works and to build on the work of other software developed outside the project was probably one of our least successful activities.

Berlich:  Can you describe EGI.eu's role and where its mission differs from that of EGEE ?

Newhouse: EGI.eu will primarily focus on sustainable coordination of the resource providers in Europe to provide an integrated, secure, reliable infrastructure to meet the needs of its user community.

Berlich:  Where do the NGIs fit in?

Newhouse: NGIs are at the heart of EGI.eu as its stakeholders and as the major providers of the resources within their own national infrastructures. Frequently, the same NGIs will also have strong relationships with the local national elements of the European wide user communities that EGI will support.

Berlich:  Who do you see as the big users of the infrastructure in the future?

Newhouse: Having access to an integrated European infrastructure provides greatest returns to research communities that have European wide collaborations — and beyond. Within Europe the ESFRI projects — i.e., other large international research collaborations. (ESFRI is the European Strategy Forum on Research Infrastructures, a strategic instrument to develop the scientific integration of Europe and strengthen its international outreach.)

Berlich:  What is EGI's policy on industry involvement?

Newhouse: Very open to collaborations where it provides benefits to our user community. I see several areas relating to the infrastructure itself: as providers of cloud resources, providers of software for deployment on the infrastructure for applications, and for management of the infrastructure through standard operational tools. We are also able to support commercial organizations using our resources for pre-competitive research work.

Berlich:  What role does virtualization play and what interfaces will exist between grids and clouds?

Newhouse: Virtualization has had a radical impact on the way data centers in the commercial space deliver computing resources to support their transactional workloads to provide more efficient data centers, both in terms of human resources and in their energy footprint. Similar challenges now confront many of the data centers within EGEE and now EGI as they attempt to support the different service environments required by the increasingly diverse application communities using the production infrastructure.

Introducing a virtualization layer across the European Research Infrastructure could move the software deployment decisions away from the sites and back into the virtual organizations using the infrastructure.

Providing a secure, authorized and accounted-for mechanism across Europe for starting virtual machines on remote sites is in many ways no different from the currently agreed-upon procedures for starting jobs on remote sites. Virtual organization managers, or operations staff acting on their behalf, would prepare, deploy and monitor the software required by that domain.

The Forum’s main auditorium was an appropriate place to talk about architecture. Image courtesy GridTalk

Through the virtualization layer, different virtual organizations would be able to deploy the software needed by their community at an update cycle appropriate to their own work. The workload produced within the ERA is primarily based around data. The high performance research networks around Europe enable the rapid predictable movement of data between sites — many of which have the ability to store many petabytes of data. This capability and associated cost is distinct from that offered by many commercial cloud providers, mainly because the business model within the research networks makes this usage free to the end-user.

However, for activity that is more computing- rather than data- focused, such an architecture provides a bridge to commercial cloud providers as additional VMs can be deployed into a commercial cloud and their services integrated into the broader infrastructure available for a VO.

Berlich:  How do you rate industry interest in grid technologies, from EGEE's perspective?

Jones: A number of industrial applications have been developed for the EGEE environment. However, from the outset, industry doesn’t seem to have been very interested in shared infrastructures. Our technology has been deployed in-house by companies, though. That is why we adopted an Open Source approach with a business-friendly Open Source license. We thus believe that the process of technology transfer from research to industry has succeeded.

Berlich:  Bob, what will you do after the end of EGEE ?

Jones: I will remain at CERN and will still be involved in grids and e-Infrastructures, as well as their use by the research communities in Europe.

—Rüdiger Berlich for iSGTW. Berlich is responsible for the dissemination and outreach activities of the Swiss/German EGEE federation.