Opinion ? Reports of grid?s death are greatly exaggerated

BY CRAIG LEE Craig Lee was named President/CEO of the Open Grid Forum in October 2007. He is a Senior Scientist in the Computer Systems Research Department of The Aerospace Corporation, a non-profit research and development center funded by the US government. At the Aerospace Corporation, Lee advises civil, commercial and governmental agencies on all issues relating to high-performance parallel and distributed computing and is responsible for the transfer of maturing technologies into new application domains. This work has led naturally to Lee?s involvement in the Open Grid Forum where he was a working group chair and an area director prior to becoming president.

Last week, Amazon Web Services announced the launch of Cluster Compute Instances for Amazon EC2, which aims to provide high-bandwidth, low-latency instances for high performance computing.

The announcement was met with a variety of responses from the blogosphere and media. Based on the claim that Amazon had benchmarked its new cluster service at spot 146 on the top500.org list, Bill St. Arnaud asked his readers, ?Should funding agencies ban purchase of HPC clusters by university researchers?? The Register?s Dave Rosenberg, meanwhile, pronounced, ?Amazon sounds death knell for rocket-science grids.?

As always, we?d caution readers to reserve judgment until independent investigations can confirm advertised performance and cost. But today Craig Lee, President and CEO of the Open Grid Forum, responds with his own take on reports of grid?s death.

Mark Twain?s retort when seeing his obituary in the New York Journal seems highly appropriate here. While computing may be viewed as an intellectual endeavor governed by logical objectivity, it is nonetheless surrounded to a degree by hype, hyperbole and fashion. For those who want a clear understanding of what?s happening here, and where distributed computing technology is going, let?s go beneath the marketing hyperbole, and separate concept from implementation from buzzwords.

It?s historical fact that the grid concept came out of the ?big science? arena ? out of a desire to share data and processing capabilities. As such, grids were designed and built by computer scientists to support the way scientists do their work, i.e. staging data through FTP and submitting large jobs to batch queue managers. Doing so, however, required a secure, federated environment to manage identity, discovery, and resource allocation across administrative domains.

In the early years of this century, this concept of resource integration caught the imagination of industry. Why? Presumably because they thought that lucrative markets would develop around resource federation and distributed systems. However, the existing grid implementations at the time turned out to be way too hard and too complicated for the faint-hearted to install and maintain ? that is to say, it had a poor value proposition for most organizations in the marketplace.

Enter cloud computing.

Cloud computing is a fantastic concept for the on-demand provisioning of computing resources ? processing, storage, communication, and services ? through a relatively simple API. This is enabling the commoditization of these resources, thereby creating readily identifiable business models and value propositions. Of course, many different types of computing infrastructures have been built for different computing requirements. HPC centers have been built around massive numbers of processors and associated storage to run tightly coupled codes. Data centers with classic three-tier architectures have been built to achieve massive throughput for web applications.

It is not surprising then that the one size fits all approach of some commercially available public clouds, such as Amazon EC2, would not fit everybody. Tightly coupled applications suffer on EC2 because of insufficient communication bandwidth. It is equally unsurprising that Amazon could deploy a reconfigured cloud infrastructure to provide virtual clusters with acceptable HPC performance.

The fact that a range of computing resources can now be acquired on-demand for a reasonable market price will certainly drive the marketplace. For all the reasons cited by Bill St. Arnaud and others, there will be business and environmental decisions to be made around the cost and carbon footprint involved in owning your own compute resources. The fundamental business trade-off will have to be made in terms of how much ?base? to own versus how much ?peak? to rent. Even if commercial public clouds cannot be used for security or regulatory issues, enterprise clouds will be deployed to realize many of the same benefits of economy of scale, improved utilization, and ease of application deployment. To be sure, commodity, on-demand computing resources will become a fixture on the computing landscape.

But is this the end of the story?

As clouds mature, cloud users will certainly want to develop more extensive application systems that go beyond the basic client-provider model of commercial public clouds. Different governmental and scientific organizations will want to collaborate for national and scientific goals. Businesses will also want to collaborate where it delivers value and gives them an edge in the marketplace. It is easy to imagine a business-to-business scenario that requires the exchange of both data and VMIs. Clearly this will require interoperability and resource federation.

To this end, people are starting to talk about inter-clouds, federated clouds, or community clouds. Securely managing such systems will require federated identity management and role-based authorization to instantiate virtual organizations. Distributed workflow management tools will be required to manage data transfers and process scheduling. Organizations such as the International Grid Trust Federation will have to be set-up to make it all work. This secure management of sets of distributed resources is far beyond what the current public cloud providers are offering.

To sum it all up in one phrase ? grids are about federation; clouds are about provisioning. To say ?data can cross enterprise and data center boundaries in new ways? is to elide the issues of security and governance across administrative domains. These issues have to be addressed regardless of what technology is being used. To say ?grid computing required too much hardware, too much software and way too much money? is to ignore two items:

1. the hardware, software and money that Amazon has sunk into EC2 and Virtual Clusters, and
2. the fact that clouds by themselves do nothing to reduce the ?vast complexity? of federation management.

The important achievement to focus on here is that virtualization has enabled a simple client-provider API for the on-demand provisioning of resources. If you need an inter-cloud, a grid, or whatever you want to call it, some type of virtual organization management will be necessary. The challenge to the distributed infrastructures community is to make such inter-cloud deployment as easy as ?next, next, install.? This will be critical for inter-clouds to build a self-sustaining market.

Buzzwords become loaded with baggage from previous implementations and hype resulting in unrealized expectations. (It is hard to imagine that cloud computing will be completely immune to unrealized expectations!) The advent of cloud computing is indeed an important evolutionary step, but the federation concept is looming larger than ever ? whatever we call it.