Content about Data management

Feature - G-eclipse: easier interface to both grids and clouds The g-Eclipse interface in action.Image courtesy of g- Eclipse Consortium The g- Eclipse Consortium released the g- Eclipse framework, which developers claim will provide an easy-to-use workbench for accessing both grid and cloud infrastructures. The software provides a graphical workbench that enables seamless access with the same simplicity as accessing the Internet from a browser. "This enables interoperabilty between different grid and cloud infrastructures on the client side," said Harald Kornmayer, a researcher at NEC Laboratories Europe, who led the project.It currently supports EGEE's gLite grid middleware (aimed for scientific domains), and the GRIA middleware (used by industry and commerce), as well as AmazonWeb Services' cloud computing and storage offerings.A key feature of the g- Eclipse workbench is its independence from the underlying grid and cloud technology. "We started with the goal of accessing available scientific grid infra

Feature - Rough waters: fighting modern-day pirates with technology

Map of reported pirate incidents in the Gulf of Aden, off the coast of Somalia. (Click for large, high-resolution version.) Image above courtesy of UNOSAT; image on front page courtesy of sxc.hu

In the past year, maritime shipping has suffered a resurgence of piracy, at a level that the world has not seen since the early 18th century. Sailors working off the Horn of Africa have been particularly hard hit: last year, records show that 125 ships were attacked and 45 seized.
Real numbers are likely much higher, as piracy is believed to be widely under-reported. One of the world's busiest shipping lanes, about 20,000 ships annually pass through the Gulf of Aden on their way to and from the Suez Canal — carrying a tenth of world trade.
Unlike the popular image of pirates seen in movies and books, modern pirates are more likely to wield machine guns than muskets; and the crime remains as difficult to fight as it ever

Feature - Cruise control for eScience Image courtesy of howstuffworks.com. The paper “Feedback-Controlled Resource Sharing for Predictable eScience” by Sang-min Park and Marty Humphrey of the University of Virginia figured in the list of finalists for the SC08 Best Student Paper. This article is based on Park’s SC08 presentation on this work.As dynamic, data-driven applications become more common on the grid, it’s tougher to guarantee when any given job will run, and more importantly, finish. For time-critical applications—weather, coastal hazard prediction, and other disaster-prevention—a late result is as bad as no result.Standard batch-mode scheduling cannot predictably support time-dependent, adaptive computing demand, says Sang-min Park, a Ph.D. candidate at the University of Virginia. It can’t change a job’s completion time with fine granularity or cope reliably with unanticipated disturbances. Deadline-guaranteed processing is the key to success for time-critical applications.A

Feature - Opportunistic storage increases grid job success rate Processed events by DZero per week from May to October 2008. The vertical scale goes to 12M events. (Click image for per site representation.) Image courtesy of DZero. The DZero high-energy physics experiment at Fermilab, an Open Science Grid user, typically submits 60,000-100,000 simulation jobs per week at 23 sites. The experiment’s application executables make many requests for input data in quick succession.  Due to the lack of storage local to the processing sites, up until recently much of DZero’s simulated data had to transfer in real-time over the wide area network, leading to high latencies, job timeouts and job failures.OSG worked with member institutions to allow DZero to use opportunistic storage, that is, idle storage on shared machines, at several sites. This represents the first successful deployment of opportunistic storage on OSG, and opens the door for other OSG Virtual Organizations. With allocations of up to 1 TB at sites where it p

Feature - Leading digital data research group moves to North Carolina DICE and the distributed data delugeFor more than 10 years the DICE group’s Storage Research Broker (SRB) data grid has been used by research teams worldwide to automate all aspects of manipulation of large, distributed data files, including discovery, access, retrieval, management, replication, archiving and analysis. DICE most recently developed iRODS, the open source Integrated Rule-Oriented Data System, which introduced user-settable rules that automate complex management policies, helping users tame today’s mushrooming collections of digital data.The team has worked on national and international projects, providing data management systems for major grid and distributed research projects, including the Southern California Earthquake Center, the TeraGrid, the Worldwide University Network, California Digital Library-Digital Preservation Repository, the Laboratory for the Ocean Observatory Knowledge Integration Grid, the Biomedical Informatics Researc

Feature - GridFTP moves your data and your news Cluster-to-cluster data movement with GridFTP using up to 64 nodes on each end of TeraGrid’s 30 Gbit/s wide area network between Urbana, IL and San Diego, CA.Images courtesy of Raj Kettimuthu (plot) and Corky and Holly Siegel (duck). GridFTP, a data transfer protocol optimized for high-bandwidth wide-area networks, handles an average of more than 2.5 million data transfers a day.The Large Hadron Collider, the Southern California Earthquake Center, the Relativistic Heavy Ion Collider, the Laser Interferometer Gravitational Wave Observatory, the European Space Agency, the Disaster Recovery Center in Japan and even the British Broadcasting Corporation use it. Based on the old workhorse FTP, GridFTP supports reliable and restartable data transfers and provides extensions for high-performance operation and security. It is a specification (meaning anyone can write code to implement it) for which standards are defined within the Open Grid Forum. Globus Allianc

  Announcement - Digital Repositories: Interoperability Using Grid Technologies, June 5, Barcelona Image courtesy of OGF The OGF-Europe Community Outreach Seminar, 5 June 2008, within OGF23 in Barcelona, Spain, will explore Digital Repositories (DR) from diverse perspectives, bringing valuable insight into domain requirements, user case studies and best practices. Special emphasis will be placed on interoperability and standardisation.Researchers increasingly depend on open access to information. DR enables access to the new knowledge base and encourages use of scientific content in a standardised, open way. Government agencies show signs of increased adoption, and the Humanities, in addition to Science, the early adopter, are starting to plug into such repositories. What is more, these new tools facilitate public appreciation and understanding of scientific and research advances. Seminar highlights:State-of-the-art and future visions with perspectives from government, research and industry. User case studies from specific do

  Feature - Cell biologists dig new DiGS Cell biologists at the University of Edinburgh and University of Oxford are using grid technology called DiGS to support their investigation into the transport and anchoring of mRNA in Drosophila flies. Image courtesy of Wikipedia As cell biology moves from labs to computer desktops, researchers must deal with increasingly vast volumes of data. Traditional ways of sharing this data—such as physically shipping hard drives—just cannot keep up. UK Quantum Chromodynamics has been working to solve these data management challenges using grid technology since 2002, developing QCDGrid to share, publish and preserve their large datasets. And although UKQCD are a physics outfit, they’ve made their software—renamed DiGS or Distributed Grid Storage—available to the wider scientific community.First off the block: biology“The EPCC team focused initially on the needs of particle physicists,” explains George Beckett of the Edinburgh P

Image of the week - A map of all things science Science-related activity on Wikipedia: Overlaid are 3599 math (blue), 6474 science (green), and 3164 technology (yellow) related articles. All other articles are shown as grey dots. This map is part of the Places & Spaces exhibition. An interactive version of the full Wikipedia visualization is also available. Image courtesy of InfoVis LabKaty Börner knows what Wikipedia looks like in English, all 2.1 million articles of it. Börner, a professor at Indiana University Bloomington, is director of the Information Visualization (InfoVis) Laboratory, and along with her InfoVis colleagues, Börner has analyzed and “visualized” the network of knowledge that is Wikipedia in English.The image on the right is a science-themed version of the same visualization, created using science-related Wikipedian activity. To construct it, the team laid out a sample of article images on a circular grid, positioning similar, linked articles close together. The result is a mo