Mapping the stars — Gaia's big data challenge

Yesterday, the European Space Agency's Gaia satellite fired its thrusters to complete the process of bringing it into a stable orbit around the L2 Lagrange point. From this point, approximately 1,500,000km from the Earth's surface, the satellite will survey over one billion stars over a five-year period. The mission will enable the creation of the most detailed ever 3D map of our galaxy and will provide new tests of Albert Einstein’s general theory of relativity.

While the first data from Gaia won't be made public for almost two years, dealing with the data from the mission presents a significant challenge. Over its lifetime Gaia will send over 100 terabytes of raw data back to Earth. This data will be distributed via high-speed cables to the European Space Astronomy Centre, after being stored in dedicated science data servers at the European Space Operations Centre.

A pan-European team of expert scientists and software developers, known as the Gaia Data Processing and Analysis Consortium, or DPAC, will handle the data. They have set up a network of 30 centres, staffed by 450 people, to crunch the raw data, including a 6-teraFLOPS supercomputer at the French National Centre for Space Studies in Toulouse.  Nevertheless, it will take years to transform the million billion bytes of input into a catalog with useable 3D maps. This catalog is expected to reach over one petabyte, which is equivalent to around 200,000 DVDs.

- Andrew Purcell