Expanding infrastructure for e-science in Brazil

In 2009, São Paulo State University (Universidade Estadual Paulista, or UNESP), in Brazil, began operating GridUNESP, one of the largest multi-campus grid infrastructures in Latin America fully dedicated to scientific research. With 23 campuses distributed throughout São Paulo, UNESP is the second largest university in Brazil. A research group led by Sergio Novaes formed the Center for Scientific Computing (CSC), which now operates two main clusters: São Paulo Research and Analysis Center (SPRACE) — dedicated to CERN; and GridUNESP — dedicated to university researchers and students. GridUNESP was a spinoff of SPRACE, which provides computation and storage for the data produced by the Compact Muon Solenoid experiment at CERN.

GridUNESP is designed to take advantage of the distributed scientific research at UNESP. The grid consists of a central cluster located in São Paulo city, with seven secondary clusters at different UNESP campuses spread over the state. The eight sites are connected through the KyaTera research and education network for São Paulo, provided by FAPESP (Fundação de Amparo à Pesquisa do Estado de São Paulo) in partnership with Telefonica Brazil. GridUNESP is also connected to the US Internet2 Network through the Americas Lightpaths (AmLight) links.

The number of researchers using GridUNESP and its resources has been growing. In its four years of operation, GridUNESP has processed 6 million jobs and logged 46.5 million CPU hours. Outside researchers affiliated with or connected to UNESP researchers can also use the grid. A major upgrade of the GridUNESP physical infrastructure is planned for 2014.

“This year is our first major GridUNESP upgrade,” says Rogério Iope, a CSC systems engineer who began working with SPRACE in 2005. “We have a budget of around $1 million to buy new hardware, install a new no-break system, and upgrade our network link, allowing us to reach speeds of up to 100 gigabits per second. This will also enable us to reach the US networks such as Internet2 — and thus US Open Science Grid (OSG) resources — at 100 gigabits per second.”

A new web portal is facilitating job submissions to GridUNESP. With development led by UNESP CSC team members Gabriel Winckler and Beraldo Costa, the portal offers an easy-to-use platform for scientists to manage the execution of computer simulations. Providing a modern web interface, it enables researchers to explore all of the computing power of the grid with just a few clicks from a computer, tablet, or even a smartphone.

The platform also incorporates pre-installed applications related to a variety of scientific areas, so a researcher without computing expertise can access all the resources. Conversely, it provides command-line tools and workflow execution for more experienced users. The portal is extensible through a RESTful API, and all code is available as open-source (GPL3 license).

Another important milestone was establishing a GridUNESP certificate authority. Together with ANSP (Academic Network of São Paulo), the UNESP CSC team created a certificate authority for the state of São Paulo. According to Sergio Lietti, the team member responsible for the deployment, ANSPGridCA (ANSP Grid Certification Authority) began issuing GridUNESP certificates in March 2013. Following nine months of testing, certificate production is now open to the entire academic community of São Paulo and researchers no longer need to rely on a US certificate authority.

“There is also a new thematic project being prepared to be submitted to FAPESP,” Iope says. “Our main partner — Federal University of São Carlos — is leading the project and is in the same state. Our aim is to build infrastructure for e-Science in the state. We are planning to deploy distributed infrastructure at different universities, starting with UNESP and Federal University of São Carlos and in the long term involving other institutions. The idea is to work on cloud infrastructure that has integration between clouds and with grids, so we should be able to run OSG middleware on top of it. That would open opportunities statewide.”

From the very beginning, GridUNESP established a formal partnership with the OSG. The GridUNESP VO (virtual organization) was established in December 2009, as the first OSG VO located outside of the US. As a multipurpose VO, it includes projects from astronomy, biology and biophysics, biomedical engineering, chemistry, computer science, the geosciences, materials science, meteorology, and physics. This partnership enables GridUNESP to use the OSG middleware stack to integrate its computational resources and share them with other institutions.

In his 2014 GridUNESP VO annual report to the OSG, Iope writes that GridUNESP VO resource usage — measured against the mean occupancy of computing resources — has grown from 19% in 2010 to 53% in 2011, 74% in 2012, and a peak of 89% in 2013.

“The OSG has been a good opportunity for our researchers to have contact with the concept of grid computing,” adds Iope. “At one time, each researcher had their own small cluster. The idea of deploying the grid showed researchers that it is possible to run jobs from a central system to distributed resources that may be spread out. Since UNESP is the most distributed university in Brazil, it was ideal to show that distributed infrastructure can be tied together.”

After establishing the formal partnership with the OSG in 2009, GridUNESP researchers started using the OSG middleware stack and sharing resources with other US institutions. GridUNESP can run jobs from several different US VOs. “We have had a very strong partnership with OSG,” says Iope. “In 2010, we organized the São Paulo OSG School with the full support of the OSG education, outreach, and training team, and brought five OSG experts to Brazil. Our attendees learned how to maximize their research using grid resources.”

By 2011, local users started submitting more jobs. GridUNESP now sees more local usage than jobs coming from outside. When resources are free, however, OSG users from anywhere in the US can use them. “The upgrade that is happening this year,” says Iope, “will bring about increased usage and even more partnership with the OSG.”

Iope is one of the main collaborators on the GridUNESP upgrade and is deeply involved in the e-Science project. He will be responsible for building the education and training infrastructure for new e-Science users and has been busy formulating ideas. He plans to use HUBzero, an open-source platform developed at Purdue University in Indiana, US, to enable scientists and researchers to create dynamic websites that support their activities.

“In the e-Science project, we will need to provide dynamic content for researchers and for teachers,” Iope says. “This is my main involvement at the moment. We are also working closely with Intel in Brazil to deploy a Manycore Testing Lab (MTL). We will be deploying the first MTL outside of the US. It will be tied to building an educational platform to use for teaching grad students about parallel computing.”