World’s fastest supercomputer at ORNL more about research than rankings

The No. 1 ranking for the new Titan supercomputer, designating it as the most powerful in the world, was clearly appreciated in East Tennessee on Monday.

But even as they celebrated a return to the top spot, scientists at Oak Ridge National Laboratory, where the giant computer is based, said they are focused more on research than rankings.

“We love being No. 1,” said Bronson Messer, acting group leader for scientific computing at the National Center for Computational Science at ORNL. “It’s great recognition. But what really matters is what science will do with the machine.”

Titan received the first-place ranking in a semiannual Top500 list that was released Monday at the SC12 supercomputing conference in Salt Lake City, Utah. The ranking program uses a series of linear equations to test computer systems around the world. The test showed Titan, a Cray XK7 system, is capable of reaching a speed of 17.59 petaflops, or more than 17,000 trillion calculations per second.

“The new Top500 list clearly demonstrates the U.S. commitment to applying high-performance computing to breakthrough science, and that’s our focus at Oak Ridge,” ORNL Director Thom Mason said. “We’ll deliver science from Day One with Titan, and I look forward to the advancements the Titan team will make in areas such as materials research, nuclear energy, combustion, and climate science.”

Messer said acceptance testing on the $100 million machine could be complete by the end of February.

As big as a basketball court, Titan is 10 times faster than Jaguar, the computer system it replaced. Jaguar, which was capable of about 2.5 petaflops, had ranked as the world’s fastest computer in November 2009 and June 2010.

Titan is theoretically capable of 27 petaflops, or 27 quadrillion calculations per second.

“We’re building machines that are really pushing the envelope,” Mason said. “Titan is absolutely going to be outstanding.”

An ORNL press release said scientists began using parts of Titan while it was still under construction for research in materials science, fuel combustion, and nuclear power.

Titan is a hybrid system that combines traditional central processing units, or CPUs, with graphic processing units, or GPUs, which were first created for computer games. The combination allows the more powerful Titan to fit into the same space as Jaguar while using only slightly more electricity.

That’s important because supercomputers can consume megawatts of power.

“It’s not practical or affordable to continue increasing supercomputing capacity with traditional CPU-only architecture,” said Jeff Nichols, ORNL associate laboratory director for computing and computational sciences. “Combining GPUs and CPUs is a responsible move toward lowering our carbon footprint, and Titan will enable scientific leadership by providing unprecedented computing power for research in energy, climate change, materials, and other disciplines.”

Buddy Bland, Titan project director at the Oak Ridge Leadership Computing Facility, said the new supercomputer uses about nine megawatts of electricity—roughly the amount required for 9,000 homes—compared to approximately seven megawatts for Jaguar.

But the big performance boost “will allow U.S. scientists and industry to address problems they could only dream of tackling before,” Bland said.

Titan was an upgrade to Jaguar and re-used about $20 million worth of equipment, including cabinets, cooling systems, and cables.

ORNL officials gave local media a tour of Titan on Monday. The system includes 200 black air-conditioned, water-cooled cabinets each packed with 24 gray metal “blades” roughly the size of a pizza box. Each blade has four nodes, and each node has access to a shared networking chip and is built from a 16-core AMD processor and an NVIDIA GPU accelerator.

The supercomputer uses one GPU for each CPU. The GPUs are more powerful and energy-efficient than CPUs, but the GPUs can only solve math equations and can’t perform other functions such as communicating with the Internet or writing files, so they need CPUs, Bland said.

He said anyone can apply to use Titan under the U.S. Department of Energy’s Innovative and Novel Computational Impact on Theory and Experiment program, or INCITE. However, proposals are peer-reviewed to make sure that applicants know how to use the computer effectively and to decide which projects make the most sense and are most important for the country. Most users don’t pay, but businesses do if they want to keep the results proprietary.

Bland said there are about 50 large projects that will use the machine next year, with more than half of those from universities. Of the other half, most are from DOE, and the rest are from other federal agencies and industry.

Titan’s move to No. 1 on the Top500 list on Monday pushed the Sequoia supercomputer at Lawrence Livermore National Laboratory down one spot from No. 1 to No. 2. That IBM BlueGene/Q system, which will be used for classified nuclear stockpile security work, had achieved 16.32 petaflops.

Other systems in the Top 5 were Fujitsu’s K computer at the RIKEN Advanced Institute for Computational Science in Kobe, Japan, which was No. 3; a BlueGene/Q system named Mira at Argonne National Laboratory near Chicago at No. 4; and a BlueGene/Q system named JUQUEEN at the Forschungszentrum Juelich in Germany, which placed fifth. It has been upgraded and is now the most powerful system in Europe.

A Top500 press release said there are now 23 systems with petaflop performance on the latest list, just 4.5 years after the debut of Roadrunner, the world’s first petaflop supercomputer.

The TOP500 list is compiled by Hans Meuer of the University of Mannheim, Germany; Erich Strohmaier and Horst Simon of Lawrence Berkeley National Laboratory; and Jack Dongarra, a distinguished professor in the Electrical Engineering and Computer Science Department and the director of the Innovative Computing Laboratory at the University of Tennessee in Knoxville.

Bland said the supercomputers can help make economically significant predictions, helping scientists understand such things as the impacts of carbon dioxide emissions on ice sheets and sea levels, how to burn materials efficiently, and methods to produce ethanol from switchgrass or poplar trees.

He said a trucking company used Jaguar to design air foils to help tractor-trailers meet California fuel economy standards. Virtual wind tunnel testing on Jaguar predicted the results of actual performance tests within 1 percent, Bland said, cutting the design and testing phase in half and allowing the company to bring its product to market ahead of its competitors.

Although it’s primarily an important scientific project, the high-performance computing work at ORNL helps East Tennessee in other ways, Bland said, keeping the lab economically competitive, bringing in high-speed Internet connections, and providing good high-paying jobs.

“The economic impact to East Tennessee is enormous,” Bland said.

He doesn’t expect Titan to be around for a long time. Supercomputers typically last four to five years, Bland said. In five years, he said, DOE will likely be able to buy and run a new supercomputer that costs less to operate than it will cost to continue operating Titan.

Mason said Titan’s No. 1 ranking is at least the fourth first-place ranking for an ORNL computer, although benchmarking tests haven’t always been available.

Read the ORNL press release here.