The Oak Ridge National Laboratory launched a new era of scientific supercomputing today with Titan, a system capable of churning through more than 20,000 trillion calculations each second—or 20 petaflops—by employing a family of processors called graphic processing units first created for computer gaming. Titan will be 10 times more powerful than ORNL’s last world-leading system, Jaguar, while overcoming power and space limitations inherent in the previous generation of high-performance computers.
Titan, which is supported by the U.S. Department of Energy, will provide unprecedented computing power for research in energy, climate change, efficient engines, materials, and other disciplines, and pave the way for a wide range of achievements in science and technology.
The Cray XK7 system contains 18,688 nodes, with each holding a 16-core AMD Opteron 6274 processor and an NVIDIA Tesla K20 graphics processing unit, or GPU, accelerator. Titan also has more than 700 terabytes of memory. The combination of central processing units, the traditional foundation of high-performance computers, and more recent GPUs will allow Titan to occupy the same space as its Jaguar predecessor while using only marginally more electricity.
“One challenge in supercomputers today is power consumption,” said Jeff Nichols, associate laboratory director for computing and computational sciences. “Combining GPUs and CPUs in a single system requires less power than CPUs alone and is a responsible move toward lowering our carbon footprint. Titan will provide unprecedented computing power for research in energy, climate change, materials and other disciplines to enable scientific leadership.”
Because they handle hundreds of calculations simultaneously, GPUs can go through many more than CPUs in a given time. By relying on its 299,008 CPU cores to guide simulations and allowing its new NVIDIA GPUs to do the heavy lifting, Titan will enable researchers to run scientific calculations with greater speed and accuracy.
“Titan will allow scientists to simulate physical systems more realistically and in far greater detail,” said James Hack, director of ORNL’s National Center for Computational Sciences. “The improvements in simulation fidelity will accelerate progress in a wide range of research areas such as alternative energy and energy efficiency, the identification and development of novel and useful materials and the opportunity for more advanced climate projections.”
Titan will be open to select projects while ORNL and Cray work through the process for final system acceptance. The lion’s share of access to Titan in the coming year will come from the Department of Energy’s Innovative and Novel Computational Impact on Theory and Experiment program, better known as INCITE.
Researchers have been preparing for Titan and its hybrid architecture for the past two years, with many ready to make the most of the system on day one. Among the flagship scientific applications on Titan:
Materials Science
The magnetic properties of materials hold the key to major advances in technology. The application WL-LSMS provides a nanoscale analysis of important materials such as steels, iron-nickel alloys and advanced permanent magnets that will help drive future electric motors and generators. Titan will allow researchers to improve the calculations of a material’s magnetic states as they vary by temperature.
“The order-of-magnitude increase in computational power available with Titan will allow us to investigate even more realistic models with better accuracy,” noted ORNL researcher and WL-LSMS developer Markus Eisenbach.
Combustion
The S3D application models the underlying turbulent combustion of fuels in an internal combustion engine. This line of research is critical to the American energy economy, given that three-quarters of the fossil fuel used in the United States goes to powering cars and trucks, which produce one-quarter of the country’s greenhouse gases.
Titan will allow researchers to model large-molecule hydrocarbon fuels such as the gasoline surrogate isooctane; commercially important oxygenated alcohols such as ethanol and butanol; and biofuel surrogates that blend methyl butanoate, methyl decanoate and n-heptane.
“In particular, these simulations will enable us to understand the complexities associated with strong coupling between fuel chemistry and turbulence at low preignition temperatures,” noted team member Jacqueline Chen of Sandia National Laboratories. “These complexities pose challenges, but also opportunities, as the strong sensitivities to both the fuel chemistry and to the fluid flows provide multiple control options which may lead to the design of a high-efficiency, low-emission, optimally combined engine-fuel system.”
Nuclear Energy
Nuclear researchers use the Denovo application to, among other things, model the behavior of neutrons in a nuclear power reactor. America’s aging nuclear power plants provide about a fifth of the country’s electricity, and Denovo will help them extend their operating lives while ensuring safety. Titan will allow Denovo to simulate a fuel rod through one round of use in a reactor core in 13 hours; this job took 60 hours on the Jaguar system.
Climate Change
The Community Atmosphere Model-Spectral Element simulates long-term global climate. Improved atmospheric modeling under Titan will help researchers better understand future air quality as well as the effect of particles suspended in the air.
Using a grid of 14-kilometer cells, the new system will be able to simulate from one to five years per day of computing time, up from the three months or so that Jaguar was able to churn through in a day.
“As scientists are asked to answer not only whether the climate is changing but where and how, the workload for global climate models must grow dramatically,” noted CAM-SE team member Kate Evans of ORNL. “Titan will help us address the complexity that will be required in such models.”
ORNL is managed by UT-Battelle for the Department of Energy. The Department of Energy is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit http://science.energy.gov.
TJ says
Is this the super computer funded by NSA for decryption of citizen communications?
TJ says
So the agency had one major ingredient—a massive data storage facility—under way. Meanwhile, across the country in Tennessee, the government was working in utmost secrecy on the other vital element: the most powerful computer the world has ever known.
The plan was launched in 2004 as a modern-day Manhattan Project. Dubbed the High Productivity Computing Systems program, its goal was to advance computer speed a thousandfold, creating a machine that could execute a quadrillion (1015) operations a second, known as a petaflop—the computer equivalent of breaking the land speed record. And as with the Manhattan Project, the venue chosen for the supercomputing program was the town of Oak Ridge in eastern Tennessee, a rural area where sharp ridges give way to low, scattered hills, and the southwestward-flowing Clinch River bends sharply to the southeast. About 25 miles from Knoxville, it is the “secret city†where uranium- 235 was extracted for the first atomic bomb. A sign near the exit read: what you see here, what you do here, what you hear here, when you leave here, let it stay here. Today, not far from where that sign stood, Oak Ridge is home to the Department of Energy’s Oak Ridge National Laboratory, and it’s engaged in a new secret war. But this time, instead of a bomb of almost unimaginable power, the weapon is a computer of almost unimaginable speed.
In 2004, as part of the supercomputing program, the Department of Energy established its Oak Ridge Leadership Computing Facility for multiple agencies to join forces on the project. But in reality there would be two tracks, one unclassified, in which all of the scientific work would be public, and another top-secret, in which the NSA could pursue its own computer covertly. “For our purposes, they had to create a separate facility,†says a former senior NSA computer expert who worked on the project and is still associated with the agency. (He is one of three sources who described the program.) It was an expensive undertaking, but one the NSA was desperate to launch.
Known as the Multiprogram Research Facility, or Building 5300, the $41 million, five-story, 214,000-square-foot structure was built on a plot of land on the lab’s East Campus and completed in 2006. Behind the brick walls and green-tinted windows, 318 scientists, computer engineers, and other staff work in secret on the cryptanalytic applications of high-speed computing and other classified projects. The supercomputer center was named in honor of George R. Cotter, the NSA’s now-retired chief scientist and head of its information technology program. Not that you’d know it. “There’s no sign on the door,†says the ex-NSA computer expert.
At the DOE’s unclassified center at Oak Ridge, work progressed at a furious pace, although it was a one-way street when it came to cooperation with the closemouthed people in Building 5300. Nevertheless, the unclassified team had its Cray XT4 supercomputer upgraded to a warehouse-sized XT5. Named Jaguar for its speed, it clocked in at 1.75 petaflops, officially becoming the world’s fastest computer in 2009.