University of Michigan and NVIDIA Integrate Software to Cut AI Training Power Use by Up to 50%

Software developed at University of Michigan Engineering can reduce the energy needed to train artificial intelligence models by up to 50 percent without compromising training speed or performance, according to a report from TechXplore . Researchers from the university are now collaborating with NVIDIA to integrate the optimization tools into Megatron-LM, NVIDIA's open-source framework for large language model training.

Megatron-LM serves as both a direct training framework and a base for proprietary pipelines, including the infrastructure NVIDIA uses to train its Nemotron models. The optimization addresses energy waste inherent in parallel processing, where training workloads are partitioned across hundreds or thousands of GPUs. In standard training runs, differences in workload distribution cause faster GPUs to finish early and idle at elevated power states while waiting for the slowest processor to complete its cycle.

The Michigan framework resolves these bottlenecks in two distinct stages. The first tool, Perseus, slows down processors that are outpacing others so that all GPUs finish simultaneously, reducing training energy use by up to 30 percent according to a 2024 study. The second tool, Kareus, coordinates processor cores and kernel execution schedules within each GPU to resolve memory and bandwidth inefficiencies introduced by altered clock speeds.

In research presented at the USENIX Symposium on Operating Systems Design and Implementation in July, Kareus reduced training energy for two leading open-source models by an additional 10 to 30 percent, bringing combined potential energy savings to 40 to 50 percent for large-scale models.

"AI is becoming power-constrained as models become larger and more people want to train models, but we can achieve the same training speed and performance using less energy," said Ruofan Wu, a doctoral student in computer science and engineering at U-M who led Kareus's development and spent the summer integrating the tools into Megatron-LM.

Mosharaf Chowdhury, associate professor of computer science and engineering at U-M, noted that while the team is currently deploying the software with NVIDIA, the underlying parallelization and scheduling principles apply universally across distributed AI training frameworks.