Nvidia has offered more details on the new features coming to its GPUs thanks to the release of DLSS 4, as well as how some of its technologies, such as its new transformer models and enhanced Frame Generation. In an interview with Digital Foundry, Bryan Catanzaro, vice president of applied deep learning research at Nvidia, spoke about the technology.
Speaking on Frame Generation, Catanzaro explained how Nvidia’s Optical Flow accelerator was used. The technology, originally used for Nvidia’s automotive division’s research into self-driving cars, is an evolution of the company’s video encoding algorithms. However, Optical Flow was difficult to improve, so Nvidia decided to switch things up with a fully AI-based solution for DLSS 4’s take on frame generation.
“When we built Nvidia DLSS 3 Frame Generation, we absolutely needed hardware acceleration to compute Optical Flow,” explained Catanzaro. “We didn’t have enough Tensor Cores, and we didn’t have an Optical Flow algorithm that was good enough.
“We hadn’t developed a real-time Optical Flow algorithm that ran on Tensor Cores that could fit our compute budget. We had the Optical Flow accelerator, which Nvidia had been building for years as an evolution of our video encoder technology, and it’s also been a part of our automotive computer vision acceleration for self-driving cars.”
“It made sense for us to use that for Nvidia DLSS 3 Frame Generation. But the difficult part about any sort of hardware implementation of an algorithm like Optical Flow is that it’s really difficult to improve it. It is kind of what it is, and the failures that arose from that hardware Optical Flow, we couldn’t undo them with a smarter neural network until we decided to just replace it and go with a fully AI-based solution, so that’s what we’ve done for Frame Generation in DLSS 4.”
With the new transformer model, the AI used for frame generation can learn things much quicker while simultaneously also being less heavy on the GPU’s hardware. While Nvidia hasn’t confirmed the release of new DLSS 4 technologies for older GPUs from the RTX 30 series, Catanzaro does mention that the possibility is there.
“I think this is primarily a question of optimization and also engineering and then the ultimate user experience,” said Catanzaro when asked if we could see the new transformer-based frame generation in RTX 30-series graphics cards. “We’re launching this Frame Generation, the best Multi Frame Generation technology, with the 50 Series, and we’ll see what we’re able to squeeze out of older hardware in the future.”
Nvidia unveiled DLSS 4 alongside its new GeForce RTX 50-series graphics cards during CES 2025 earlier this month. Among other things, DLSS 4 will make use of more efficient algorithms to provide features like super resolution and frame generation. The upcoming RTX 50-series cards will also have multi-frame generation as a feature, allowing for three extra frames for every single frame to be generated.
DLSS 4, with its more efficient features, will be available for Nvidia GPU users through a software update.
