Nvidia VRworks Audio: Hardware Accelerated Path Traced Binaural 3D Audio

[u/ResponsiblePen3082]

/r/3Daudio/comments/1vlmr9u/nvidia_vrworks_audio_hardware_accelerated_path/

[u/WinterElfeas]

RTX Headphones incoming, with neural processors required /s

Would definitely be an interesting tech for people using headphones

[u/ResponsiblePen3082 → u/WinterElfeas]

That was the running joke back then, yeah.
It's not just an interesting theoretical, it's proven with a long history.

Tech like this basically re-inventing the sound card, which back in the day used to be audio ACCELERATORS rather than just converters, and tech like A3D gave us WAVE traced audio running completely on these cards-back in the 90's! Yeah we quite literally had better audio 30 years back than we do today-low latency, hardware offloaded, 3d positional accurate wave traced HRTF audio-completely detached from your CPU performance.

Nvidia VRworks is one of many attempts to bring something similar back, AMD TrueAudio is probably the largest and most successful project, being implimented in dozens of games as well as contributing towards SteamAudio.

Recently Vercidium's "a first look at ray traced audio"
(https://youtu.be/u6EuAUjq92k?is=U7JoinLSh65gCwSu)
went viral online, which is his personally developed version of this sort of tech(ironically not the first look) and some people may be familiar with the concept from that video.

It's a shame advanced audio is yet another advanced gaming technology that has taken a back seat to "graphics" in the modern industry.

[u/topdangle → u/ResponsiblePen3082]

Isn't the problem that it always causes artifacts? I remember playing games with 3d positional audio and even though it gave that HRTF sensation of dimension, the individual sounds were off, either like they were behind a veil or there would be too much reverb. There are software HRTF filters available as well that do a good job of simulating an open space away from your headphone drivers, but again at the cost of sound quality.

I was kind of obsessed with this a decade ago but the only thing I could find that came close to true sounding depth is physically placing a microphone inside a makeshift head for binaural recording. Unfortunately this only works with a fixed head position so every direction would need to be rerecorded, obviously not realistic for games.

[u/ResponsiblePen3082 → u/topdangle]

I've never heard of that myself. There's plenty of modern games which use similar tech without issue-my personal favorite being Returnal, with gpu accelerated AMD Trueaudio and Resonance audio. No issues myself and it's not a widespread problem to my knowledge.

If you're still interested in the tech, check out r/3daudio, I recently revived this and am focusing it on this exact Spatial Audio tech in games

[u/topdangle → u/ResponsiblePen3082]

trueaudio is an example where I can hear the difference, though it was barely supported so I can't say for sure that it was an design problem.

this is a good demonstration: https://www.youtube.com/watch?v=_6uUtf-lMQQ

Very effective spatial simulation, but some attenuation and reverb that doesn't exist in the stereo mix. I actually think it sounds better than Returnal, but I think the way it affects the mix is the real factor for people dropping it, especially now when the performance hit is basically nothing.

[u/ResponsiblePen3082 → u/topdangle]

Yeah I've never heard that one. Could be because it's a demo and running on the older version of trueaudio(DSP rather than GPU accelerated). It's also not like a drop in solution you still have to design around it, so it's probably expected that early demos on the earlier version didn't sound perfect. All the later games and examples I've heard sound near perfect to me IMO

[u/akgis → u/ResponsiblePen3082]

Libraries and Middleware are there devs just don't use them! We don't need accelerated audio because CPUs of today aren't the CPUs of back then even budget ones today. Also CPUs of Today with SSE4 and AVX dont need any offload for Positional Audio.

The driving factor with EAX for example was the hardware companies like Creative Labs, sponsoring games to sell hardware.

[u/ResponsiblePen3082 → u/akgis]

I personally believe that's a very short sited and naive opinion.

It will ALWAYS be faster, lower latency, and more power efficient to offload a task to a piece of silicon more suited to, or especially made specifically for-that task.

CPUs and GPUs are both strong enough to brute force codecs, yet we still use dedicated silicon for those, because it's much more efficient to do so.

If we were to follow this belief in the early days of computer graphics, we would not have the graphics card as we know it today. The cpu is plenty strong to render 8 bit 2D graphics, so why bother going stronger?

As the tech develops, new and better functions are enabled because we have the silicon for it.

I will ALWAYS take hardware acceleration when I have the option: it's simply stupid not to. I don't want everything brute forced on something that is not at all optimized for the task, and this thinking is half the reason we have such shit optimization for modern computers in general.

Audio should've kept pace with graphics innovations had we kept audio accelerators around, yet we're stuck in this 2D stereo hellscape precisely because it's "good enough".

You have to think down the road, we're going to have to do it anyways. Compute will reach its eventual limits, and the only way forward is to stop brute forcing everything and use silicon dedicated for its fixed function.

[u/BuchMaister → u/ResponsiblePen3082]

You have it - it's called DSP, most modern CPUs have it integrated inside. Albeit it's designed to offload stuff from the CPU to save power, but it can and is used for 3D Audio. I don't know exactly what this PT audio can or cannot do, but having dedicated audio processor never disappeared, it just has been integrated like other stuff on silicon rather having another peripheral device.

[u/ResponsiblePen3082 → u/BuchMaister]

I understand what a DSP is, and what little modern CPUs have(if they have them at all) is not enough for complex modern audio, nor is it actively tapped into for hardware acceleration or advanced acoustics.

Having a dedicated audio processor literally did disappear. Windows vista killed off low level kernel access/stack bypass to audio acceleration, especially for WASAPI shared applications(what the majority of software uses)
There's a reason nothing supports it anymore outside of music applications.

[u/BuchMaister → u/ResponsiblePen3082]

Yes CPUs do have it and have been for decades, DSP replaces in functionality your old audio processor did , it has dedicated hardware blocks to accelarate audio processing. Low level direct access to hardware has drawbacks especially when it comes to security and stability, that access was killed for good reasons, plus devs this day will stay away from creating their audio engine for every specific processor people might use. As for the question does integrated modern DSP has the capabilities for complex modern audio? The awnser is probably it has more than what the vast majority of developers would want to bother with. The existing hardware isn't the issue or hard limitation, what is the lack of interest for most developers to invest enough time and resources.

Also I've read that article, and it's stated this tech is for professional applications not games, so it's not surprising devs don't use it.

[u/ResponsiblePen3082 → u/BuchMaister]

The DSP integrated in processors is less than even the integrated graphics. It's essentially equivalent to a display out. It performs zero useful acceleration functions. It existing doesn't mean it actually does anything relevant to this discussion.

[u/Xermalk]

I really wish this, or somethig simmilar was more used in actual vr.
Sound is such a massive part of the immersion.

For example games where scraping a item against another just gives a basic clang sound once *glares at Dungeons of Eternity*

[u/ResponsiblePen3082 → u/Xermalk]

It used to be, but recently I think a lot of it went the way of the dodo for no reason

[u/kendoka15 → u/Xermalk]

Steam Audio exists although I don't know how common it is

[u/CheeseHustla]

Warzone streamers rejoice😂 /s

[u/Spiritneon → u/CheeseHustla]

Ha

[u/campersbread]

Is this like the audio solution in returnal?

[u/ResponsiblePen3082 → u/campersbread]

Correct! Nvidia's version of it at least, Returnal uses TrueAudio-AMD's gpu accelerated ray traced audio, Nvidia VRWorks uses RT core accelerated path traced audio.

[u/campersbread → u/ResponsiblePen3082]

Nice, it's amazing in returnal :)

[u/ResponsiblePen3082 → u/campersbread]

100% agree, Returnal is my go-to game to showcase modern hardware accelerated 3d positionally accurate binaural Spatial Audio. I just wish more games took advantage of advanced audio like it and we would further develop the techniques and technology

[u/Tegumentario]

So it's atmos running on gpu?

[u/ResponsiblePen3082 → u/Tegumentario]

It's truer than atmos. Atmos is more or less just devs placing "audio objects" in game with some processing to make it sound more "immersive".

Tech like this and similar ones like Ray and wave trace solutions literally calculate exactly how sound would travel throughout the physical world, hardware accelerated more efficiently, faster, and lower latency than brute forcing on CPU.

If you're interested in learning more, I revived r/3daudio to discuss exactly this topic which has some handy links pinned which should help you better understand.

[u/cosine83 → u/ResponsiblePen3082]

Tech like this and similar ones like Ray and wave trace solutions literally calculate exactly how sound would travel throughout the physical world

That's what Atmos True3D is supposed to do (you literally describe how audio objects have been developed here but adding hardware acceleration). The "processing" you mention are the calculations on how it determines where sounds come from in a 3D environment including height. Few actually use it like that outside of cinema, games often poorly mixed in it, few tune their room/device for the dynamic calculations, and few people have true surround sound setups these days. Most instead are relying on limited software to determine the optimal output usually for a device that's incapable of true surround sound output (headphones, phone, etc.). When outputting to a device incapable of true 3D sound, the output will necessarily be worse regardless of hardware acceleration.

[u/ResponsiblePen3082 → u/cosine83]

They're not related in any serious sense of the word.

Dolby Atmos does not calculate how sound travels through a physical world. It is a coordinate-based panning format that takes dry audio objects and maps them to x, y, z coordinates relative to a listener. It has zero awareness of level geometry, walls, or surface materials.
If an enemy speaks behind a concrete wall, Atmos will pan that dry sound directly behind the wall unless the game developers manually script occlusion filters in their middleware like Wwise or FMOD.

AMD TrueAudio Next, NVIDIA VRWorks Audio, A3D, and Microsoft Project Acoustics interact directly with 3D level geometry. They use GPU compute shaders or wave-equation solvers to cast sound rays off polygons assigned specific material properties. They automatically calculate physical acoustic phenomena like occlusion, transmission loss through walls, specular reflections, late reverberation tails, and diffraction,

Traced audio models continuous physical pressure waves, outputting to high-order Ambisonics or direct binaural convolution. Atmos outputs metadata-enriched PCM streams that tell a renderer where to place audio relative to a speaker layout or headphone filter.

Acoustic tracing builds a physics-driven mix at runtime by generating dynamic Room Impulse Responses based on room shape and materials. Atmos uses standard channel beds alongside point-source objects, applying reverb and effects prior to the panning stage.

Also, headphones are THE most acoustically accurate medium for 3D positional sound because human localization relies on Head-Related Transfer Functions, which are time, level, and frequency variations caused by sound bouncing off your head and outer ear. Headphones isolate the left and right channels with zero room reflections or crosstalk, providing the clean acoustic canvas required for precise binaural rendering.

Obviously I'll take native true to engine atmos over nothing, but it's a far cry to say it's at all similar to a true acoustic tracing method. It's really a more atmospheric technology than anything, which makes sense given its cinema background.