For everything we hear about graphics, a fair amount of work also goes into the game’s sound. This is an understatement but games in general can live or die by the quality of audio they offer. It’s not always about creating the best SFX or soundtrack though. There has to be an infrastructure capable of handling these sounds and playing them back in 3D. Firelight Technologies handles this with FMOD but has since grown to creating interactive events and music while also facilitating connections between the game and the audio data being created. Such tools are all the more important in the current generation of consoles like the Xbox One and PS4 which boast significantly more powerful audio hardware than their predecessors.
GamingBolt spoke to Firelight CEO Brett Paterson about FMOD and the tools it offered along with its future potential. How does FMOD take advantage of technology like the Xbox One’s on-board audio processor? We find out below, along with other details.
Rashid K. Sayed: To begin with, can you tell us about yourself and FMOD?
Brett Paterson: My name is Brett Paterson, and I am the CEO of Firelight Technologies and original developer on FMOD, which has been available for over 10 years now. FMOD started out as a simple programmer interface to load sounds and play them in 3D, but since then the product (and company) has grown, and FMOD has transformed into a high end, professional tool suite called FMOD Studio, which includes a Digital Audio Workstation style interface (think Ableton, Logic, Pro Tools) for creating interactive sound events and music, and a runtime component which you plug into your game called the FMOD Studio runtime API, which is what the programmers use to link up the game to the audio data.
"FMOD Studio currently uses the audio processor on the Xbox One for two things: XMA decoding and sample rate conversion. This is one of the most expensive parts of the process when processing audio in real-time...
Rashid K. Sayed: Can you briefly talk about the various FMOD Studio Plugins?
Brett Paterson: FMOD Studio now ships with 3 plugins from pro-audio companies typically known for their plugins in the music industry. We have McDSP’s ML1 (Master Limiter) filter for prolevel mastering of the game’s audio mix, and AudioGaming’s synthesis plugin for real-time generation of wind and rain effects. Lately we also added support for GenAudio’s 3D virtualization plugin for headphones called AstoundSound. This lets you get much better spatialization of sound than the default 3D panner within FMOD. We will gradually be adding more cool and interesting plugins like this over time.
Rashid K. Sayed: Tell us what kind of updates have you made to the middleware to support the new consoles PS4 and Xbox One?
Brett Paterson: FMOD Studio has added native support for hardware accelerated decompression of XMA data on Xbox One and ATRAC9 and resampling support on PlayStation 4, which allows more sounds to be played at once. We also made some updates to support recording and peripherals such as headsets to allow recording and playback through external devices. Otherwise the core feature set / functionality is the same for all platforms.
Rashid K. Sayed: From architecture point of view, how deeply is FMOD involved? Do you use the GPU at all or is the process all CPU based?
Brett Paterson: FMOD Studio does all of its mixing and signal processing on the CPU, which is a departure from the previous FMOD Ex engine. It allows us total flexibility and control over the signal, and avoids the limitations of dedicated hardware such as lack of cross-platform support. We haven’t put any processing onto GPUs yet, though we do support AMD’s TrueAudio hardware which is GPU based, and PC only at the moment.
Rashid K. Sayed: How does FMOD takes advantage of the on board audio processor on the Xbox One? Since it has a dedicated audio CPU, does FMOD make a difference in such a scenario?
Brett Paterson: FMOD Studio currently uses the audio processor on the Xbox One for two things: XMA decoding and sample rate conversion. This is one of the most expensive parts of the process when processing audio in real-time, so it takes a significant load off the CPU and allows for more effects like reverb, EQ filters and other filters.
Rashid K. Sayed: The PlayStation 4 does not have a dedicated audio processor. Does it make FMOD implementation a bit different or harder from the Xbox One?
Brett Paterson: On PlayStation 4 there is hardware to allow decoding of ATRAC compressed audio, so it is very similar to the Xbox One in regards to offloading decoding from the CPU.
"The cloud is great for sending tasks and data offline and waiting for it to come back at a later time, but audio is typically a very low latency process, which means we haven’t thought much about it yet.
Rashid K. Sayed: I understand that the middleware will have compatibility with Unity but what about Unreal Engine 4?
Brett Paterson: Our Unity engine integration for FMOD Studio is already on our website and ready to download. We also have a working UE4 integration that is one of the first middleware integrations that use the UE4 plugin system. Epic has invited us to speak about how we did this at their room at the GameConnect Australia Pacific conference in Melbourne this month. The UE4 integration is very smooth and is available to use by contacting us at our support email address, and depending on the time of printing will be or already is available from our download page very soon.
Rashid K. Sayed: Given that PS4 and Xbox One have slower CPUs, does it impact the performance of the middleware in anyway?
Brett Paterson: The PS4 has a distinct advantage over the PS3, due to having a simpler, more PC like architecture without the limitations that the PS3 had. The PS3 had an SPU architecture where we ran our mixing and decoding, but it had limited memory space to work with, and because the SPUs don’t have direct access to main memory, we had to access everything through DMA transfers instead. This bottleneck slowed down the amount of voices and effects you could process on the SPU, but the PS4 doesn’t have this limitation, and it also has dedicated decoding hardware, which means it can process a lot more data.
As for Xbox One, the CPU is smarter than its predecessor, so even though it has a lower clock speed the out of order processing and more advanced CPU instructions let us process twice as much data than before.
Rashid K. Sayed: I am sure you must be following up with the latest advancements of cloud technology. Microsoft seems to be pushing for improving Xbox One’s performance using cloud. Do you think FMOD on Xbox One could use that?
Brett Paterson: The cloud is great for sending tasks and data offline and waiting for it to come back at a later time, but audio is typically a very low latency process, which means we haven’t thought much about it yet. Maybe it could be used for processing geometry outside of the game for creating convolution reverb impulse response data; I’m imagining in the future we could think of more ideas for this!
Rashid K. Sayed: Do you think the GDDR5 memory makes any difference or gives any advantages in the way FMOD is implemented on the PS4?
Brett Paterson: Audio deals with large amounts of raw compressed sound information. The faster memory bandwidth is, the less cycles it takes to read and decompress audio data. There is certainly a benefit, because unintuitively, less compressed information can consume CPU more time to manipulate than highly compressed data, because there is more data to transfer. Higher bitrates in MP3 for example can now be used at the cost of more memory being used, but without the overhead of the bandwidth stalls.
"Today the shift is moving from more sounds and voices to more pro-level DSP effects, mixing and mastering, which is made possible by the new generation of consoles. Game Audio is getting closer and closer to being similar in workflow to that of movie and TV production houses.
Rashid K. Sayed: How does implementation of FMOD differ in terms of cross generation development?
Brett Paterson: FMOD’s main advantage is being transparent across all platforms. We generally scale across platforms by letting the user use less complex decompression formats (such as ADPCM vs. Vorbis) and scaling how many audible voices you can hear at once, which reduces load on the CPU. The good thing about FMOD Studio’s virtual voice system though is that you can still play 1,000 sounds at once on a low end system and not have to worry about the CPU usage, as it ‘virtualizes’ the quieter, less important sounds and only makes the important, loud sounds audible and these are the ones that are mixed on the CPU.
Rashid K. Sayed: Ever since the beginning of new generation, what is the number one demand from developers in regards to FMOD?
Brett Paterson: Today the shift is moving from more sounds and voices to more pro-level DSP effects, mixing and mastering, which is made possible by the new generation of consoles. Game Audio is getting closer and closer to being similar in workflow to that of movie and TV production houses. It helps that FMOD Studio is so similar in interface to these tools, so we are seeing movie and TV people moving into games and being comfortable with our tools.
Rashid K. Sayed: What advances is Firelight Technologies doing in the indie development scene?
Brett Paterson: Since UE4 and Unity are now embracing the indie scene with new licensing models, there is a demand for FMOD Studio to have plugins or integrations with them both.
We have listened and produced some pretty slick integrations which indies are using right now. Our big news earlier in the year though was that FMOD Studio is now free for indies. Developers that have a small budget can use totally free, and there is no technical limit on what you can do, for example there is no limit on number of sounds.
Rashid K. Sayed: Is there anything else you want to tell our readers before we take off?
Brett Paterson: The latest developments in FMOD are the AstoundSound 3D spatialization plugin, and built in Convolution reverb. Also we are about to launch our Unreal 4 integration for free. More information can be found here.