"FMOD Studio has always come with mastering style plugins that let you produce a professional mix, like McDSP’s master limiter for example, and Dolby surround sound mixing…"

It’s been over two years since we last interviewed you and a lot has changed in the games development sector. Can you tell us how Firelight has moved ahead in the last several months?

FMOD Studio has been in development for several years now, but in the last months we’ve been working diligently on new features that really bring FMOD Studio to the forefront of audio development.

Can you tell us about the most recent advancements for FMOD including version 1.10?

The big 3 changes for the new FMOD Studio 1.10 release are:

• Smart Presets: ‘Smart presets’ solve the challenge of maintaining large projects, making it easy to manage audio parameters, effects and automations. They allow these things to be created separately from an audio event, and modularly re-used in many events at once. It’s a huge improvement in workflow for sound designers.
• Enhanced Live update/editing: Tools including FMOD Studio have let you view and edit some properties over a network connection, while the game is running, but FMOD Studio 1.10 takes it a step further. FMOD Studio 1.10 now allows you to add and delete events, as well as whole sound files live, and have them appear immediately in the game without recompiling.
• Enhanced per-platform settings: FMOD Studio 1.10 now enhances the ability to scale from a mobile phone all the way up to a high powered PC. Tracks, DSP sound effects, encoding settings and more can be switched on and off on a per platform basis.

There’s also a bunch of new stuff to help surround sound and VR development.

What are some of the latest plugins for FMOD that you feel AAA developers could take advantage of, especially given how so many players put a lot on emphasis on OST/Music these days?

FMOD Studio has always come with mastering style plugins that let you produce a professional mix, like McDSP’s master limiter for example, and Dolby surround sound mixing, but the biggest change in plugins lately has been revolving around VR. Oculus/Facebook and Google now produce FMOD compatible plugins that allow advanced 3D over headset positioning which allows for greater immersion in games than before.

"We can now see huge projects with tens of thousands of sounds and behaviours. Being able to manage such huge asset bases is a big priority now."

The way developers approach sound has changed a lot in the last couple of years. What are some of the more unique challenges that developers face with regards to sound effects and how FMOD helps solve these, especially during the last 2-3 years?

Sound effects in games are not simple wave files any more. They are complex, multi-layered beasts which react in real-time to what is happening in the game.

To make this easier FMOD Studio allows for a whole range of unique, user interface friendly methods to branch and layer audio. As games get bigger, workflow is the biggest issue, and new features in FMOD Studio allow for more efficient workflow. This includes reusing assets, which smart presets are good for, scaling from mobile to AAA which per platform settings are great for, and build time turnaround, which live editing is great for.

How has the workflow of game audio evolved over the past two years? Do you see it as being closer to film and TV production houses than before?

We can now see huge projects with tens of thousands of sounds and behaviours. Being able to manage such huge asset bases is a big priority now. Big audio teams now have multiple people all working on the same project at once. That’s why things like audio tool source control integration is important.

In the past, mixing and mastering was not such a big priority with game development, but it is a big focus now and the tools and plugins are available to achieve this. Supporting end-point technologies like Dolby Atmos is now possible in games, which originally was for film, is available now for games, and it’s a great step towards movie like experiences.

With Unreal Engine 4 and Unity having made such a big impact for indie development, what are some of the changing demands from FMOD (aside from plugins for FMOD Studio and it being free for indies)?

We have definitely had to focus more on supporting Unity and Unreal in the last few years, rather than treating them as a distraction to our main purpose. Lately Unity and Unreal have taken more focus for us so we have specialized people just on the integrations. They are the largest user base now so we spend a lot of time on them, you can see from the front page of our new website we focus on how easy it is to get FMOD into engine middleware like UE4 and Unity.

"Compressed audio decoding is done by the Xbox dedicated hardware, and not the CPU, but these days compressed audio is becoming less of an overhead…"

We’ve seen spatialization plugins for VR platforms but where do you FMOD in the future when it comes to virtual reality hardware?

FMOD Studio is always ready to support the latest technologies when it comes to VR.

In our last interview, you revealed that FMOD’s process is CPU based. How hard has it been to optimize FMOD on the Jaguar based processors of the PS4 and Xbox One, especially given the high demands of developers these days?

The processors in the PS4 and Xbox are quite easy to develop with, as they are compatible with each other and the PC platforms now. Our main focuses for optimization on consoles is to support their native hardware codecs for highly compressed audio, which they both have, and support any special effects or features they might offer.

FMOD Studio offers GPU accelerated convolution reverb on PC and Xbox One, this can improve performance a lot for such a demanding effect.

I am sure you must have heard about the Xbox One X and its hardware specifications. As we all know, it boasts 12GB of GDDR5 memory. What are the possibilities of FMOD for those who handle lots of compressed sound on the Xbox One X?

The changes allow the developer to store more sounds and make them higher quality if they want, but the changes for the new consoles don’t affect audio as much as graphics.

The Xbox One X features a slightly better CPU than the Xbox One. Do you think the bump will make a meaningful impact on the Xbox One X since FMOD is based on the CPU? Any chance, it can hold back FMOD from truly shining?

Compressed audio decoding is done by the Xbox dedicated hardware, and not the CPU, but these days compressed audio is becoming less of an overhead, compared to the heavy demand of a big project, and the complex mixing structures and special effects they use. Unless the game is Xbox One X specific, which as far as I know is not allowed, the games still have to limit themselves to the power of the original Xbox One.

"The new machines are mostly designed to enhance graphic capability and console hardware changes generally do not affect audio features."

On the other hand, we have the PS4 Pro, which has slightly less CPU power but noticeably less memory bandwidth and amount. Do you think it’s possible for developers to achieve the same level of FMOD performance on the Pro compared to the Xbox One X?

The machines are not different enough in terms of audio to comment on.

Do you think FMOD tech is being held back due to Sony and Microsoft’s compatibility requirement across PS4/PS4 Pro and Xbox One X/Xbox One?

The new machines are mostly designed to enhance graphic capability and console hardware changes generally do not affect audio features. Both consoles are already very powerful, so these days it is more up to the creativity of the sound designers rather than technical limitations.

How is development going on to support the Nintendo Switch?

FMOD has supported Nintendo Switch for a while now and is being used by developers.

Is there anything else you want to tell our readers before we let you go?

If users haven’t looked at FMOD Studio for a while I would highly recommend looking at the new releases which have come leaps and bounds in terms of features, since their initial iterations. More information can be found at our new website.

Latest update: It seems like Uncharted 4: A Thief’s End may be using the extra processing power as well. A tweet from Naughty Dog lead designer Robert Cogburn possibly hints at its usage.

Original Story:

Last week it was widely reported that Sony had unlocked the 7th CPU core of the PS4 for developers. The information was taken from FireLight Technologies’ website, a company that deals with audio middleware [FMOD], when they revealed that their tech was using the PS4’s 7th core. The change log has since been removed but we got in touch with FireLight to get a confirmation regarding the same.

Brett Paterson from FireLight Technologies confirmed to GamingBolt via email that the 7th CPU core of the PS4 is indeed available for the developers. However its allocation is not dynamic which possibly means that only a certain percentage of the core’s processing units are available to the developers. Brett also revealed a few details regarding how FMOD is using the 7th core. FMOD is shared with other OS libraries on the core and in case FMOD doesn’t receive enough CPU, the debugging profiler needs to be used to check out what is causing the overhead. [Read more about PS4’s Razor profiler here]

According to previous reports, only 6 of the 8 cores were available for developers which means the 7th core will definitely help development in someway. In case of FMOD, the developer can simple offload the audio middleware on the 7th core while ensuring other OS libraries and code are working properly.

Microsoft on the other hand also provided developers with the Xbox One’s 7th core but with a dynamic range of 50-80% utilization. This has helped several developers improve frame rate and other performance parameters of Xbox One games. It will be interesting to see how this will help the developers working on the PS4. If you are interested in learning more about FMOD and FireLight technologies, check out GamingBolt’s interview here.

GamingBolt spoke to Firelight CEO Brett Paterson about FMOD and the tools it offered along with its future potential. How does FMOD take advantage of technology like the Xbox One’s on-board audio processor? We find out below, along with other details.

Rashid K. Sayed: To begin with, can you tell us about yourself and FMOD?

Brett Paterson: My name is Brett Paterson, and I am the CEO of Firelight Technologies and original developer on FMOD, which has been available for over 10 years now. FMOD started out as a simple programmer interface to load sounds and play them in 3D, but since then the product (and company) has grown, and FMOD has transformed into a high end, professional tool suite called FMOD Studio, which includes a Digital Audio Workstation style interface (think Ableton, Logic, Pro Tools) for creating interactive sound events and music, and a runtime component which you plug into your game called the FMOD Studio runtime API, which is what the programmers use to link up the game to the audio data.

"FMOD Studio currently uses the audio processor on the Xbox One for two things: XMA decoding and sample rate conversion. This is one of the most expensive parts of the process when processing audio in real-time...

Rashid K. Sayed: Can you briefly talk about the various FMOD Studio Plugins?

Brett Paterson: FMOD Studio now ships with 3 plugins from pro­-audio companies typically known for their plugins in the music industry. We have McDSP’s ML1 (Master Limiter) filter for pro­level mastering of the game’s audio mix, and AudioGaming’s synthesis plugin for real-time generation of wind and rain effects. Lately we also added support for GenAudio’s 3D virtualization plugin for headphones called AstoundSound. This lets you get much better spatialization of sound than the default 3D panner within FMOD. We will gradually be adding more cool and interesting plugins like this over time.

Rashid K. Sayed: Tell us what kind of updates have you made to the middleware to support the new consoles PS4 and Xbox One?

Brett Paterson: FMOD Studio has added native support for hardware accelerated decompression of XMA data on Xbox One and ATRAC9 and resampling support on PlayStation 4, which allows more sounds to be played at once. We also made some updates to support recording and peripherals such as headsets to allow recording and playback through external devices. Otherwise the core feature set / functionality is the same for all platforms.

Rashid K. Sayed: From architecture point of view, how deeply is FMOD involved? Do you use the GPU at all or is the process all CPU based?

Brett Paterson: FMOD Studio does all of its mixing and signal processing on the CPU, which is a departure from the previous FMOD Ex engine. It allows us total flexibility and control over the signal, and avoids the limitations of dedicated hardware such as lack of cross­-platform support. We haven’t put any processing onto GPUs yet, though we do support AMD’s TrueAudio hardware which is GPU based, and PC only at the moment.

Rashid K. Sayed: How does FMOD takes advantage of the on board audio processor on the Xbox One? Since it has a dedicated audio CPU, does FMOD make a difference in such a scenario?

Brett Paterson: FMOD Studio currently uses the audio processor on the Xbox One for two things: XMA decoding and sample rate conversion. This is one of the most expensive parts of the process when processing audio in real-time, so it takes a significant load off the CPU and allows for more effects like reverb, EQ filters and other filters.

Rashid K. Sayed: The PlayStation 4 does not have a dedicated audio processor. Does it make FMOD implementation a bit different or harder from the Xbox One?

Brett Paterson: On PlayStation 4 there is hardware to allow decoding of ATRAC compressed audio, so it is very similar to the Xbox One in regards to offloading decoding from the CPU.

"The cloud is great for sending tasks and data offline and waiting for it to come back at a later time, but audio is typically a very low latency process, which means we haven’t thought much about it yet.

Rashid K. Sayed: I understand that the middleware will have compatibility with Unity but what about Unreal Engine 4?

Brett Paterson: Our Unity engine integration for FMOD Studio is already on our website and ready to download. We also have a working UE4 integration that is one of the first middleware integrations that use the UE4 plugin system. Epic has invited us to speak about how we did this at their room at the GameConnect Australia Pacific conference in Melbourne this month. The UE4 integration is very smooth and is available to use by contacting us at our support email address, and depending on the time of printing will be or already is available from our download page very soon.

Rashid K. Sayed: Given that PS4 and Xbox One have slower CPUs, does it impact the performance of the middleware in anyway?

Brett Paterson: The PS4 has a distinct advantage over the PS3, due to having a simpler, more PC like architecture without the limitations that the PS3 had. The PS3 had an SPU architecture where we ran our mixing and decoding, but it had limited memory space to work with, and because the SPUs don’t have direct access to main memory, we had to access everything through DMA transfers instead. This bottleneck slowed down the amount of voices and effects you could process on the SPU, but the PS4 doesn’t have this limitation, and it also has dedicated decoding hardware, which means it can process a lot more data.

As for Xbox One, the CPU is smarter than its predecessor, so even though it has a lower clock speed the out of order processing and more advanced CPU instructions let us process twice as much data than before.

Rashid K. Sayed: I am sure you must be following up with the latest advancements of cloud technology. Microsoft seems to be pushing for improving Xbox One’s performance using cloud. Do you think FMOD on Xbox One could use that?

Brett Paterson: The cloud is great for sending tasks and data offline and waiting for it to come back at a later time, but audio is typically a very low latency process, which means we haven’t thought much about it yet. Maybe it could be used for processing geometry outside of the game for creating convolution reverb impulse response data; I’m imagining in the future we could think of more ideas for this!

Rashid K. Sayed: Do you think the GDDR5 memory makes any difference or gives any advantages in the way FMOD is implemented on the PS4?

Brett Paterson: Audio deals with large amounts of raw compressed sound information. The faster memory bandwidth is, the less cycles it takes to read and decompress audio data. There is certainly a benefit, because unintuitively, less compressed information can consume CPU more time to manipulate than highly compressed data, because there is more data to transfer. Higher bitrates in MP3 for example can now be used at the cost of more memory being used, but without the overhead of the bandwidth stalls.

"Today the shift is moving from more sounds and voices to more pro­-level DSP effects, mixing and mastering, which is made possible by the new generation of consoles. Game Audio is getting closer and closer to being similar in workflow to that of movie and TV production houses.

Rashid K. Sayed: How does implementation of FMOD differ in terms of cross generation development?

Brett Paterson: FMOD’s main advantage is being transparent across all platforms. We generally scale across platforms by letting the user use less complex decompression formats (such as ADPCM vs. Vorbis) and scaling how many audible voices you can hear at once, which reduces load on the CPU. The good thing about FMOD Studio’s virtual voice system though is that you can still play 1,000 sounds at once on a low end system and not have to worry about the CPU usage, as it ‘virtualizes’ the quieter, less important sounds and only makes the important, loud sounds audible and these are the ones that are mixed on the CPU.

Rashid K. Sayed: Ever since the beginning of new generation, what is the number one demand from developers in regards to FMOD?

Brett Paterson: Today the shift is moving from more sounds and voices to more pro­-level DSP effects, mixing and mastering, which is made possible by the new generation of consoles. Game Audio is getting closer and closer to being similar in workflow to that of movie and TV production houses. It helps that FMOD Studio is so similar in interface to these tools, so we are seeing movie and TV people moving into games and being comfortable with our tools.

Rashid K. Sayed: What advances is Firelight Technologies doing in the indie development scene?

Brett Paterson: Since UE4 and Unity are now embracing the indie scene with new licensing models, there is a demand for FMOD Studio to have plugins or integrations with them both.

We have listened and produced some pretty slick integrations which indies are using right now. Our big news earlier in the year though was that FMOD Studio is now free for indies. Developers that have a small budget can use totally free, and there is no technical limit on what you can do, for example there is no limit on number of sounds.

Rashid K. Sayed: Is there anything else you want to tell our readers before we take off?

Brett Paterson: The latest developments in FMOD are the AstoundSound 3D spatialization plugin, and built in Convolution reverb. Also we are about to launch our Unreal 4 integration for free. More information can be found here.

It’s easy to get caught up in the resolution war and the power of graphics with the Xbox One and PS4 but there are several other benefits that the technology bestows. Speaking to Firelight Technologies’ Brett Paterson, GamingBolt learned how the PS4’s GDDR5 RAM presents advantages to the audio framework FMOD. Firelight’s FMOD is a special set of tools for creating interactive audio and is used across just about every single game you can think. Dragon Age, World of Warcraft, Forza – roughly 1500 different games use FMOD.

So what advantage does the PS4 bring to FMOD with its memory? Paterson stated that, “Audio deals with large amounts of raw compressed sound information. The faster memory bandwidth is, the less cycles it takes to read and decompress audio data. There is certainly a benefit, because unintuitively, less compressed information can consume CPU more times to manipulate than highly compressed data, because there is more data to transfer.

“Higher bitrates in MP3 for example can now be used at the cost of more memory being used, but without the overhead of the bandwidth stalls.”

What are your thoughts on this, especially with the emphasis on updates for the PS4 and Xbox One serving to unlock more of the their respective consoles’ power? Let us know in the comments.