For those of you who are not aware, Dynamixyz is specializes in developing tools and technologies for facial capture for video games and movies. GamingBolt recently spoke to the company’s CEO, Gaspard Breton and asked him about his thoughts on the PS4 and Xbox One and whether their technology will cater to them.

“Yes, of course! Our technology is dedicated to preparing data which can be played with any hardware, system, game engine… We’re very much looking forward to the new console generation. The new computing power will allow studios to create even more realistic characters, which puts more pressure on the quality of facial animation. This makes our technology all the more relevant, as quality is always our top-priority,” he said.

But the main bottleneck for them is the lack of computation power and he is still to come across consoles and gaming hardware that can accurarely do facial capture above 30 frames per second. “For us, the bottleneck is the computational power when doing real time. We are still missing a machine which is able to track accurately, retarget, animate and render at more than 30 FPS (in real-time),” he said.

Thoughts? Let us know in the comments below.

Rashid Sayed: How did the partnership between Dynamixyz and Vicon came about?

Gaspard Breton: Dynamixyz and Vicon have been in touch since the creation of Dynamixyz. Our company met for the first time in fall 2012, we were so excited to meet such a pioneering company in the field of motion capture! Dynamixyz is a strong R&D oriented company and is always looking for new hardware.

Designing an HMC is such a difficult task and HMC must be chosen wisely depending on the production constraints (accuracy, weight, wobbling, solidity, wireless transmission capability…), so we thought it best to make sure our system works with any hardware. Vicon was designing one of the first multiview camera systems at that time, so it was natural for us to see what could be achieved combining Vicon’s hardware and our face tracking technology.

Rashid Sayed: Dynamixyz has developed a system, Performer-Multiview which when used with Cara allows detailed facial capture. Can you please let us know how this works?

Gaspard Breton: Dynamixyz’ facial tracking system is relying on 3D models by nature. The system first builds a kind of digital double of the actor with all possible expressions. Then it deforms the model and renders it in all camera planes so that it matches the actor’s expressions in all 4 views. In a way, tracking parameters can be considered as animation parameters of the digital double and are quite easy to retarget to a virtual character because they are of the same nature.

"Our technology is dedicated to preparing data which can be played with any hardware, system, game engine... We're very much looking forward to the new console generation. The new computing power will allow studios to create even more realistic characters, which puts more pressure on the quality of facial animation."

Rashid Sayed: What other features have you planned for Vicon’s facial motion capture system Cara?

Gaspard Breton: What is really interesting about the system is its ability to track both marker-based and markerless features. A lot of people are used to working with markers and have developed dedicated skills to do point-based retargeting. They still can do it with our system, but we can also provide other types of complimentary data which can be very useful when retargeting.

Tracking has its limits (what about the eyes for example). The strength really is the ability to have the best of both worlds within one single data stream: marker tracking for the forehead and cheeks, and markerless feature tracking for the eyes, the lips, …

Rashid Sayed: How did you go about planning this solution and how much work went into developing it over the years? What kind of advantages does it hold over traditional motion capture?

Gaspard Breton: We started working on this type of mocap system back in 2003 while collecting data for the needs of a high fidelity audio-visual speech production system. No mocap system was really capable of recording what we needed (inner lips contour for instance, very precise lips movement, wrinkles…) so we started developing our own.

With a video based system, every pixel of the face becomes a marker and the tracking is, then, extremely precise. Moreover, it delivers tracking data in a very compact format which can be used for retargeting in a very straightforward manner.

Rashid Sayed: What are your thoughts on the next generation of gaming, namely the PS4 and Xbox One? Will your technology cater to these consoles?

Gaspard Breton: Yes, of course! Our technology is dedicated to preparing data which can be played with any hardware, system, game engine… We’re very much looking forward to the new console generation. The new computing power will allow studios to create even more realistic characters, which puts more pressure on the quality of facial animation.

This makes our technology all the more relevant, as quality is always our top-priority. For us, the bottleneck is the computational power when doing real time. We are still missing a machine which is able to track accurately, retarget, animate and render at more than 30 FPS (in real-time).

"We think hardware like Cara can bring this level of quality compared to single view systems, because it captures 3D movements more accurately and helps solving mouth configuration ambiguities."

Rashid Sayed: Furthermore, if your technology is ever used for the PS4 and Xbox One do you think with Direct X 11.1, large amounts of RAM you will be able to achieve extremely detailed animation/motion capture?

Gaspard Breton: By essence, our technology only deals with traditional rig-based animation parameters so the in-game animation process does not really need large amounts of memory. The tracking parameters we deliver are really condensed, i.e., we do not surface scan all vertices and deliver meshes or something of this kind.

However, facial animation will greatly benefit from highly detailed face models and rigs, and from higher frame rates enabled by the new hardware. So, to us, more memory simply means more details for the character models and higher frame rates!

Rashid Sayed: Talking from the hardware perspective, the PS4 has 8GB of GDDR5 RAM compared to the Xbox One’s 8GB of DDR3 RAM. Do you think that extra memory on the PS4 will matter for detailed animation/motion capture?

Gaspard Breton: As mentioned above, we are more suffering from computational power in order to deliver high frame rates. So most important is how the hardware deals with multithreading and the speed of the memory cache!

Rashid Sayed: Do you think Cara will help in bringing a more CG-level of quality to motion capture in games? How long will it be till this is the norm for next gen-consoles and PC alike?

Gaspard Breton: Animating and rendering realistic human faces can be very difficult because our eyes are experts for finding artifacts which dramatically lower our acceptation of the character, this is the well-known “Uncanny Valley”… So, let’s say, achieving really high quality renderings, like photorealistic, is only a tool which brings more details to the skin, that is to say, will enable our eyes to have more anchor points to assess the quality of the movements.

In a few words, the more high quality rendering you achieve, the more realistic animation you have to deliver. We think hardware like Cara can bring this level of quality compared to single view systems, because it captures 3D movements more accurately and helps solving mouth configuration ambiguities (especially regarding depth). These details are going to become more and more important because of the high quality rendering which will emphasize any problem!

"We think there will never be saturation for facial capture. More than half of the part of the brain dedicated to shape recognition is dedicated to processing faces, that is to say it is one of the brain's main hobbies :-)"

Rashid Sayed: Do you think the PS4 and Xbox One are capable CG-Level facial capture in the next few years?

Gaspard Breton: I have been in the field for 15 years now, and have always heard about people asking if high quality facial animation is really important. My answer is yes, it is really important, but if you want to do it, you have to do it well. In my opinion, with the consoles being more and more powerful and rendering capacities increasing continuously (remember subsurface scattering equations used for rendering human skin were written in the 80’s), there is now a real need for high quality facial animation and it will become more and more important. It seems to me capture, animation and rendering techniques are ready now, it will more be a question of gameplay!

Rashid Sayed: With both Microsoft and Sony pushing for increased indie game presence on their consoles, will your solution cater to independent developers and how?

Gaspard Breton: We are currently considering a new range of offers for indies that would allow them to use our technology (software and hardware) while staying within their budget limitations and still getting the highest possible quality. The danger for Indies is to compromise on quality for price and we are investigating the best way to meet their quality needs within their financial requirements.

Rashid Sayed: Do you think there is a saturation level for facial capture? How much more can one push this technology? What are the problems in achieving perfection?

Gaspard Breton: We think there will never be saturation for facial capture. More than half of the part of the brain dedicated to shape recognition is dedicated to processing faces, that is to say it is one of the brain’s main hobbies 🙂 Once the perfection aspect is solved (and we are still far from it), then will come the repeatability problem. Imagine, in the future, a really gorgeous and realistic facial interface is the front desk of the AI taking care of your house.

It is so advanced you can hardly make a distinction with a real human. Well, maybe, at first, you won’t be able, but as time goes by, your brain will, without any doubt, notices all temporal patterns and recognizes all predefined sequences played repeatedly. And then you’ll get bored and angry to have been fooled. We think there is a great challenge in developing a capture and animation system which is able to modify and add randomness into the movements as real humans do, sometimes just because of physiological noise.

Rashid Sayed: Is there any difference in using Cara for video games and using it for movies?

Gaspard Breton: I would say this is more a matter of budget than technology whatever industry is considered:-) The problem is usually projects require several systems because there could be many actors performing live, which could dramatically increase the costs. The good thing with video games is that, contrary to movies which are more linear, the amount of production can be quite big and the use of such system then makes a lot of sense.

Rashid Sayed: How is your technology different from other competitors, say Faceware?

Gaspard Breton: People used to say about our technology that it is one of the most accurate in the market right now although the setup process can be considered as a bit tedious (a simplified workflow is on its way for the next version). However, once the setup is done, the system becomes completely automatic and can track and retarget any sequence without any manual intervention.

So, I would say our system performs greatly and it is really interesting especially for medium and large projects (i.e. above several minutes of production). Also, our workflow is highly unified, that is to say, when the setup is achieved you can do high quality tracking as well as real time which means it is capable of doing a good previsualization, close to the end results.

Rashid Sayed: Is there anything else you want to tell us about your technology?

Gaspard Breton: Just want to say Dynamixyz is really a R&D oriented company. We developed trackers for single view as well as multiview systems and even 3D sensors . People don’t really know about it but we also develop systems for dermatology, security and authentication as well as medical. We are always really happy to develop partnerships with companies wishing to outsource some of their R&D and love to help them finding the best solution for their needs.

As the power of consoles increases, achieving feats we never thought possible – namely photo-realistic visuals running at 60 FPS outputted to a 1080p resolution – one begins to wonder just what the PS4 and Xbox One aren’t capable of (the former more so than the latter). Will the day be far off when we finally see facial capture animation attain CG-levels?

GamingBolt spoke to CEO and co-founder of Dynamixyz Gaspard Breton about whether the PS4 and Xbox One could achieve such levels of facial capture within the coming years. Breton replied, “I have been in the field for 15 years now, and have always heard about people asking if high quality facial animation is really important. My answer is yes, it is really important, but if you want to do it, you have to do it well.

“In my opinion, with the consoles being more and more powerful and rendering capacities increasing continuously (remember subsurface scattering equations used for rendering human skin were written in the 80’s), there is now a real need for high quality facial animation and it will become more and more important. It seems to me capture, animation and rendering techniques are ready now, it will more be a question of gameplay!”

One thing is for sure – with games like The Witcher 3: Wild Hunt and The Order: 1886, the future of CG-quality visuals on consoles seems closer than ever. What are your thoughts? Let us know in the comments.

CEO and co-founder Gaspard Breton stated that “We strongly believe in markerless facial motion capture as it is the only technology able to capture the so many subtle movements of the face accurately and for every region including inner lips contour, wrinkles, skin color changes, eye movements… With that new release, Performer is now a 2 year-old product which has proven its accuracy and efficiency.”

Breton states that more information will be revealed on July 21st to 25th at Siggraph 2013. There are apparently no risks involved as Breton states that that Performer 7 will be “using a HMC with a 3D sensor operating at very close range and safe for the eyes. This new technology integrates easily in Performer’s workflow and will deliver very high quality data.”

Other new features involved include monitoring in console mode, improvements in relation to background tasks and the GUI. Interested developers can head here for evaluating the new release.