As games become more and more photo-realistic, the ability to accurately model and animate faces – be they humans, beasts or otherworldly creatures – remains as strong as ever. The technologies have advanced significantly since the old days of pre-rendered CG, as developers are employing tools like Faceware to bring facial animation and the emotions that come with it into the game.
We recently spoke to Jake Fenske, Line Producer at CounterPunch Studios about Faceware, work-flows and even the amount of effort it took to model the Psycho from Crysis 3. CPS specializes in facial animation, and has produced it for many other top games such as Max Payne 3, Red Dead Redemption and more, using Faceware and their own technology in the process.
Ravi Sinha: Can you explain the entire process by which you used Faceware for Crysis 3? What were the advantages it presented versus traditional means of scanning and animating?
Jake Fenske: We animated 48 minutes of facial animation for in-game cinematics for Crysis 3. We use Faceware for all of our facial animation as it allows us to efficiently translate a captured performance from an actor to our proprietary facial rigs. Using Faceware is much more efficient and cuts our resourcing needs significantly.
The process for using Faceware is simple and requires only a few steps. First, the performance of an actor must be captured on video. Then our team uses the Faceware Analyzer to track the movement of the pixel data of the video. This data is exported into controller data that can be transferred to the facial rig. The Faceware Retargeter takes the data created by the Analyzer and transfers it to the facial rig with our animators establishing specific poses and controlling the process. When the data has been retargeted to the character you end up with a good level of animation data. This allows our team to focus more time on perfecting animation data to achieve the high quality animation that we achieved on Crysis 3.
Ravi Sinha: What other tools and technologies were brought in for the facial development in Crysis 3 aside from Faceware? How well did the various processes gel together?
Jake Fenske: We have a proprietary rigging process that allows us to create high quality facial rigs (a set of interconnected “bones” used to animate a 3D modeling mesh) for not only features but for gaming budgets as well. We have worked with Faceware often and our rigging technology has been developed alongside the development of Faceware.
Ravi Sinha: How many man hours did it take altogether to scan Pyscho’s face? How much faster was that than traditional methods? How many man hours did it take to animate his face? How much faster was that?
Jake Fenske: Typically the process of capturing the performance, rigging the characters, and animating can take several months but we perform each task diligently to cut the schedule and meet strict delivery dates. Although there is no way of getting around the true development time to achieve the true characteristics that make up each character. Our goals are to capture every nuance of a character’s face and performance.
Ravi Sinha: Roughly speaking how many polygons went it to building Pyshco’s face? More details on this will be appreciated.
Jake Fenske: After an actor is scanned to create the character, the density of the mesh (the surface representation used to draw the character) is extremely high. Psycho’s scanned mesh was around 50,000 polygons. Unfortunately this is too high for a current generation game engine so the character must be retopologized to create a usable head model. Psycho’s game ready head was approximately 3,500 polygons, which is much less costly in the game.
Ravi Sinha: Was the pipeline overtly complex with all these technologies used in conjunction? How did you manage it all?
Jake Fenske: We utilized the same pipeline we developed for other games that we have worked on in the past including GTA IV, Red Dead Redemption and Max Payne 3. We spend the majority of development time in the beginning establishing the look development and art direction for each character. This includes several levels of animation testing and building expressions and pose libraries to achieve a lifelike performance. Once we’ve achieved all of these benchmarks the process is standard for delivering up to hundreds of minutes of animation in a relatively short amount of time. Generally it cuts about 50% of the time it would take to hand animate.
Ravi Sinha: Crysis 3 features arguably the most realistic facial animations we’ve ever seen. But how well does that fit into the larger scheme of things, when you have massive photo-realistic levels, explosions and massive aliens vying for the player’s attention, and for the hardware’s processing power?
Jake Fenske: Now more than ever developers are trying to push the quality levels of all aspects of their games. Fewer characters have helmets or obstructive elements. There are also more cameras being set to extreme close ups to show off the quality that the game has achieved. This is the same for facial performance. In order for a player to be immersed in the game the facial quality has to be believable because, let’s face it, people want to imagine and fantasize that they are those players. Bad facial animation will take the player right out of the experience.
Ravi Sinha: It’s understandable if specific numbers can’t be given, but there must’ve been a sizable amount of resources allocated to the facial animation process, right?
Jake Fenske: In our experience, the budgets are actually quite limited. Because of this we have developed solutions that make the process much more efficient to keep the costs down basing our margins on volume.
Ravi Sinha: Talking about how all these tools were used in conjunction – and also about how the consoles somewhat limited the graphical potential of Crysis 3 – what do you see next gen consoles like the PS4 and Xbox One providing in terms of visual fidelity?
Jake Fenske: We are starting to develop technologies built around realtime applications. We are also starting to have discussions with Stereoscopic companies to project out where we think the gaming industry is going. We think one day people will be playing with holograms.
Ravi Sinha: As you must be aware, Xbox One will have 8GB of DDR3 RAM and the PS4 will have 8 GB of DDR5 RAM. Does the PS4 gives any advantage over the new Xbox in the way you guys will approach facial capture for next generation of games?
Jake Fenske: Anytime there is an increase in the hardware there will be an improvement in the level of facial quality that can be achieved. We will be able to use higher resolution models, rigs and textures. This only increases the believability in the facial animation and the deformation quality our facial rigs. Whether the PS4 has an advantage over the Xbox One is yet to be seen on our end but both are a significant upgrade from the current generation.
Ravi Sinha: Do you see a time when realistic facial animation will be as easy and fast to develop as high-profile games (as most developers praise the PS4 for being able to do)?
Jake Fenske: No matter how much technology advances, there is still no answer for the emotion a human being brings to a character. The fidelity will increase and so will the need for people who can bring life and emotion to a character.
Ravi Sinha: Concerning all the realistic modeling of artificial characters in video games, do you see a time when the Uncanny Valley might pop up in gaming?
Jake Fenske: One of the key factors with the Uncanny Valley implies that the visuals are disturbing enough to take you out of the visual experience. No one who has played Crysis 3 mentions the Uncanny Valley, which says that players are able to be fully immersed in the experienced. The concept we have been working toward the last few years (from GTA IV to Crysis 3) was to cross over to the other side of the uncanny valley and bring real life to a character. We are currently building photo real models for feature films using the same approach. We believe that gaming has already solved that problem with games like Crysis 3.
Ravi Sinha: Given all the time and effort put into such realistic facial animation, has the team ever considered just going straight into movie production?
Jake Fenske: We are currently in production building photo real characters and digital doubles for several features set to release over the next two years. These characters all employ the same technology that was used in Crysis 3. As an example, please feel free to check out Ek Tha Tiger, a record setting feature film released in India last year.
Special thanks to Eric Schumacher from Neology Concepts for setting the interview up.