Vir2elle is a synthetic agent, developed at AT&T Research Labs, that can produce video-realistic speech animations. Vir2elle's realistic appearance stems from the so-called "image-based" or "sample-based" approach. Sample images of facial parts are first collected during a training session. The subject is recorded using a standard video camera while speaking a corpus of training sentences. Each recorded video frame is labeled with its phonetic content using a Phoneme Aligner. The data is then compressed and stored in a "model" file. The creation of the model file is a one-time operation and Vir2elle is subsequently able to produce ANY speech animation from the model file. Vir2elle uses concatenative synthesis to reassemble stored images of facial parts into a new animation that is lip-synchronized with the audio track. Vir2elle requires, as input, a "string of phonemes" to produce an animation. Such data is available from off-the-shelve Text-To-Speech (TTS) synthesizers. Alternatively, a recorded audio track can be phonetically labeled to produce even more realistic and convincing animations.

 

To produce emotions and facial expressions, Vir2elle accepts special markers called emoticons within the text input. The insertion of emotions happens during pauses in the speech. Parameterized emotion snippets are inserted and can be adjusted for duration and intensity.

 

My role:

Researchers spent four years training and parametrizing one 3D model to produce photo-realistic text to speech. However, it would have been difficult to spend the same amount of time to train every new model. My task, performed in 12 weeks during the summer of 2002, was to design and implement a system that would take an existing parameterized mesh, including all anchor points, and apply it over a new point cloud from a 3D scanner. The system would then map the texture of the new face to the deformed mesh of the old face now applied over the new model.

This technique is similar to what’s used in Hollywood make up studios for a long time and has been portrayed various times in movies like Mission Impossible.