Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control

Andreea Stef; Kaveen Perera; Hubert P. H.  Shum; Edmond Ho

doi:10.1109/SKIMA.2018.8631536

Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control

Andreea Stef, Kaveen Perera, Hubert P. H. Shum, Edmond Ho

Computer and Information Sciences Department

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Citations (Scopus)

47 Downloads (Pure)

Abstract

Given the complexity of human facial anatomy, animating facial expressions and lip movements for speech is a very time-consuming and tedious task. In this paper, a new text-to-animation framework for facial animation synthesis is proposed. The core idea is to improve the expressiveness of lip-sync animation by incorporating facial expressions in 3D animated characters. This idea is realized as a plug-in in Autodesk Maya, one of the most popular animation platforms in the industry, such that professional animators can effectively apply the method in their existing work. We evaluate the proposed system by conducting two sets of surveys, in which both novice and experienced users participate in the user study to provide feedback and evaluations from different perspectives. The results of the survey highlight the effectiveness of creating realistic facial animations with the use of emotion expressions. Video demos of the synthesized animations are available online at https://git.io/fx5U3

Original language	English
Title of host publication	2018 12th International Conference on Software, Knowledge, Information Management & Applications (SKIMA)
Subtitle of host publication	Phnom Penh, Cambodia 3 – 5 December 2018.
Place of Publication	Piscataway, NJ
Publisher	IEEE
Pages	70-78
Number of pages	8
ISBN (Electronic)	9781538691410
ISBN (Print)	9781538691427
DOIs	https://doi.org/10.1109/SKIMA.2018.8631536
Publication status	Published - 4 Feb 2019
Event	12th International Conference on Software, Knowledge, Information Management and Applications - Institute of Technology of Cambodia, Phnom Penh, Cambodia Duration: 3 Dec 2018 → 5 Dec 2018 http://skimanetwork.info/

Publication series

Name	International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Proceedings
ISSN (Print)	2373-082X
ISSN (Electronic)	2573-3214

Conference

Conference	12th International Conference on Software, Knowledge, Information Management and Applications
Abbreviated title	SKIMA 2018
Country/Territory	Cambodia
City	Phnom Penh
Period	3/12/18 → 5/12/18
Internet address	http://skimanetwork.info/

Keywords

Lip-sync
Facial animation
Facial expressions
Emotion
Character animation

Access to Document

10.1109/SKIMA.2018.8631536

Stef et al - Synthesizing Expressive Facial and Speech AAMAccepted author manuscript, 3.4 MB

Interaction-based Human Motion Analysis
Shum, H. (PI)
Engineering and Physical Sciences Research Council
1/11/14 → 30/04/16
Project: Research

Cite this

Stef, A., Perera, K., Shum, H. P. H., & Ho, E. (2019). Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control. In 2018 12th International Conference on Software, Knowledge, Information Management & Applications (SKIMA): Phnom Penh, Cambodia 3 – 5 December 2018. (pp. 70-78). (International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Proceedings). IEEE. https://doi.org/10.1109/SKIMA.2018.8631536

Stef, Andreea ; Perera, Kaveen ; Shum, Hubert P. H. et al. / Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control. 2018 12th International Conference on Software, Knowledge, Information Management & Applications (SKIMA): Phnom Penh, Cambodia 3 – 5 December 2018.. Piscataway, NJ : IEEE, 2019. pp. 70-78 (International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Proceedings).

@inproceedings{464a2716da7d4e68a12aaa6c91c0cb67,

title = "Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control",

abstract = "Given the complexity of human facial anatomy, animating facial expressions and lip movements for speech is a very time-consuming and tedious task. In this paper, a new text-to-animation framework for facial animation synthesis is proposed. The core idea is to improve the expressiveness of lip-sync animation by incorporating facial expressions in 3D animated characters. This idea is realized as a plug-in in Autodesk Maya, one of the most popular animation platforms in the industry, such that professional animators can effectively apply the method in their existing work. We evaluate the proposed system by conducting two sets of surveys, in which both novice and experienced users participate in the user study to provide feedback and evaluations from different perspectives. The results of the survey highlight the effectiveness of creating realistic facial animations with the use of emotion expressions. Video demos of the synthesized animations are available online at https://git.io/fx5U3",

keywords = "Lip-sync, Facial animation, Facial expressions, Emotion, Character animation",

author = "Andreea Stef and Kaveen Perera and Shum, {Hubert P. H.} and Edmond Ho",

year = "2019",

month = feb,

day = "4",

doi = "10.1109/SKIMA.2018.8631536",

language = "English",

isbn = "9781538691427",

series = "International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Proceedings",

publisher = "IEEE",

pages = "70--78",

booktitle = "2018 12th International Conference on Software, Knowledge, Information Management & Applications (SKIMA)",

address = "United States",

note = "12th International Conference on Software, Knowledge, Information Management and Applications, SKIMA 2018 ; Conference date: 03-12-2018 Through 05-12-2018",

url = "http://skimanetwork.info/",

}

Stef, A, Perera, K, Shum, HPH & Ho, E 2019, Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control. in 2018 12th International Conference on Software, Knowledge, Information Management & Applications (SKIMA): Phnom Penh, Cambodia 3 – 5 December 2018.. International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Proceedings, IEEE, Piscataway, NJ, pp. 70-78, 12th International Conference on Software, Knowledge, Information Management and Applications, Phnom Penh, Cambodia, 3/12/18. https://doi.org/10.1109/SKIMA.2018.8631536

Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control. / Stef, Andreea; Perera, Kaveen; Shum, Hubert P. H. et al.
2018 12th International Conference on Software, Knowledge, Information Management & Applications (SKIMA): Phnom Penh, Cambodia 3 – 5 December 2018.. Piscataway, NJ: IEEE, 2019. p. 70-78 (International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control

AU - Stef, Andreea

AU - Perera, Kaveen

AU - Shum, Hubert P. H.

AU - Ho, Edmond

PY - 2019/2/4

Y1 - 2019/2/4

N2 - Given the complexity of human facial anatomy, animating facial expressions and lip movements for speech is a very time-consuming and tedious task. In this paper, a new text-to-animation framework for facial animation synthesis is proposed. The core idea is to improve the expressiveness of lip-sync animation by incorporating facial expressions in 3D animated characters. This idea is realized as a plug-in in Autodesk Maya, one of the most popular animation platforms in the industry, such that professional animators can effectively apply the method in their existing work. We evaluate the proposed system by conducting two sets of surveys, in which both novice and experienced users participate in the user study to provide feedback and evaluations from different perspectives. The results of the survey highlight the effectiveness of creating realistic facial animations with the use of emotion expressions. Video demos of the synthesized animations are available online at https://git.io/fx5U3

AB - Given the complexity of human facial anatomy, animating facial expressions and lip movements for speech is a very time-consuming and tedious task. In this paper, a new text-to-animation framework for facial animation synthesis is proposed. The core idea is to improve the expressiveness of lip-sync animation by incorporating facial expressions in 3D animated characters. This idea is realized as a plug-in in Autodesk Maya, one of the most popular animation platforms in the industry, such that professional animators can effectively apply the method in their existing work. We evaluate the proposed system by conducting two sets of surveys, in which both novice and experienced users participate in the user study to provide feedback and evaluations from different perspectives. The results of the survey highlight the effectiveness of creating realistic facial animations with the use of emotion expressions. Video demos of the synthesized animations are available online at https://git.io/fx5U3

KW - Lip-sync

KW - Facial animation

KW - Facial expressions

KW - Emotion

KW - Character animation

U2 - 10.1109/SKIMA.2018.8631536

DO - 10.1109/SKIMA.2018.8631536

M3 - Conference contribution

SN - 9781538691427

T3 - International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Proceedings

SP - 70

EP - 78

BT - 2018 12th International Conference on Software, Knowledge, Information Management & Applications (SKIMA)

PB - IEEE

CY - Piscataway, NJ

T2 - 12th International Conference on Software, Knowledge, Information Management and Applications

Y2 - 3 December 2018 through 5 December 2018

ER -

Stef A, Perera K, Shum HPH, Ho E. Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control. In 2018 12th International Conference on Software, Knowledge, Information Management & Applications (SKIMA): Phnom Penh, Cambodia 3 – 5 December 2018.. Piscataway, NJ: IEEE. 2019. p. 70-78. (International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Proceedings). doi: 10.1109/SKIMA.2018.8631536

Synthesizing Expressive Facial and Speech Animation by Text-to-IPA Translation with Emotion Control

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Projects

Interaction-based Human Motion Analysis

Cite this