Conversational Human Audio-visual Talking Dialogue Generation

Junhao Song; Lluis Guasch; Xilin He; Zhongyu Yang; Yingfang Yuan; Weicheng Xie; Linlin Shen; Haijun Lin; Shizhe Liu; Wei Pang; Siyang Song

doi:10.1007/978-3-032-37252-9_33

Conversational Human Audio-visual Talking Dialogue Generation

Junhao Song, Lluis Guasch, Xilin He, Zhongyu Yang, Yingfang Yuan, Weicheng Xie, Linlin Shen, Haijun Lin, Shizhe Liu, Wei Pang, Siyang Song^*

^*Corresponding author for this work

School of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Chapter › peer-review

10 Downloads (Pure)

Abstract

Large-scale dyadic interactive audio-visual dialogue (DIAD) datasets provide fundamental data resources for developing humanoid interactive virtual agents and digital humans. However, collecting such data is time-consuming, expensive, and ethically sensitive. To address this, we propose CHAT, a new dyadic interactive audio-visual dialogue generation (DIADG) framework that generates diverse, paired, and mutually responsive speech-face dialogue clips from a single textual prompt. CHAT unifies large language models and talking face models with interactive audio and facial behaviour refinement modules, enabling the generation of aligned dyadic dialogue clips with diverse contents and facial identities. Experiments show that CHAT outperforms existing related methods designed for similar tasks under both objective and subjective evaluations. Moreover, our synthesised CHAT-AVD-50k dataset serves as effective pre-training data for downstream interactive head generation, consistently improving PerFRDiff and ReactDiff on REACT 2024. CHAT offers a scalable alternative to the costly and ethically sensitive collection of real dyadic interaction data.

Original language	English
Title of host publication	Computer Vision – ECCV 2026
Subtitle of host publication	19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings, Part XLIV
Editors	Paolo Favaro, Zuzana Kukelova, Atsuto Maki, Anna Rohrbach, Konrad Schindler, Federico Tombari
Place of Publication	Cham, Switzerland
Publisher	Springer
Pages	597-616
Number of pages	21
Edition	1st
ISBN (Electronic)	9783032372529
ISBN (Print)	9783032372512
DOIs	https://doi.org/10.1007/978-3-032-37252-9_33
Publication status	Published - 10 Sept 2026
Event	The 19th European Conference on Computer Vision - ECCV 2026 - Malmö Arena and Malmömässan, Malmö, Sweden Duration: 8 Sept 2026 → 12 Sept 2026 Conference number: 19th https://eccv.ecva.net/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	17044
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	The 19th European Conference on Computer Vision - ECCV 2026
Abbreviated title	ECCV 2026
Country/Territory	Sweden
City	Malmö
Period	8/09/26 → 12/09/26
Internet address	https://eccv.ecva.net/

Keywords

Audio-visual Human Behaviour Synthesis
Human Behavioural Response Generation
Diffusion Model
Reaction Generation

Access to Document

10.1007/978-3-032-37252-9_33

2607.02799v1Accepted author manuscript, 4.14 MBLicence: CC BY

Cite this

Song, J., Guasch, L., He, X., Yang, Z., Yuan, Y., Xie, W., Shen, L., Lin, H., Liu, S., Pang, W., & Song, S. (2026). Conversational Human Audio-visual Talking Dialogue Generation. In P. Favaro, Z. Kukelova, A. Maki, A. Rohrbach, K. Schindler, & F. Tombari (Eds.), Computer Vision – ECCV 2026: 19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings, Part XLIV (1st ed., pp. 597-616). (Lecture Notes in Computer Science ; Vol. 17044). Springer. https://doi.org/10.1007/978-3-032-37252-9_33

Song, Junhao ; Guasch, Lluis ; He, Xilin et al. / Conversational Human Audio-visual Talking Dialogue Generation. Computer Vision – ECCV 2026: 19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings, Part XLIV. editor / Paolo Favaro ; Zuzana Kukelova ; Atsuto Maki ; Anna Rohrbach ; Konrad Schindler ; Federico Tombari. 1st. ed. Cham, Switzerland : Springer, 2026. pp. 597-616 (Lecture Notes in Computer Science ).

@inbook{80ad6c3848964fd8af72cd0bad52b924,

title = "Conversational Human Audio-visual Talking Dialogue Generation",

abstract = "Large-scale dyadic interactive audio-visual dialogue (DIAD) datasets provide fundamental data resources for developing humanoid interactive virtual agents and digital humans. However, collecting such data is time-consuming, expensive, and ethically sensitive. To address this, we propose CHAT, a new dyadic interactive audio-visual dialogue generation (DIADG) framework that generates diverse, paired, and mutually responsive speech-face dialogue clips from a single textual prompt. CHAT unifies large language models and talking face models with interactive audio and facial behaviour refinement modules, enabling the generation of aligned dyadic dialogue clips with diverse contents and facial identities. Experiments show that CHAT outperforms existing related methods designed for similar tasks under both objective and subjective evaluations. Moreover, our synthesised CHAT-AVD-50k dataset serves as effective pre-training data for downstream interactive head generation, consistently improving PerFRDiff and ReactDiff on REACT 2024. CHAT offers a scalable alternative to the costly and ethically sensitive collection of real dyadic interaction data.",

keywords = "Audio-visual Human Behaviour Synthesis, Human Behavioural Response Generation, Diffusion Model, Reaction Generation",

author = "Junhao Song and Lluis Guasch and Xilin He and Zhongyu Yang and Yingfang Yuan and Weicheng Xie and Linlin Shen and Haijun Lin and Shizhe Liu and Wei Pang and Siyang Song",

year = "2026",

month = sep,

day = "10",

doi = "10.1007/978-3-032-37252-9\_33",

language = "English",

isbn = "9783032372512",

series = "Lecture Notes in Computer Science ",

publisher = "Springer",

pages = "597--616",

editor = "Paolo Favaro and Zuzana Kukelova and Atsuto Maki and Anna Rohrbach and Konrad Schindler and Federico Tombari",

booktitle = "Computer Vision – ECCV 2026",

address = "Germany",

edition = "1st",

note = "The 19th European Conference on Computer Vision - ECCV 2026, ECCV 2026 ; Conference date: 08-09-2026 Through 12-09-2026",

url = "https://eccv.ecva.net/",

}

Song, J, Guasch, L, He, X, Yang, Z, Yuan, Y, Xie, W, Shen, L, Lin, H, Liu, S, Pang, W & Song, S 2026, Conversational Human Audio-visual Talking Dialogue Generation. in P Favaro, Z Kukelova, A Maki, A Rohrbach, K Schindler & F Tombari (eds), Computer Vision – ECCV 2026: 19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings, Part XLIV. 1st edn, Lecture Notes in Computer Science , vol. 17044, Springer, Cham, Switzerland, pp. 597-616, The 19th European Conference on Computer Vision - ECCV 2026, Malmö, Sweden, 8/09/26. https://doi.org/10.1007/978-3-032-37252-9_33

Conversational Human Audio-visual Talking Dialogue Generation. / Song, Junhao; Guasch, Lluis; He, Xilin et al.
Computer Vision – ECCV 2026: 19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings, Part XLIV. ed. / Paolo Favaro; Zuzana Kukelova; Atsuto Maki; Anna Rohrbach; Konrad Schindler; Federico Tombari. 1st. ed. Cham, Switzerland: Springer, 2026. p. 597-616 (Lecture Notes in Computer Science ; Vol. 17044).

Research output: Chapter in Book/Report/Conference proceeding › Chapter › peer-review

TY - CHAP

T1 - Conversational Human Audio-visual Talking Dialogue Generation

AU - Song, Junhao

AU - Guasch, Lluis

AU - He, Xilin

AU - Yang, Zhongyu

AU - Yuan, Yingfang

AU - Xie, Weicheng

AU - Shen, Linlin

AU - Lin, Haijun

AU - Liu, Shizhe

AU - Pang, Wei

AU - Song, Siyang

N1 - Conference code: 19th

PY - 2026/9/10

Y1 - 2026/9/10

N2 - Large-scale dyadic interactive audio-visual dialogue (DIAD) datasets provide fundamental data resources for developing humanoid interactive virtual agents and digital humans. However, collecting such data is time-consuming, expensive, and ethically sensitive. To address this, we propose CHAT, a new dyadic interactive audio-visual dialogue generation (DIADG) framework that generates diverse, paired, and mutually responsive speech-face dialogue clips from a single textual prompt. CHAT unifies large language models and talking face models with interactive audio and facial behaviour refinement modules, enabling the generation of aligned dyadic dialogue clips with diverse contents and facial identities. Experiments show that CHAT outperforms existing related methods designed for similar tasks under both objective and subjective evaluations. Moreover, our synthesised CHAT-AVD-50k dataset serves as effective pre-training data for downstream interactive head generation, consistently improving PerFRDiff and ReactDiff on REACT 2024. CHAT offers a scalable alternative to the costly and ethically sensitive collection of real dyadic interaction data.

AB - Large-scale dyadic interactive audio-visual dialogue (DIAD) datasets provide fundamental data resources for developing humanoid interactive virtual agents and digital humans. However, collecting such data is time-consuming, expensive, and ethically sensitive. To address this, we propose CHAT, a new dyadic interactive audio-visual dialogue generation (DIADG) framework that generates diverse, paired, and mutually responsive speech-face dialogue clips from a single textual prompt. CHAT unifies large language models and talking face models with interactive audio and facial behaviour refinement modules, enabling the generation of aligned dyadic dialogue clips with diverse contents and facial identities. Experiments show that CHAT outperforms existing related methods designed for similar tasks under both objective and subjective evaluations. Moreover, our synthesised CHAT-AVD-50k dataset serves as effective pre-training data for downstream interactive head generation, consistently improving PerFRDiff and ReactDiff on REACT 2024. CHAT offers a scalable alternative to the costly and ethically sensitive collection of real dyadic interaction data.

KW - Audio-visual Human Behaviour Synthesis

KW - Human Behavioural Response Generation

KW - Diffusion Model

KW - Reaction Generation

U2 - 10.1007/978-3-032-37252-9_33

DO - 10.1007/978-3-032-37252-9_33

M3 - Chapter

SN - 9783032372512

T3 - Lecture Notes in Computer Science

SP - 597

EP - 616

BT - Computer Vision – ECCV 2026

A2 - Favaro, Paolo

A2 - Kukelova, Zuzana

A2 - Maki, Atsuto

A2 - Rohrbach, Anna

A2 - Schindler, Konrad

A2 - Tombari, Federico

PB - Springer

CY - Cham, Switzerland

T2 - The 19th European Conference on Computer Vision - ECCV 2026

Y2 - 8 September 2026 through 12 September 2026

ER -

Song J, Guasch L, He X, Yang Z, Yuan Y, Xie W et al. Conversational Human Audio-visual Talking Dialogue Generation. In Favaro P, Kukelova Z, Maki A, Rohrbach A, Schindler K, Tombari F, editors, Computer Vision – ECCV 2026: 19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings, Part XLIV. 1st ed. Cham, Switzerland: Springer. 2026. p. 597-616. (Lecture Notes in Computer Science ). Epub 2026 Sept 9. doi: 10.1007/978-3-032-37252-9_33

Conversational Human Audio-visual Talking Dialogue Generation

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Cite this