StyleBabel: Artistic Style Tagging and Captioning

Dan Ruta; Andrew Gilbert; Pranav Aggarwal; Naveen Marri; Ajinkya Kale; Jo Briggs; Chris Speed; Halin Jin; Baldo Faieta; Alex Filipkowski; Zhe Lin; John Collomosse

doi:10.1007/978-3-031-20074-8_13

StyleBabel: Artistic Style Tagging and Captioning

Dan Ruta^*, Andrew Gilbert, Pranav Aggarwal, Naveen Marri, Ajinkya Kale, Jo Briggs, Chris Speed, Halin Jin, Baldo Faieta, Alex Filipkowski , Zhe Lin, John Collomosse

^*Corresponding author for this work

Design

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by ‘Grounded Theory’: a qualitative approach that enables annotation while co-evolving a shared language for fine-grained artistic style attribute description. We demonstrate several downstream tasks for StyleBabel, adapting the recent ALADIN architecture for fine-grained style similarity, to train cross-modal embeddings for: 1) free-form tag generation; 2) natural language description of artistic style; 3) fine-grained text search of style. To do so, we extend ALADIN with recent advances in Visual Transformer (ViT) and cross-modal representation learning, achieving a state of the art accuracy in fine-grained style retrieval.

Original language	English
Title of host publication	Computer Vision – ECCV 2022
Subtitle of host publication	17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
Place of Publication	Cham, Switzerland
Publisher	Springer
Pages	219-236
Number of pages	18
Volume	13668
ISBN (Electronic)	9783031200748
ISBN (Print)	9783031200731
DOIs	https://doi.org/10.1007/978-3-031-20074-8_13
Publication status	Published - 2022
Event	ECCV 2022: European Conference on Computer Vision (ECCV) - Expo Tel Aviv, Tel Aviv, Israel Duration: 23 Oct 2022 → 27 Oct 2022 https://eccv2022.ecva.net/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	23/10/22 → 27/10/22
Internet address	https://eccv2022.ecva.net/

Access to Document

10.1007/978-3-031-20074-8_13

Cite this

Ruta, D., Gilbert, A., Aggarwal, P., Marri, N., Kale, A., Briggs, J., Speed, C., Jin, H., Faieta, B., Filipkowski , A., Lin, Z., & Collomosse, J. (2022). StyleBabel: Artistic Style Tagging and Captioning. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII (Vol. 13668, pp. 219-236). (Lecture Notes in Computer Science). Springer. https://doi.org/10.1007/978-3-031-20074-8_13

Ruta, Dan ; Gilbert, Andrew ; Aggarwal, Pranav et al. / StyleBabel : Artistic Style Tagging and Captioning. Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII. editor / Shai Avidan ; Gabriel Brostow ; Moustapha Cissé ; Giovanni Maria Farinella ; Tal Hassner. Vol. 13668 Cham, Switzerland : Springer, 2022. pp. 219-236 (Lecture Notes in Computer Science).

@inproceedings{b065710974724686b4558d5b083ea215,

title = "StyleBabel: Artistic Style Tagging and Captioning",

abstract = "We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by {\textquoteleft}Grounded Theory{\textquoteright}: a qualitative approach that enables annotation while co-evolving a shared language for fine-grained artistic style attribute description. We demonstrate several downstream tasks for StyleBabel, adapting the recent ALADIN architecture for fine-grained style similarity, to train cross-modal embeddings for: 1) free-form tag generation; 2) natural language description of artistic style; 3) fine-grained text search of style. To do so, we extend ALADIN with recent advances in Visual Transformer (ViT) and cross-modal representation learning, achieving a state of the art accuracy in fine-grained style retrieval.",

keywords = "Datasets and evaluation, Image and video retrieval, Vision + language, Vision applications and systems",

author = "Dan Ruta and Andrew Gilbert and Pranav Aggarwal and Naveen Marri and Ajinkya Kale and Jo Briggs and Chris Speed and Halin Jin and Baldo Faieta and Alex Filipkowski and Zhe Lin and John Collomosse",

year = "2022",

doi = "10.1007/978-3-031-20074-8_13",

language = "English",

isbn = "9783031200731",

volume = "13668",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "219--236",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e} and Farinella, {Giovanni Maria} and Tal Hassner",

booktitle = "Computer Vision – ECCV 2022",

address = "Germany",

note = "ECCV 2022 : European Conference on Computer Vision (ECCV) ; Conference date: 23-10-2022 Through 27-10-2022",

url = "https://eccv2022.ecva.net/",

}

Ruta, D, Gilbert, A, Aggarwal, P, Marri, N, Kale, A, Briggs, J, Speed, C, Jin, H, Faieta, B, Filipkowski , A, Lin, Z & Collomosse, J 2022, StyleBabel: Artistic Style Tagging and Captioning. in S Avidan, G Brostow, M Cissé, GM Farinella & T Hassner (eds), Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII. vol. 13668, Lecture Notes in Computer Science, Springer, Cham, Switzerland, pp. 219-236, ECCV 2022, Tel Aviv, Israel, 23/10/22. https://doi.org/10.1007/978-3-031-20074-8_13

StyleBabel : Artistic Style Tagging and Captioning. / Ruta, Dan; Gilbert, Andrew; Aggarwal, Pranav et al.

Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII. ed. / Shai Avidan; Gabriel Brostow; Moustapha Cissé; Giovanni Maria Farinella; Tal Hassner. Vol. 13668 Cham, Switzerland : Springer, 2022. p. 219-236 (Lecture Notes in Computer Science).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - StyleBabel

T2 - ECCV 2022

AU - Ruta, Dan

AU - Gilbert, Andrew

AU - Aggarwal, Pranav

AU - Marri, Naveen

AU - Kale, Ajinkya

AU - Briggs, Jo

AU - Speed, Chris

AU - Jin, Halin

AU - Faieta, Baldo

AU - Filipkowski , Alex

AU - Lin, Zhe

AU - Collomosse, John

PY - 2022

Y1 - 2022

N2 - We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by ‘Grounded Theory’: a qualitative approach that enables annotation while co-evolving a shared language for fine-grained artistic style attribute description. We demonstrate several downstream tasks for StyleBabel, adapting the recent ALADIN architecture for fine-grained style similarity, to train cross-modal embeddings for: 1) free-form tag generation; 2) natural language description of artistic style; 3) fine-grained text search of style. To do so, we extend ALADIN with recent advances in Visual Transformer (ViT) and cross-modal representation learning, achieving a state of the art accuracy in fine-grained style retrieval.

AB - We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by ‘Grounded Theory’: a qualitative approach that enables annotation while co-evolving a shared language for fine-grained artistic style attribute description. We demonstrate several downstream tasks for StyleBabel, adapting the recent ALADIN architecture for fine-grained style similarity, to train cross-modal embeddings for: 1) free-form tag generation; 2) natural language description of artistic style; 3) fine-grained text search of style. To do so, we extend ALADIN with recent advances in Visual Transformer (ViT) and cross-modal representation learning, achieving a state of the art accuracy in fine-grained style retrieval.

KW - Datasets and evaluation

KW - Image and video retrieval

KW - Vision + language

KW - Vision applications and systems

UR - https://arxiv.org/abs/2203.05321

UR - https://www.ecva.net/papers.php

U2 - 10.1007/978-3-031-20074-8_13

DO - 10.1007/978-3-031-20074-8_13

M3 - Conference contribution

SN - 9783031200731

VL - 13668

T3 - Lecture Notes in Computer Science

SP - 219

EP - 236

BT - Computer Vision – ECCV 2022

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

A2 - Farinella, Giovanni Maria

A2 - Hassner, Tal

PB - Springer

CY - Cham, Switzerland

Y2 - 23 October 2022 through 27 October 2022

ER -

Ruta D, Gilbert A, Aggarwal P, Marri N, Kale A, Briggs J et al. StyleBabel: Artistic Style Tagging and Captioning. In Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors, Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII. Vol. 13668. Cham, Switzerland: Springer. 2022. p. 219-236. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-031-20074-8_13

StyleBabel: Artistic Style Tagging and Captioning

Abstract

Publication series

Conference

Access to Document

Other files and links

Embargoed Document

Fingerprint

Cite this