Multi3Gen

Multi-task, Multilingual, Multimodal Language Generation

Dr. Calixto has actively helped shape the Multi3Generation COST Action, a research network dedicated to the topic of multilingual and multimodal natural language generation. Dr. Calixto is currently the Dutch representative in the Action’s Management Committee.

Multi3Generation has enabled collaborations that led to many fruitful outcomes, including papers published at ICLR, ACL, and the Journal of AI Research (JAIR). Please refer to our publications for more details.

References

2024

ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models

Ilker Kesen , Andrea Pedrotti , Mustafa Dogan , Michele Cafagna , Emre Can Acikgoz , Letitia Parcalabescu , Iacer Calixto , Anette Frank , and 3 more authors

In The Twelfth International Conference on Learning Representations , Jul 2024

Bib HTML

@inproceedings{kesen-etal-2024vilma,
  title = {Vi{LMA}: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models},
  author = {Kesen, Ilker and Pedrotti, Andrea and Dogan, Mustafa and Cafagna, Michele and Acikgoz, Emre Can and Parcalabescu, Letitia and Calixto, Iacer and Frank, Anette and Gatt, Albert and Erdem, Aykut and Erkut, Erdem},
  booktitle = {The Twelfth International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=liuqDwmbQJ},
}

2022

Multi3Generation: Multitask, Multilingual, Multimodal Language Generation

Anabela Barreiro , José GC Souza , Albert Gatt , Mehul Bhatt , Elena Lloret , Aykut Erdem , Dimitra Gkatzia , Helena Moniz , and 7 more authors

In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation , Jun 2022

Abs Bib HTML

This paper presents the Multitask, Multilingual, Multimodal Language Generation COST Action – Multi3Generation (CA18231), an interdisciplinary network of research groups working on different aspects of language generation. This “meta-paper” will serve as reference for citations of the Action in future publications. It presents the objectives, challenges and a the links for the achieved outcomes.

@inproceedings{barreiro-etal-2022-multi3generation,
  title = {{M}ulti3{G}eneration: Multitask, Multilingual, Multimodal Language Generation},
  author = {Barreiro, Anabela and de Souza, Jos{\'e} GC and Gatt, Albert and Bhatt, Mehul and Lloret, Elena and Erdem, Aykut and Gkatzia, Dimitra and Moniz, Helena and Russo, Irene and Kepler, Fabio and Calixto, Iacer and Paprzycki, Marcin and Portet, Fran{\c{c}}ois and Augenstein, Isabelle and Alhasani, Mirela},
  booktitle = {Proceedings of the 23rd Annual Conference of the European Association for Machine Translation},
  month = jun,
  year = {2022},
  address = {Ghent, Belgium},
  publisher = {European Association for Machine Translation},
  url = {https://aclanthology.org/2022.eamt-1.63},
  pages = {347--348},
}

Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning

Erkut Erdem , Menekse Kuyu , Semih Yagcioglu , Anette Frank , Letitia Parcalabescu , Barbara Plank , Andrii Babii , Oleksii Turuta , and 10 more authors

J. Artif. Int. Res., May 2022

Abs Bib HTML

Developing artificial learning systems that can understand and generate natural language has been one of the long-standing goals of artificial intelligence. Recent decades have witnessed an impressive progress on both of these problems, giving rise to a new family of approaches. Especially, the advances in deep learning over the past couple of years have led to neural approaches to natural language generation (NLG). These methods combine generative language learning techniques with neural-networks based frameworks. With a wide range of applications in natural language processing, neural NLG (NNLG) is a new and fast growing field of research. In this state-of-the-art report, we investigate the recent developments and applications of NNLG in its full extent from a multidimensional view, covering critical perspectives such as multimodality, multilinguality, controllability and learning strategies. We summarize the fundamental building blocks of NNLG approaches from these aspects and provide detailed reviews of commonly used preprocessing steps and basic neural architectures. This report also focuses on the seminal applications of these NNLG models such as machine translation, description generation, automatic speech recognition, abstractive summarization, text simplification, question answering and generation, and dialogue generation. Finally, we conclude with a thorough discussion of the described frameworks by pointing out some open research directions.
@article{10.1613/jair.1.12918, author = {Erdem, Erkut and Kuyu, Menekse and Yagcioglu, Semih and Frank, Anette and Parcalabescu, Letitia and Plank, Barbara and Babii, Andrii and Turuta, Oleksii and Erdem, Aykut and Calixto, Iacer and Lloret, Elena and Apostol, Elena-Simona and Truic\u{a}, Ciprian-Octavian and \v{S}andrih, Branislava and Martin\v{c}i\'{c}-Ip\v{s}i\'{c}, Sanda and Berend, G\'{a}bor and Gatt, Albert and Korvel, Gr\u{a}zina}, title = {Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning}, year = {2022}, issue_date = {May 2022}, publisher = {AI Access Foundation}, address = {El Segundo, CA, USA}, volume = {73}, issn = {1076-9757}, url = {https://doi.org/10.1613/jair.1.12918}, doi = {10.1613/jair.1.12918}, journal = {J. Artif. Int. Res.}, month = may, numpages = {77}, alt_metric = {true}, dimensions = {true}, keywords = {natural language, neural networks} }
VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

Letitia Parcalabescu , Michele Cafagna , Lilitta Muradjan , Anette Frank , Iacer Calixto , and Albert Gatt

In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , May 2022

Abs Bib HTML Code

We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena. VALSE offers a suite of six tests covering various linguistic constructs. Solving these requires models to ground linguistic phenomena in the visual modality, allowing more fine-grained evaluations than hitherto possible. We build VALSE using methods that support the construction of valid foils, and report results from evaluating five widely-used V&L models. Our experiments suggest that current models have considerable difficulty addressing most phenomena. Hence, we expect VALSE to serve as an important benchmark to measure future progress of pretrained V&L models from a linguistic perspective, complementing the canonical task-centred V&L evaluations.
@inproceedings{parcalabescu-etal-2022-valse, title = {{VALSE}: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena}, author = {Parcalabescu, Letitia and Cafagna, Michele and Muradjan, Lilitta and Frank, Anette and Calixto, Iacer and Gatt, Albert}, booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, month = may, year = {2022}, address = {Dublin, Ireland}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.acl-long.567}, doi = {10.18653/v1/2022.acl-long.567}, pages = {8253--8280}, }