Grounded and well-rounded: a methodological approach to the study of cross-modal and cross-lingual grounding

• DOI: 10.18653/v1/2023.findings-emnlp.736
• Publication Date: 2023-12-10T21:58:19Z
• Authors: Timothee Mickus; Elaine Zosa; Denis Paperno
• Keywords: Modalities; Position paper

Grounding has been argued to be a crucial component towards the development of more complete and truly semantically competent artificial intelligence systems. Literature divided into two camps: While some argue that grounding allows for qualitatively different generalizations, others believe it can compensated by mono-modal data quantity. Limited empirical evidence emerged or against either position, which we is due methodological challenges come with studying its effects on NLP In this paper, establish framework what are—if any—of providing models richer input sources than text-only. The crux lies in construction comparable samples populations trained modalities, so tease apart qualitative from quantifiable model performances. Experiments using reveal differences behavior between cross-modally grounded, cross-lingually ungrounded models, measure both at global dataset level as well specific word representations, depending how concrete their semantics is.