Contrasting a semiotic conceptualization of translation with AI text production: The case of audio captioning

Abstract

Using a semiotically-informed material approach to the study of translation, this paper analyses an artificial intelligence (AI) system developed for automatic audio captioning (AAC), which is the automated production of written descriptions for non-lingual environmental sounds. Comparing human and AI text production processes against a semiotic framework suggests that AI uses computational methods to reach textual outcomes which humans arrive at through semiotic means. Our analysis of sound description examples produced by an AAC system makes it apparent that this distinction is useful in articulating the complex relationship between human and AI translation processes. Acknowledging the central role of semiotic meaning-construction in human text production and its arguable absence in AI computational processes allows for AI processes to be discussed under a translational framework, while still recognizing their fundamental differences from comparable human translation processes. Further, audio captioning provides a clear example of a translation task where non-lingual content must be considered on equal terms with lingual text, and our discussions illustrate how this can be achieved in computational and semiotic processes alike. Overall, this paper promotes a nuanced understanding of meaning in text production and suggests multiple fruitful points of convergence and divergence between translation theory and AI research.