My Master’s thesis investigated how people embody sounds to communicate on them using their gestures and vocalizations.
People often rely on gestures to communicate on sounds, often combining them with idiosyncratic vocalizations to express some of their qualities. We were interested in studying the links between gestures and vocalizations in sound communication, drawing inspiration from speech studies.
We first led qualitative annotation of an audio-visual database of gestural and vocal imitations of sounds to establish a set of hypotheses. We then led a controlled experiment and performed quantitative analysis of motion and sound data harvested. Our results suggested that people embody sounds using gestures that metaphorically express one salient feature of sound, along with vocalizations that attempt to reproduce all sound features as faithfully as humanly possible.
The project was developed with Guillaume Lemaitre, Frédéric Bevilacqua, Patrick Susini, and Jules Françoise in collaboration with the PDS and ISMM groups of IRCAM, in the context of the SkAT-VG European research project and the Sorbonne Université Master’s program in Engineering.