This project studied online clustering techniques for movement analysis and their application to motion-sound mapping.
We investigated Gaussian Mixture Models (GMM), a probabilistic model for representing groups of similar properties in data. In particular, we studied their ability to learn online, that is, to cluster data as it becomes available without human supervision. Our wish was to relieve humans of translating the tacit knowledge of their movements in code and have an artificial intelligence system infer it.
We implemented two interaction modes with online GMMs in the context of motion-sound mapping design, named « guiding » and « shaping ». These modes let users interact with an adaptive, reflective model that can be used for both instrument design and improvisational practices.
The project was developed with Frédéric Bevilacqua and Jules Françoise in collaboration with the ISMM group of IRCAM and School of Interactive Arts and Technology (SFU), in the context of the Sorbonne Université Doctorate in Computing.
Paper at NIME (2017)