Iterated Learning Used for Optimization of Hand-based Gestural Control in AR

Jacek Matulewski, Piotr Krzysztof Ablewski, Monika Boruta-Żywiczyńska, Jacek Jabłoński, Łukasz Sikorski

A pilot study examines whether iterated learning can optimize hand gestures used for controlling augmented reality software. There are many studies on improving the quality of gesture recognition, both static and dynamic, simple and sequential gestures; however, they are mainly focused on optimizing of machine learning algorithms and technical aspects. We found no attempts to apply methods such as iterative learning on gesture optimization. This work is grounded in research treating iterated learning as a mechanism of cultural transmission and language evolution, where repeated intergenerational transfer may lead to simplification and adaptation of signs, as well as on gesture-evolution studies suggesting that repeated transmission can streamline movement and conventionalize manual signals. The study was conducted with 28 participants assigned to four learning chains of seven generations each. Participants learned and reproduced 33 predefined AR gestures, which were then passed to the next participant in the chain. The procedure combined first-person video transmission, motion-capture recording with Rokoko Smartgloves, usability ratings, recall delay measures, and attention assessment with a Go/No-Go task; temperament was additionally measured with the EAS-D questionnaire. The findings do not support the expected optimization effect. No gesture remained unchanged across all chains and generations; the proportion of preserved gestures declined to about 30%, and neither movement volume nor path length showed a systematic reduction over time. Subjective usability also decreased across generations, with a significant negative association between generation number and average rating (r = -0.43, p = 0.0278). Measures related to recall time and attentional fatigue also failed to reveal a consistent improvement trend. Therefore, the main contribution of the study is a negative but theoretically and practically important result: iterated learning, despite its promise in language-evolution research, does not appear to be an effective method for optimizing gesture sets for AR interaction in this pilot setting. Instead, gesture usability appears to depend more strongly on symbolic transparency, gesture complexity, and individual user characteristics than on transmission alone.