Towards Integrating Pose Estimation with Neuroimaging for the Analysis of Signed Language Video Stimuli

Vandenitte, Sébastien | Hernández, Doris | Keränen, Jarkko | Jantunen, Tommi | Puupponen, Anna

Volume:: Proceedings of the LREC2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion
Venue:: Palma, Mallorca, Spain
Date:: 16 May 2026
Pages:: 477–483
Publisher:: European Language Resources Association (ELRA)
Licence:: CC BY-NC 4.0
sign-lang ID:: 26018
ISBN:: 978-2-493814-82-1

Abstract

We present our project revisiting the video stimuli of an EEG study in Finnish Sign Language to ask whether kinematic properties of the videos impacted their processing by study participants. For each stimulus, an average measure of brain responses across participants is computed. To analyse movement properties in the video stimuli, we rely on MediaPipe for pose estimation. We subsequently report on our project to perform an exploratory analysis of the kinematic properties of the videos which may affect their processing. We focus on several landmarks: the signer's right and left wrists, nose, and upper torso. Our goal is to obtain a kinematic profile of each stimulus video using several average kinematic variables: velocity and acceleration for all selected landmarks, distance between the wrists, and surface covered by the triangular area defined by the left hand, the right hand, and the nose. We conclude by discussing the potential benefits and limitations of this methodological approach.

Document Download

Paper PDF BibTeX File + Abstract

Cite as

Citation in ACL Citation Format

Sébastien Vandenitte, Doris Hernández, Jarkko Keränen, Tommi Jantunen, Anna Puupponen. 2026. Towards Integrating Pose Estimation with Neuroimaging for the Analysis of Signed Language Video Stimuli. In Proceedings of the LREC2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion, pages 477–483, Palma, Mallorca, Spain. European Language Resources Association (ELRA).

BibTeX Export

@inproceedings{vandenitte:26018:sign-lang:lrec,
  author    = {Vandenitte, S{\'e}bastien and Hern{\'a}ndez, Doris and Ker{\"a}nen, Jarkko and Jantunen, Tommi and Puupponen, Anna},
  title     = {Towards Integrating Pose Estimation with Neuroimaging for the Analysis of Signed Language Video Stimuli},
  pages     = {477--483},
  editor    = {Efthimiou, Eleni and Fotinea, Stavroula-Evita and Hanke, Thomas and Hochgesang, Julie A. and Mesch, Johanna and Schulder, Marc},
  booktitle = {Proceedings of the {LREC2026} 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion},
  maintitle = {15th International Conference on Language Resources and Evaluation ({LREC} 2026)},
  publisher = {{European Language Resources Association (ELRA)}},
  address   = {Palma, Mallorca, Spain},
  day       = {16},
  month     = may,
  year      = {2026},
  isbn      = {978-2-493814-82-1},
  language  = {english},
  url       = {https://www.sign-lang.uni-hamburg.de/lrec/pub/26018.html}
}