sign-lang@LREC Anthology

Evaluating Inter-Annotator Agreement for Non-Manual Markers in Sign Languages

Esselink, Lyke ORCID button Esselink, Lyke | Oomen, Marloes ORCID button Oomen, Marloes | Roelofsen, Floris ORCID button Roelofsen, Floris

Volume:
Proceedings of the LREC-COLING 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources
Venue:
Torino, Italy
Date:
25 May 2024
Pages:
341–351
Publisher:
ELRA Language Resources Association (ELRA) and the International Committee on Computational Linguistics (ICCL)
License:
CC BY-NC 4.0
sign-lang ID:
24042
ACL ID:
2024.signlang-1.7
ISBN:
978-2-493814-30-2

Abstract

This paper is part of a larger project that aims to create a standardized procedure for annotating non-manual markers (NMMs) in sign language data. The paper describes two approaches to evaluating inter-annotator agreement, the event-based approach and the frame-based approach, and uses a combination of these two approaches to evaluate the annotation guidelines introduced in Oomen et al. (2023). The evaluation reveals that for several labels in the annotation scheme inter-annotator agreement is rather low. This indicates that the annotations guidelines need to be further improved. We present concrete recommendations for how this may be achieved, and intend to implement these recommendations in future work. All data and analysis scripts are available.

Document Download

Paper PDF Poster BibTeX File + Abstract

BibTeX Export

@inproceedings{esselink:24042:sign-lang:lrec,
  author    = {Esselink, Lyke and Oomen, Marloes and Roelofsen, Floris},
  title     = {Evaluating Inter-Annotator Agreement for Non-Manual Markers in Sign Languages},
  pages     = {341--351},
  editor    = {Efthimiou, Eleni and Fotinea, Stavroula-Evita and Hanke, Thomas and Hochgesang, Julie A. and Mesch, Johanna and Schulder, Marc},
  booktitle = {Proceedings of the {LREC-COLING} 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources},
  maintitle = {2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation ({LREC-COLING} 2024)},
  publisher = {{ELRA Language Resources Association (ELRA) and the International Committee on Computational Linguistics (ICCL)}},
  address   = {Torino, Italy},
  day       = {25},
  month     = may,
  year      = {2024},
  isbn      = {978-2-493814-30-2},
  language  = {english},
  url       = {https://www.sign-lang.uni-hamburg.de/lrec/pub/24042.pdf}
}
Something missing or wrong?