Recordings from the DGS-Korpus project with 330 informants confirm that at least for German Sign Language (DGS) you hardly find longer stretches of signing not accompanied by any mouth activity. Independent of whether you consider mouth activity while signing as part of the sign language proper or as a parallel system interacting with sign language to jointly transport meaning, mouth activity is part of the linguistic system used by signers and should be treated as such by any corpus approach. In a purely bottom-up approach an annotation practice used for mouth activities would try to describe the phenomena and leave it to a second step to classify (e.g. between mouthing and mouth gestures) and relate (e.g. to spoken language words). For practical reasons, however, the first step is often skipped, and separate coding systems are applied to what is categorised either as mouthing derived from spoken language or mouth gesture where there is no obvious connection between the meaning expressed and any spoken language words expressing that same meaning. This happens not only for time (=budget) reasons, but also because it is difficult for coders to describe mouth visemes precisely if the sign/mouth combo already suggests what is to be seen on the mouth. While there are established coding procedures to avoid influence as far as possible (like only showing the signer’s face, provided video quality is good enough), they make the approach very time-consuming, even if not counting quality assurance measures like inter-transcriber agreement. Some projects undertaken at the IDGS in Hamburg therefore leave it with a spoken-language-driven approach: The mouth activity is classified as either mouth gesture or mouthing, and in the latter case the German word is noted down that a competent DGS signer “reads” from the lips, i.e. that word from the set of words to be expected with the co-temporal sign in its context that matches the observation. Standard orthography is used unless there is a substantial deviation. For mouth gestures, holistic labels are used. These two extremes span a whole spectrum of coding approaches that can be used for mouth activities. iLex, the Hamburg sign language annotation workbench, tries to support the whole range of solutions as good as possible. The poster w/ demo shows a variety of approaches actually in use or on the horizon and what iLex has to offer for each of those, from more time-series like systems to those evaluating co-occurrence and semantic relatedness, from novice-friendly decision trees to expert-only modes. Inter-transcriber agreement data on the examples given clearly show that a thorough analysis of data quality has to go beyond such measures.
@inproceedings{hanke:14029:sign-lang:lrec,
author = {Hanke, Thomas},
title = {Annotation of mouth activities with {iLex}},
pages = {67--70},
editor = {Crasborn, Onno and Efthimiou, Eleni and Fotinea, Stavroula-Evita and Hanke, Thomas and Kristoffersen, Jette and Mesch, Johanna},
booktitle = {Proceedings of the {LREC2014} 6th Workshop on the Representation and Processing of Sign Languages: Beyond the Manual Channel},
maintitle = {9th International Conference on Language Resources and Evaluation ({LREC} 2014)},
publisher = {{European Language Resources Association (ELRA)}},
address = {Reykjavik, Iceland},
day = {31},
month = may,
year = {2014},
language = {english},
url = {https://www.sign-lang.uni-hamburg.de/lrec/pub/14029.pdf}
}
