sign-lang@LREC Anthology

Changing the Representation: Examining Language Representation for Neural Sign Language Production

Walsh, Harry | Saunders, Ben | Bowden, Richard


Volume:
Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology: The Junction of the Visual and the Textual: Challenges and Perspectives
Venue:
Marseille, France
Date:
24 June 2022
Pages:
117–124
Publisher:
European Language Resources Association (ELRA)
License:
CC BY-NC 4.0
ACL ID:
2022.sltat-1.18
ISBN:
979-10-95546-82-5

Content Categories

Projects:
EASIER, ExTOL, SMILE II
Languages:
German Sign Language
Corpora:
DGS Corpus, RWTH-PHOENIX Weather 2014 T

Abstract

Neural Sign Language Production (SLP) aims to automatically translate from spoken language sentences to sign language videos. Historically the SLP task has been broken into two steps; Firstly, translating from a spoken language sentence to a gloss sequence and secondly, producing a sign language video given a sequence of glosses. In this paper we apply Natural Language Processing techniques to the first step of the SLP pipeline. We use language models such as BERT and Word2Vec to create better sentence level embeddings, and apply several tokenization techniques, demonstrating how these improve performance on the low resource translation task of Text to Gloss. We introduce Text to HamNoSys (T2H) translation, and show the advantages of using a phonetic representation for sign language translation rather than a sign level gloss representation. Furthermore, we use HamNoSys to extract the hand shape of a sign and use this as additional supervision during training, further increasing the performance on T2H. Assembling best practise, we achieve a BLEU-4 score of 26.99 on the MineDGS dataset and 25.09 on PHOENIX14T, two new state-of-the-art baselines.

Document Download

Paper PDF BibTeX File+ Abstract

BibTeX Export

@inproceedings{walsh:70007:sltat:lrec,
  author    = {Walsh, Harry and Saunders, Ben and Bowden, Richard},
  title     = {Changing the Representation: Examining Language Representation for Neural Sign Language Production},
  pages     = {117--124},
  editor    = {Efthimiou, Eleni and Fotinea, Stavroula-Evita and Hanke, Thomas and McDonald, John C. and Shterionov, Dimitar and Wolfe, Rosalee},
  booktitle = {Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology: The Junction of the Visual and the Textual: Challenges and Perspectives},
  maintitle = {13th International Conference on Language Resources and Evaluation ({LREC} 2022)},
  publisher = {{European Language Resources Association (ELRA)}},
  address   = {Marseille, France},
  day       = {24},
  month     = jun,
  year      = {2022},
  isbn      = {979-10-95546-82-5},
  language  = {english},
  url       = {http://www.lrec-conf.org/proceedings/lrec2022/workshops/sltat/pdf/2022.sltat-1.18}
}
Something missing or wrong?