sign-lang@LREC Anthology

Sign Language Production With Avatar Layering: A Critical Use Case over Rare Words

Kim, Jung-Ho | Hwang, Eui Jun | Cho, Sukmin | Lee, Du Hui | Park, Jong

Proceedings of the 13th International Conference on Language Resources and Evaluation (LREC 2022)
Marseille, France
20 to 25 June 2022
European Language Resources Association (ELRA)
CC BY-NC 4.0


Sign language production (SLP) is the process of generating sign language videos from spoken language expressions. Since sign languages are highly under-resourced, existing vision-based SLP approaches suffer from out-of-vocabulary (OOV) and test-time generalization problems and thus generate low-quality translations. To address these problems, we introduce an avatar-based SLP system composed of a sign language translation (SLT) model and an avatar animation generation module. Our Transformer-based SLT model utilizes two additional strategies to resolve these problems: named entity transformation to reduce OOV tokens and context vector generation using a pretrained language model (e.g., BERT) to reliably train the decoder. Our system is validated on a new Korean-Korean Sign Language (KSL) dataset of weather forecasts and emergency announcements. Our SLT model achieves an 8.77 higher BLEU-4 score and a 4.57 higher ROUGE-L score over those of our baseline model. In a user evaluation, 93.48% of named entities were successfully identified by participants, demonstrating marked improvement on OOV issues.

Document Download

Paper PDF BibTeX File BibTeX File w/Abstract

BibTeX Export

  author    = {Kim, Jung-Ho and Hwang, Eui Jun and Cho, Sukmin and Lee, Du Hui and Park, Jong},
  title     = {Sign Language Production With Avatar Layering: A Critical Use Case over Rare Words},
  pages     = {1519--1528},
  editor    = {Calzolari, Nicoletta and Fr{\'e}d{\'e}ric B{\'e}chet and Blache, Philippe and Choukri, Khalid and Cieri, Christopher and Declerck, Thierry and Goggi, Sara and Isahara, Hitoshi and Maegaard, Bente and Mariani, Joseph and Mazo, H{\'e}l{\`e}ne and Odijk, Jan and Piperidis, Stelios},
  booktitle = {13th International Conference on Language Resources and Evaluation ({LREC} 2022)},
  publisher = {{European Language Resources Association (ELRA)}},
  address   = {Marseille, France},
  day       = {20--25},
  month     = jun,
  year      = {2022},
  language  = {english},
  url       = {}
Something missing or wrong?