Sign Language Production With Avatar Layering: A Critical Use Case over Rare Words

Kim, Jung-Ho | Hwang, Eui Jun | Cho, Sukmin | Lee, Du Hui | Park, Jong C.

Volume:: Proceedings of the 13th International Conference on Language Resources and Evaluation (LREC 2022)
Venue:: Marseille, France
Date:: 20 to 25 June 2022
Pages:: 1519–1528
Publisher:: European Language Resources Association (ELRA)
License:: CC BY-NC 4.0
ACL ID:: 2022.lrec-1.163
ISBN:: 979-10-95546-72-6

Content Categories

Languages:: Korean Sign Language, Korean
Corpora:: Korean-KSL parallel corpus

Abstract

Sign language production (SLP) is the process of generating sign language videos from spoken language expressions. Since sign languages are highly under-resourced, existing vision-based SLP approaches suffer from out-of-vocabulary (OOV) and test-time generalization problems and thus generate low-quality translations. To address these problems, we introduce an avatar-based SLP system composed of a sign language translation (SLT) model and an avatar animation generation module. Our Transformer-based SLT model utilizes two additional strategies to resolve these problems: named entity transformation to reduce OOV tokens and context vector generation using a pretrained language model (e.g., BERT) to reliably train the decoder. Our system is validated on a new Korean-Korean Sign Language (KSL) dataset of weather forecasts and emergency announcements. Our SLT model achieves an 8.77 higher BLEU-4 score and a 4.57 higher ROUGE-L score over those of our baseline model. In a user evaluation, 93.48% of named entities were successfully identified by participants, demonstrating marked improvement on OOV issues.

Document Download

Paper PDF BibTeX File + Abstract

Cite as

Citation in ACL Citation Format

Jung-Ho Kim, Eui Jun Hwang, Sukmin Cho, Du Hui Lee, Jong C. Park. 2022. Sign Language Production With Avatar Layering: A Critical Use Case over Rare Words. In Proceedings of the 13th International Conference on Language Resources and Evaluation (LREC 2022), pages 1519–1528, Marseille, France. European Language Resources Association (ELRA).

BibTeX Export

@inproceedings{kim-etal-2022-layering:lrec,
  author    = {Kim, Jung-Ho and Hwang, Eui Jun and Cho, Sukmin and Lee, Du Hui and Park, Jong C.},
  title     = {Sign Language Production With Avatar Layering: A Critical Use Case over Rare Words},
  pages     = {1519--1528},
  editor    = {Calzolari, Nicoletta and B{\'e}chet, Fr{\'e}d{\'e}ric and Blache, Philippe and Choukri, Khalid and Cieri, Christopher and Declerck, Thierry and Goggi, Sara and Isahara, Hitoshi and Maegaard, Bente and Mariani, Joseph and Mazo, H{\'e}l{\`e}ne and Odijk, Jan and Piperidis, Stelios},
  booktitle = {13th International Conference on Language Resources and Evaluation ({LREC} 2022)},
  publisher = {{European Language Resources Association (ELRA)}},
  address   = {Marseille, France},
  day       = {20--25},
  month     = jun,
  year      = {2022},
  isbn      = {979-10-95546-72-6},
  language  = {english},
  url       = {http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.163}
}