SignBLEU: Automatic Evaluation of Multi-channel Sign Language Translation

Kim, Jung-Ho | Huerta-Enochian, Mathew | Ko, Changyong | Lee, Du Hui

Volume:: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Venue:: Torino, Italy
Date:: 20 to 25 May 2024
Pages:: 14796–14811
Publisher:: ELRA Language Resources Association (ELRA) and the International Committee on Computational Linguistics (ICCL)
License:: CC BY-NC 4.0
ACL ID:: 2024.lrec-main.1289
ISBN:: 978-2-493814-10-4

Content Categories

Projects:: Development of interactive sign language interpretation service based on artificial intelligence for the hearing impaired, Development of Korean Sign Language translation service technology for the deaf in medical environment
Languages:: American Sign Language, German Sign Language, Korean Sign Language, English, German, Korean
Corpora:: DGS Corpus, NCSLGR, NIASL2021

Abstract

Sign languages are multi-channel languages that communicate information through not just the hands (manual signals) but also facial expressions and upper body movements (non-manual signals). However, since automatic sign language translation is usually performed by generating a single sequence of glosses, researchers eschew non-manual and co-occurring manual signals in favor of a simplified list of manual glosses. This can lead to significant information loss and ambiguity. In this paper, we introduce a new task named multi-channel sign language translation (MCSLT) and present a novel metric, SignBLEU, designed to capture multiple signal channels. We validated SignBLEU on a system-level task using three sign language corpora with varied linguistic structures and transcription methodologies and examined its correlation with human judgment through two segment-level tasks. We found that SignBLEU consistently correlates better with human judgment than competing metrics. To facilitate further MCSLT research, we report benchmark scores for the three sign language corpora and release the source code for SignBLEU at https://github.com/eq4all-projects/SignBLEU.

Document Download

Paper PDF BibTeX File + Abstract

Cite as

Citation in ACL Citation Format

Jung-Ho Kim, Mathew Huerta-Enochian, Changyong Ko, Du Hui Lee. 2024. SignBLEU: Automatic Evaluation of Multi-channel Sign Language Translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14796–14811, Torino, Italy. ELRA Language Resources Association (ELRA) and the International Committee on Computational Linguistics (ICCL).

BibTeX Export

@inproceedings{kim-etal-2024-signbleu:lrec,
  author    = {Kim, Jung-Ho and Huerta-Enochian, Mathew and Ko, Changyong and Lee, Du Hui},
  title     = {SignBLEU: Automatic Evaluation of Multi-channel Sign Language Translation},
  pages     = {14796--14811},
  editor    = {Calzolari, Nicoletta and Kan, Min-Yen and Hoste, Veronique and Lenci, Alessandro and Sakti, Sakriani and Xue, Nianwen},
  booktitle = {2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation ({LREC-COLING} 2024)},
  publisher = {{ELRA Language Resources Association (ELRA) and the International Committee on Computational Linguistics (ICCL)}},
  address   = {Torino, Italy},
  day       = {20--25},
  month     = may,
  year      = {2024},
  isbn      = {978-2-493814-10-4},
  language  = {english},
  url       = {https://aclanthology.org/2024.lrec-main.1289}
}