Data-Driven Sub-Units, Modeling Structure of Multiple Cues for Continuous Sign Language Recognition

Pitsikalis, Vassilis | Theodorakis, Stavros | Maragos, Petros

Volume:: Proceedings of the LREC2010 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies
Venue:: Valletta, Malta
Date:: 22 and 23 May 2010
Pages:: 196–203
Publisher:: European Language Resources Association (ELRA)
License:: CC BY-NC
sign-lang ID:: 10049

Content Categories

Languages:: American Sign Language
Corpora:: RWTH-BOSTON-400

Abstract

We investigate the automatic phonetic modeling of sign language based on phonetic sub-units, which are data driven and without any prior phonetic information. Visual processing is based on a probabilistic skin color model and a framewise geodesic active contour segmentation; occlusions are handled by a forward-backward prediction component leading finally to simple and effective region-based visual features. For sign-language modeling we propose a modeling structure for data-driven sub-unit construction. This utilizes the cue that is considered crucial to segment the signal into parts; at the same time we also classify the segments by implicitly assigning labels of Dynamic or Static type. This segmentation and classification step disentangles Dynamic from Static parts and allows us to employ for each type of segment the appropriate cue, modeling and clustering approach. The constructed Dynamic segments are exploited at the model level via hidden Markov models (HMMs). The Static segments are exploited via k-means clustering. Each Dynamic or Static part, exploits the appropriate cue related to the movement. We propose that the movement cues are normalized in order to be translation and scale invariant. We apply the proposed modeling for further combination of the movement trajectory individual cues. The proposed approaches are evaluated in recognition experiments conducted on the continuous sign language corpus of Boston University (BU-400) showing promising preliminary results.

Document Download

Paper PDF BibTeX File + Abstract

Cite as

Citation in ACL Citation Format

Vassilis Pitsikalis, Stavros Theodorakis, Petros Maragos. 2010. Data-Driven Sub-Units, Modeling Structure of Multiple Cues for Continuous Sign Language Recognition. In Proceedings of the LREC2010 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, pages 196–203, Valletta, Malta. European Language Resources Association (ELRA).

BibTeX Export

@inproceedings{pitsikalis:10049:sign-lang:lrec,
  author    = {Pitsikalis, Vassilis and Theodorakis, Stavros and Maragos, Petros},
  title     = {Data-Driven Sub-Units, Modeling Structure of Multiple Cues for Continuous Sign Language Recognition},
  pages     = {196--203},
  editor    = {Dreuw, Philippe and Efthimiou, Eleni and Hanke, Thomas and Johnston, Trevor and Mart{\'i}nez Ruiz, Gregorio and Schembri, Adam},
  booktitle = {Proceedings of the {LREC2010} 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies},
  maintitle = {7th International Conference on Language Resources and Evaluation ({LREC} 2010)},
  publisher = {{European Language Resources Association (ELRA)}},
  address   = {Valletta, Malta},
  day       = {22--23},
  month     = may,
  year      = {2010},
  language  = {english},
  url       = {https://www.sign-lang.uni-hamburg.de/lrec/pub/10049.pdf}
}