Linguistically-driven Framework for Computationally Efficient and Scalable Sign Recognition

Metaxas, Dimitris | Dilsizian, Mark | Neidle, Carol

Volume:: Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)
Venue:: Miyazaki, Japan
Date:: 7 to 12 May 2018
Pages:: 1711–1718
Publisher:: European Language Resources Association (ELRA)
Licence:: CC BY-NC 4.0
ACL ID:: L18-1271
ISBN:: 979-10-95546-00-9

Content Categories

Languages:: American Sign Language
Lexical Databases:: ASLLVD

Abstract

We introduce a new general framework for sign recognition from monocular video using limited quantities of annotated data. The novelty of the hybrid framework we describe here is that we exploit state-of-the art learning methods while also incorporating features based on what we know about the linguistic composition of lexical signs. In particular, we analyze hand shape, orientation, location, and motion trajectories, and then use CRFs to combine this linguistically significant information for purposes of sign recognition. Our robust modeling and recognition of these sub-components of sign production allow an efficient parameterization of the sign recognition problem as compared with purely data-driven methods. This parameterization enables a scalable and extendable time-series learning approach that advances the state of the art in sign recognition, as shown by the results reported here for recognition of isolated, citation-form, lexical signs from American Sign Language (ASL).

Document Download

Paper PDF BibTeX File + Abstract

Cite as

Citation in ACL Citation Format

Dimitris Metaxas, Mark Dilsizian, Carol Neidle. 2018. Linguistically-driven Framework for Computationally Efficient and Scalable Sign Recognition. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018), pages 1711–1718, Miyazaki, Japan. European Language Resources Association (ELRA).

BibTeX Export

@inproceedings{metaxas-etal-2018-linguistically:lrec,
  author    = {Metaxas, Dimitris and Dilsizian, Mark and Neidle, Carol},
  title     = {Linguistically-driven Framework for Computationally Efficient and Scalable Sign Recognition},
  pages     = {1711--1718},
  editor    = {Calzolari, Nicoletta and Choukri, Khalid and Cieri, Christopher and Declerck, Thierry and Goggi, Sara and Hasida, Koiti and Isahara, Hitoshi and Maegaard, Bente and Mariani, Joseph and Mazo,  H{\'e}l{\`e}ne and Moreno, Asuncion and Odijk, Jan and Piperidis, Stelios and Tokunaga, Takenobu},
  booktitle = {11th International Conference on Language Resources and Evaluation ({LREC} 2018)},
  publisher = {{European Language Resources Association (ELRA)}},
  address   = {Miyazaki, Japan},
  day       = {7--12},
  month     = may,
  year      = {2018},
  isbn      = {979-10-95546-00-9},
  language  = {english},
  url       = {https://aclanthology.org/L18-1271}
}