Automatic recognition of Sign Language structures in RGB videos: the detection of pointing and lexical signs - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2019

Automatic recognition of Sign Language structures in RGB videos: the detection of pointing and lexical signs

Résumé

This work presents a generic approach to tackle continuous Sign Language Recognition (SLR) in ordinary RGB Sign Language videos. While usual SLR systems only analyze SL through the lexical level, it is shown here that both lexical and SL-specific features can be accurately detected and localized. This paves the way to better understanding Sign Language linguistics, as well as benefiting the Deaf community in need of SL-specific tools like video querying. Generic human motion features are first extracted from videos, so that a very compact modeling is obtained and future training time is limited. A Recurrent Neural Network is then trained on those generic features to recognize lexicon and SL-specific structures like pointing. Applied to the French Sign Language corpus Dicta-Sign, pointing signs detection gets a 78% sequence-wise F1-score on 4 seconds chunks. The network also gets a 70% sequence-wise F1-score for the detection of lexical signs with less than 50 instances in the training set. These are very promising results for a first SLR trial on a French Sign Language corpus, given its relatively short total duration, its low image resolution and frame rate, and the unconstrained nature of the recorded dialogs.
Fichier principal
Vignette du fichier
preprint.pdf (3.35 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02146368 , version 1 (12-06-2019)

Identifiants

  • HAL Id : hal-02146368 , version 1

Citer

Valentin Belissen, Michèle Gouiffès, Annelies Braffort. Automatic recognition of Sign Language structures in RGB videos: the detection of pointing and lexical signs. 2019. ⟨hal-02146368⟩
161 Consultations
110 Téléchargements

Partager

Gmail Facebook X LinkedIn More