syncWRITER

Integrating Video into the Transcription and Analysis of Sign Language

Abstract

1. Introduction

• high expense in time
• comparability
• verifiability
• access to data for analysis purposes

• The source material is video instead of audio, video quality often is poor in order to avoid irritating lighting conditions.
• In any case, a secondary language is required to transcribe sign language data because of the lack of a written form of sign language. The transcriber must be highly skilled in both the sign language and the secondary language.
• There are no well-established notation conventions comparable to either written oral languages or to phonetic alphabets for oral languages (e.g. IPA).
  • Notation systems require an extremely high effort in time: A survey on Sign Language Linguistics List in June 1994 resulted in quoted ratios anywhere from 1:10 to 1:120 (i.e. two hours transcription time needed for one minute of video), largely depending on the notation used. The lack of standards in notation systems and the variety of systems in use turn out to be a big obstacle for the validation as well as evaluation of transcribed data.{1}
  • Glosses, on the other hand, take less time to write, but they are considered to be coarse and to have a highly interpretative character. Furthermore, glosses mean a shift in language already to describe the form: The more detailed you want the glosses to be, the more linguistic instinct for the written oral language is required.{2}

Direct integration of video

Binding the video and the transcription together not only makes the final results much more transparent, but also facilitates the segmentation and transcription process. From the technical point of view, it accelerates the process and cuts down costs by avoiding all the hassle in working with a VCR (playing back and forth, back and forth, etc.). Moreover, video integration allows "partial work": If needed, you can concentrate on just a few aspects in your transcription - only those needed for your analysis. The direct link between the transcription document and the original video material provides a skeleton which allows you to always find the necessary context.

Ease of correction

As you all know, in conventional techniques (which include text processing on a computer) it may take hours just to apply a small change in an almost-ready transcription. Unfortunately, transcription work to a large degree consists of refining and revising. The computer, not the user, should take care of how to rearrange the transcription layout after applying changes.

Full access to data

The user should not be restricted to a necessarily limited set of analysis functions provided by the programmer, but s/he should rather be able to search for exmplex configurations, to automatically manipulate data, and to interact with other programs.

2. Editing with syncWRITER

Fig 1 and 2: Two tracks with three words each, before and after synchronisation.

Fig. 3 to 5: In conventional techniques, you could achieve the same result by adding spaces (figure 3), but that makes things difficult to change. If you change text (here the second word in the second track is refined), alignment gets lost (figure 4). With synchronisation, on the other hand, alignment is automatically preserved (figure 5).

The number of tracks in a document is determined by the user and can be changed at any time. You are free to use text formatting options such as font, style, size, and color either to make things clearer or as part of your coding conventions.


Fig. 6: Multi-line transcription.

In the example shown{4} we use a three-line gloss transcription{5} as well as one line for notating the form of the signs, using HamNoSys - the Hamburg Notation System for Signs that has been in development at our institute over the past years{6}. You are completely free in what to use the tracks for. One might want to add tracks for translation into a given spoken language, comments, grammatical analysis, etc.

So far you have seen how syncWRITER enables you to set up multi-line transcriptions and how it helps in editing these structures. What now really makes syncWRITER a valuable tool in the sign language domain is the integration of digitized video into the document.

As you can add text (or picture) tracks, you can choose to add movie tracks to your document. Directly in the program you can open the movie in its own window and play it back, in full motion or frame-by-frame, forward and backward, single-step and pause, repeat it as often as needed - without doing any harm to your valuable video equipment.


Fig. 7: A thumbnail and its movie window. (False colors due to 8-bit reduction on WWW.)

For difficult-to-analyse passages, syncWRITER offers a loop function for just the segments in question to be viewed over and over again - possibly at reduced speed.{7}

The next step then is to segment the video, usually into sentences or signs. This is done directly from the movie window while you are viewing the video. In the movie track, each segment is represented by a thumbnail which is automatically created in the segmentation process. Clicking on a thumbnail opens the play-back window showing only that specific segment.

Now, with the video segmented, you synchronize the thumbnails with transcription text as you enter it. Even with the video clips attached, synchronicity is maintained over all editing operations.


Fig. 8: A transcription sample with text and video synchronized.

3. Output from syncWRITER

4. Data Analysis with syncWRITER

One example is a search agent{8} that shows the next two-handed sign in the document. The interesting point about this is that syncWRITER does not know anything about neither signs in general nor two-handed signs in particular. Instead, the search agent looks through the HamNoSys tracks (considering HamNoSys to be a standard font). If you are familiar with HamNoSys you probably know that in order to find two-handed signs you cannot simply look for a certain character, but you have to look for a couple of different patterns.

In the same way you can construct search agents as complex as "What is the average length of wh-questions in this sign language text?", "Find the next one-handed sign that took more than 1.5 seconds to produce!" or "Find signs with circle movements and the mouth pattern 'open mouth, teeth revealed', but only if they occur in questions!"

5. Commanding other Programs and Devices from within syncWRITER

6. Status and Perspectives

Students learn to carry out transcriptions on their own or as members of a team, following notational conventions agreed upon beforehand.

They all experience the typical pitfalls both in technical aspects and in transcription methodology. Using syncWRITER leaves time to get (relatively short) transcriptions ready and to work with them, better motivating students because they experience what can be done with the results of their expenditures both in time and work. These introductory courses are the basis for study papers in phase II seminars, where students produce transcriptions for their analysis approaches. Of course, technical support is provided for video digitization as well as for the construction of complex search agents.

For international cooperation in research, we hope to provide a common basis to work with and to build upon which is flexible enough to meet any special requirements or preferences.

Footnotes

References