Subunit Boundary Detection for Sign Language Recognition Using Spatio-temporal Modelling

Abstract

The use of subunits offers a feasible way to recognize sign language with large vocabulary. The initial step is to partition signs into elementary units. In this paper, we firstly define a subunit as one continuous hand action in time and space, which comprises a series of interrelated consecutive frames. Then, we propose a solution to detect the subunit boundary according to spatio-temporal features using a three-stage hierarchy: in the first stage, we apply hand segmentation and tracking algorithm to capture motion speeds and trajectories; in the second stage, the obtained speed and trajectory information are combined to locate subunit boundaries; finally, temporal clustering by dynamic time warping (DTW) is adopted to merge similar segments and refine the results. The presented work does not need prior knowledge of the types of signs and is robust to signer behaviour variation. Moreover, it can provide a base for high-level sign language understanding. Experiments on many real-world signing videos show the effectiveness of the proposed work.