LRNeuNet: an Attention Based Deep Architecture for Lipreading from Multitudinous Sized Videos

Speech Perception is considered as a sound-related aptitude, which is characteristically multimodal on the grounds that delivers speech requires the movements of the lips, teeth, and tongue of the speakers which are normally noticeable in vis-vis correspondence. Though sign language has been developed and deaf people are usually quite adept at lipreading, it can become a very difficult task for varying speakers with different accents and speaking speeds. A lot of models have been proposed to capture the speaker's lip movements and predicting the sentences spoken by them by observing the movements and contours. This too is a difficult task as a lot of consonants have the same lip movement which leads to the necessity of having sequential models, which take into account the words that have been spoken previously and the words that are spoken afterwards. Speakers' mouths may additionally look of varying kinds and so might their appearances in images. We propose a model for this purpose. Our LRNeuNet operates using SpatioTemporal Convolutional Neural Network (STCNN), ResNet, Bi-directional Gated Recurrent Unit (Bi-GRU), SpatioTemporal Pyramidal Pooling(STPP) and Hybrid Connectionist Temporal Classification (CTC)/ Attention. The STCNN incorporates Maxout layer. We use STPP to give a fixed-length representation for training by pooling the features in arbitrary regions. In the hidden neural layer, Hybrid CTC/ Attention model partially removes the error of the assumption of conditional independence of CTC. Our proposed model LRNeuNet achieved 1.2% CER and 2.7% WER on the GRID Corpus dataset.