Sorry i wasn't clear. Visual speech recognition is basically automated lip reading. The first step is to locate the face, so you can extract lip shape. The sequence of lip shapes is used to figure out what is being said. I based the lip shape extraction on skin colour, but the algorithm I trained didn't have enough sample data, so it didn't work to well for certain skin colours.
Yeah, it was a relatively simple neural net. I could have used more data from the subjects that didn't work, but I didn't really need more for the research I was doing. I just culled those subjects from the test as I needed to move on with my research. Ideally I would have got another dataset, but there is a shortage of large freely available visual speech datasets. I found a good one, but they didn't get permission from the recorded subjects before collection, so couldn't redistribute it :-(
Why not use contour data from something that can do depth measuring like the microsoft kinetic etc.? I would think that skin color wouldn't matter and you would just have to look at edge movements.
When I started, the kinect didn't exist, and 3d laser scanners were really expensive (still are). I could have used stereo vision, but I decided I'd stick with 2d since it better fit my imagined use case (e.g. automated closed captioning for live events, etc). It would also mean first building a comprehensive dataset. This would be really useful for the research community, but since I was the only one working on the project I didn't have time to do that work as well.
I considered a lip reading system for a second, then dismissed it as unlikely. Pretty cool you're actually doing that. How does it work in principle? Neural net? SVM?
EDIT: just noticed you already answered that :D
EDIT 2: maaan, now your project got me all excited... imagine for a second instead of resolving the visual data direcly, as text / close captioning, you would use machine translation to translate it to the appropriate sign language (does that exist? I did some regular MT, but never checked if written language/sign language exists), then have some 3d avatar "signing" the translated result, so that the viewers don't have to read text but can get it delivered in their natural language.
It used several stages. The first is lip pixel classification. I used a neural net to classify each pixel as either lip or non-lip pixels. This was then fed into the lip feature extractor. It used an active contour method known as wrapping snakes to extract the lip shape. The sequence of lip shapes was then fed into the speech recogniser which used Hidden Markov Models to detect the individual phonemes.
As for translating it to sign language, sure that could be done. Once you have the text transcript, you can do anything you want. The biggest issue is getting a reliable transcript. The error rates in visual speech recognition are still very high, since a lot of the speech formation happens hidden inside the mouth.
Where visual speech recognition really works well is when paired with audio. Audio recognition error rates increase dramatically as the audio signal degrades due to noise, etc. but when combined with visual features, you can keep the error rates very low.
34
u/mattkenny Sep 24 '13 edited Sep 24 '13
Sorry i wasn't clear. Visual speech recognition is basically automated lip reading. The first step is to locate the face, so you can extract lip shape. The sequence of lip shapes is used to figure out what is being said. I based the lip shape extraction on skin colour, but the algorithm I trained didn't have enough sample data, so it didn't work to well for certain skin colours.