Summary:
Lee and Tsai implemented a vision-based hand gesture recognition system to classify 20 hand TSL signs. The system used hand features based on visual distances, and 8 reflective markers were placed on the hand to assist in these readings. The features are then sent into a back-propogation neural network (BPNN) that had 15 features as inputs and the 20 gesture probabilities as outputs.
The features used include the distances between a wrist point and the finger tips, and the distances between each finger pair (spread).
10 students tested the system and produced 2788 gestures, of which half went to training and the other half to testing. The authors tested on neural networks with 2 hidden layers varying in size from 25 x 25 to 250 x 250. The best results were with the BPNN with 250 x 250 hidden nodes, with a testing accuracy of 94.65%. Two gestures were heavily confused because the only difference was the length of the finger shown (i.e., the fingers were bent in one gesture).
Discussion:
This was a pretty decent use of neural nets, and I'm glad that they gave the results at different hidden layers and the recognition rates for each gesture. In fact, now that I think about it, I'm just glad they gave results. These are definitely the best results I've seen and quite promising: one of their main issues was a good feature to distinguish between bent fingers and non-bent fingers.
The differences between 150x150 and 250x250 are statistically insignificant, but they might be more significant when more gestures are added. I especially like that there is little discrepancy between training and testing sets, which hopefully indicates that their approach works for the general user.
Showing posts with label sign language. Show all posts
Showing posts with label sign language. Show all posts
Sunday, March 23, 2008
Wednesday, March 5, 2008
Temporal Classification: Extending the Classification Paradigm to Multivariate Time Series
Summary:
Kadous created TClass, which uses metafeatures of gesture data to form a syntactic representation of a gesture.
The system was tested on two data sets: Auslan and Nintendo data. Auslan is an Australian sign language that has signs not like ASL, although the overall ideas of hand shape, location, and movement are present in the language. The Nintendo data comes from a Powerglove test set comprised of 95 basic hand movements.
Tests were conducted using a Powerglove (P5) and a Flock of Birds. On Kadous's tests, the initial error rate for TClass was extremely high compared to the best error rates (for both sets of data). Using AdaBoost, the system's accuracy became more tolerable, but it was never as good as a fine, hand-picked set of features.
Discussion:
I have mixed feelings about this system. I like the addition of metafeatures that are readable with TClass, but I also don't quite know what to make of the system's poor accuracy in some cases. The accuracy results presentation was confusing, since the author gave horrible results first, then semi-poor results after when using AdaBoost, but the horrible results also included a TClass with AdaBoost (AB) field, so what the hell is going on? Also, the explanation that the Nintendo dataset is "hard" does not fly; if a "naive" algorithm beats you , you cannot say that poor results are because of the test set.
Nevertheless, I think that research in this area of trying to find both accurate and understandable results is worthwhile.
Kadous created TClass, which uses metafeatures of gesture data to form a syntactic representation of a gesture.
The system was tested on two data sets: Auslan and Nintendo data. Auslan is an Australian sign language that has signs not like ASL, although the overall ideas of hand shape, location, and movement are present in the language. The Nintendo data comes from a Powerglove test set comprised of 95 basic hand movements.
Tests were conducted using a Powerglove (P5) and a Flock of Birds. On Kadous's tests, the initial error rate for TClass was extremely high compared to the best error rates (for both sets of data). Using AdaBoost, the system's accuracy became more tolerable, but it was never as good as a fine, hand-picked set of features.
Discussion:
I have mixed feelings about this system. I like the addition of metafeatures that are readable with TClass, but I also don't quite know what to make of the system's poor accuracy in some cases. The accuracy results presentation was confusing, since the author gave horrible results first, then semi-poor results after when using AdaBoost, but the horrible results also included a TClass with AdaBoost (AB) field, so what the hell is going on? Also, the explanation that the Nintendo dataset is "hard" does not fly; if a "naive" algorithm beats you , you cannot say that poor results are because of the test set.
Nevertheless, I think that research in this area of trying to find both accurate and understandable results is worthwhile.
Wednesday, February 27, 2008
American Sign Language Recognition in Game Development for Deaf Children
Summary:
Brashear et al. use GT2k to create an American Sign Language game for deaf children. The system, called CopyCat, teaches language skills to children by having them sign various sentences to interact with a game environment.
A Wizard of Oz study was used to gather data and design their interface. A desk, mouse, and chair was used in the study, along with a pink glove. The students pushed a button and then signed a gesture, and the data was collected using the glove and an IEEE 1394 video camera. The users were 9- to 11-year-olds.
The hand is pulled from the video image by its bright color. The image pixel data is converted to a HSV color space histogram, which is used to binarize the data and find the hand. Accelerometers are also used to track hand movement in x, y, and z positions.
The data from five children was analyzed for user-dependent and -independent models. User-dependence was validated in a 90/10 (training/testing) split, with word accuracy in the low 90s and and sentence accuracy around 70%. The standard deviation for the sentence accuracy is very high, with approximately at 12% deviation.
User-independent models were lower with an average word accuracy of 86.6% and a sentence accuracy of 50.64%.
Discussion:
I like the author's user study with the Wizard of Oz to collect real-world data from children. The system's performance (in essence, GT2k's performance) was very low with sentences, which indicates that segmentation is the largest issue with the toolkit. I'm also worried about the 90/10 split for the user dependent models. That is a huge ratio of training to testing data, and it might be skewing the results to show higher than normal accuracy.
Brashear et al. use GT2k to create an American Sign Language game for deaf children. The system, called CopyCat, teaches language skills to children by having them sign various sentences to interact with a game environment.
A Wizard of Oz study was used to gather data and design their interface. A desk, mouse, and chair was used in the study, along with a pink glove. The students pushed a button and then signed a gesture, and the data was collected using the glove and an IEEE 1394 video camera. The users were 9- to 11-year-olds.
The hand is pulled from the video image by its bright color. The image pixel data is converted to a HSV color space histogram, which is used to binarize the data and find the hand. Accelerometers are also used to track hand movement in x, y, and z positions.
The data from five children was analyzed for user-dependent and -independent models. User-dependence was validated in a 90/10 (training/testing) split, with word accuracy in the low 90s and and sentence accuracy around 70%. The standard deviation for the sentence accuracy is very high, with approximately at 12% deviation.
User-independent models were lower with an average word accuracy of 86.6% and a sentence accuracy of 50.64%.
Discussion:
I like the author's user study with the Wizard of Oz to collect real-world data from children. The system's performance (in essence, GT2k's performance) was very low with sentences, which indicates that segmentation is the largest issue with the toolkit. I'm also worried about the 90/10 split for the user dependent models. That is a huge ratio of training to testing data, and it might be skewing the results to show higher than normal accuracy.
Thursday, February 21, 2008
3D Visual Detections of Correct NGT Sign Production
Summary:
Lichtenauer et al. created an interactive Dutch sign language system that would help train children to use the correct gesture. Their system has various requirements including: working under mixed lighting, being user independent, having immediate response, adaptive to skill level, and invariance to valid signs.
The authors' system uses two cameras to digitally track a person's head and hands, and a touch screen is placed in front of the user for software interactivity. The skin color of the person is first determined by finding the face, which is done by having a system's operator press a pixel inside of the face and a pixel around the outside of the head. These pixels than provide a way to train the skin color model of the system, which is a a Gaussian perpendicular in RGB space. The face and hands are separated into a Left and Right RGB distribution; the authors feel that a light source will typically coming from one direction, such as an open window. Hands are detected through their number of skin pixels, and the motion of a hand starts the tracking.
The system uses fifty 2D and 3D properties (features) related to hand location and movement. These properties are assumed to be independent, and base classifiers for each figure are computed and summed together to get a total classification value. These base classifiers use Dynamic Time Warping (DTW) to find the correspondence between two feature signals over time. These classifiers are trained with the "best" 50% of the training set for each feature. A sign is classified as correct if the average classifier probability for a class is above a threshold.
The results from the authors mention that they achieve "95% true positives" of the data.
Discussion:
In class, we have already discussed the issue of having a 95% positive rate, since the system is set up so that each symbol is known and the user is supposed to gesture the correct system. Always returning true will produce 100% accuracy.
I think the larger issue is that the classifier itself needs to be tested independent of the system. Theoretically, a separate classifier can be fine tuned for each gesture so that it can correctly recognize a single gesture 100% of the time. The issues involved with using a generic classifier will then be avoided.
Lichtenauer et al. created an interactive Dutch sign language system that would help train children to use the correct gesture. Their system has various requirements including: working under mixed lighting, being user independent, having immediate response, adaptive to skill level, and invariance to valid signs.
The authors' system uses two cameras to digitally track a person's head and hands, and a touch screen is placed in front of the user for software interactivity. The skin color of the person is first determined by finding the face, which is done by having a system's operator press a pixel inside of the face and a pixel around the outside of the head. These pixels than provide a way to train the skin color model of the system, which is a a Gaussian perpendicular in RGB space. The face and hands are separated into a Left and Right RGB distribution; the authors feel that a light source will typically coming from one direction, such as an open window. Hands are detected through their number of skin pixels, and the motion of a hand starts the tracking.
The system uses fifty 2D and 3D properties (features) related to hand location and movement. These properties are assumed to be independent, and base classifiers for each figure are computed and summed together to get a total classification value. These base classifiers use Dynamic Time Warping (DTW) to find the correspondence between two feature signals over time. These classifiers are trained with the "best" 50% of the training set for each feature. A sign is classified as correct if the average classifier probability for a class is above a threshold.
The results from the authors mention that they achieve "95% true positives" of the data.
Discussion:
In class, we have already discussed the issue of having a 95% positive rate, since the system is set up so that each symbol is known and the user is supposed to gesture the correct system. Always returning true will produce 100% accuracy.
I think the larger issue is that the classifier itself needs to be tested independent of the system. Theoretically, a separate classifier can be fine tuned for each gesture so that it can correctly recognize a single gesture 100% of the time. The issues involved with using a generic classifier will then be avoided.
Subscribe to:
Posts (Atom)