Showing posts with label glove. Show all posts
Showing posts with label glove. Show all posts

Monday, April 14, 2008

Feature selection for grasp recognition from optical markers

Summary:

Chang et al. reduced the number of markers needed on a vision-based hand grasp system from 30 to 5 while retaining around a 90% recognition rate.

Six different grasps are used for classification: cylindrical, spherical, lumbrical, two-finger pinch, tripod, and lateral tripod. The posterior probabilities for a class yk are modeled with a softmax function, which divides the exp value of an observation sequence with the class weights, divided by the sum of all exp(weights * obs) values.

The weight values are determined by maximum conditional likelihood estimation from the training set of observations and classes (X, Y). Gradient descent is used to find the log likelihood with respect to the weights. Input features are found using a "sequential wrapper algorithm" that examines one feature at a time with respect to a target class.

Grasp data measured 38 objects being grasped with a full set of 30 markers. An "optimal", small set of markers was chosen by forward and backward selection.

The results indicate that the small marker set of 5 markers has between a 92-97% "accuracy retention" rate.


Discussion:

Reducing the number of sensors using the forward and backward selection is nice, but simply having a few more sensors increases the accuracy to the actual plateau point. From 10 on there is almost no change in accuracy, but between 5 and 10 sensors the accuracy can jump 5%, or 1/20, which is a huge percentage when taking into account user frustration.

Glove-TalkII--A Neural-Network Interface which Maps Gestures to Parallel Formant Speech Synthesizer Controls

Summary:

Fels and Hinton created Glove-TalkII, a system designed to synthesize voice using complicated glove and feet controls.

The artificial vocal track (AVT) is controlled using a CyberGlove, ContactGlove, polhemus sensor, and foot pedal. The ContactGlove controls 9 stop consonants, such as CH, T, and NG. The foot pedal controls the volume of the speech. Hand position corresponds to a vowel sound. Hand postures map to nonstop consonant phonemes.

The neural networks used include a vowel/consonant network to determine if the sensors are reading a vowel or consonant, and then separate vowel and consonant networks to distinguish between the phonemes.

A single user had to undergo 100 hours of training to be able to use the system.


Discussion:

Impractical. I'm shocked that they had someone train the system for 100 hours, and the fact that it takes a person that long to train the system should indicate that this is a poor way to synthesize voice. The person's final voice is even described as "intelligible and somewhat natural-sounding", which is not a good complement.

Requiring a person to walk around with a one-handed keyboard and type their words is a better solution. The keyboard wouldn't even have a foot pedal.

Thursday, March 6, 2008

A Hidden Markov Model Based Sensor Fusion Approach for Recognizing Continuous Human Grasping Sequences

Summary:

Bernardin et al. created a system to recognize human grasping gestures using a CyberGlove and pressure sensor data. Basic grasps are distinguished in 14 different ways by Kamakura's grasping primitives. These grasps include "5 power grasps, 4 intermediate grasps, 4 precision grasps, and one thumbless grasp".

To recognize these grasps, an 18 sensor CyberGlove is used, along with finger tip and palm sensors. 14 different pressure sensors are sewn into a glove, which is worn under the CyberGlove. The sensor data is passed into HMMs for recognition. A 9-state HMM is built for each gesture using the HTK. After each grasp, the grasped object must be released.

On a total of 112 training gestures from 4 users, the user dependent models were between 77 and 92%, whereas a user independent model was in the low 90s for all 4 users. This is most likely due to the increase in training data when all the user data is combined.


Discussion:

I thought the use of HMMs in this paper was actually quite good. The problem I have with HMMs is that they are absolutely horrible and explode when data is not properly segmented. In the case of grasps, it is probably less likely that somebody is going to go from 1 grasp to another without releasing the object they are holding. For most, general cases, the computer can assume that the lack of tactile input from the palm would indicate a grasp has ended.

Wednesday, February 27, 2008

American Sign Language Recognition in Game Development for Deaf Children

Summary:

Brashear et al. use GT2k to create an American Sign Language game for deaf children. The system, called CopyCat, teaches language skills to children by having them sign various sentences to interact with a game environment.

A Wizard of Oz study was used to gather data and design their interface. A desk, mouse, and chair was used in the study, along with a pink glove. The students pushed a button and then signed a gesture, and the data was collected using the glove and an IEEE 1394 video camera. The users were 9- to 11-year-olds.

The hand is pulled from the video image by its bright color. The image pixel data is converted to a HSV color space histogram, which is used to binarize the data and find the hand. Accelerometers are also used to track hand movement in x, y, and z positions.

The data from five children was analyzed for user-dependent and -independent models. User-dependence was validated in a 90/10 (training/testing) split, with word accuracy in the low 90s and and sentence accuracy around 70%. The standard deviation for the sentence accuracy is very high, with approximately at 12% deviation.

User-independent models were lower with an average word accuracy of 86.6% and a sentence accuracy of 50.64%.


Discussion:

I like the author's user study with the Wizard of Oz to collect real-world data from children. The system's performance (in essence, GT2k's performance) was very low with sentences, which indicates that segmentation is the largest issue with the toolkit. I'm also worried about the 90/10 split for the user dependent models. That is a huge ratio of training to testing data, and it might be skewing the results to show higher than normal accuracy.

Monday, February 18, 2008

Real-time Locomotion Control by Sensing Gloves

Summary:

Komura and Lam propose using P5 gloves to control character motion. The authors feel that using a "walking fingers" can provide a more tangible interface for controlling motion than traditional joystick or keyboard techniques.

The authors use a P5 glove for their gesture capture, and the user first calibrates the fingers by moving them in time with a given walking animation displayed on a computer screen. This calibration happens by a simple function comparing the cycle of the user's fingers versus the cycle of the animation.

After calibration, the user's fingers should be in-sync with the walking motions. For animating quadrupeds, there might need to be a phase shift between the back and front legs.

To test their system, the authors used a CyberGlove and had users play mock games with characters jumping and navigating a maze. Their results showed that navigating with the glove is potentially easier in terms of the number of collisions in a maze, and the glove and keyboard controls allow maze navigation in approximately the same time.


Discussion:

There's not much to say about this paper. The results that they gave were odd, since User 2 completed the maze with a keyboard in 18 seconds but had 22 collisions, and with the glove in 31 seconds with 3 collisions. I'm not sure what to make of that data...

Other than that, the research aspect of this paper basically took a finger sine and mapped it to an animation's sine. It might make navigating in certain games easier, but only if you need to control the speed of the character with better precision.

Wednesday, February 13, 2008

A Dynamic Gesture Recognition System for the Korean Sign Language (KSL)

Summary:

Kim, Jang, and Bien use fuzzy min-max neural networks to recognize a small set of 25 basic Korean Sign Language gestures. The authors use two data gloves, each with 10 flex sensors, 3 location (x, y, z) sensors, and 3 orientation (pitch, yaw, roll).

Kim et al. find that the 25 gestures they use contain 10 different direction types, shown below

The authors also discovered that the data often has derivations within 4 inches of other data, so the x and y coordinates are split into 8 separate regions from -16 to 16 inches, with 4 inch ticks. The change in x, y direction (CD) is recorded for each time step simply as + and - symbols, and this data is recorded for four steps. CD change templates are then made for the 10 directions, D1 ... D10.

The 25 gestures contain 14 different hand gestures based on finger flex position. This flex value is sent to a fuzzy min-max neural network (FMMN) that separates the flex angles within a 10-dimensional "hyper box".

To classify a full gesture, the change of direction is first taken and compared against the templates, and then the flex angles are run through the FMNN. If the total (accuracy/probability) value is above a threshold, the gesture is classified.

The authors achieve approximately 85% accuracy.


Discussion:

Although this paper had some odd sections and interesting choices, such as making the time step 1/15th of a second and having gestures over 4/15ths of a second, the overall idea is quaint. I appreciate that the algorithm separates the data into two categories--direction change and flex angle--and separates the two components to hierarchically choose gestures.

I still do not like the use of neural networks, but if they work I am willing to forgive. My annoyance is also alleviated by the fact that the authors provide thresholds and numerical values for some equations within the network.

I'm very curious why they chose those 10 directions (from the figure). D1 and D8 could be confused if the user is sloppy, and D4 and D7 can be confused with their unidirection counterparts if the user is does their gestures slower than 1/4 of a second. Which is, of course, absurd.

Wednesday, February 6, 2008

Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation

Ip, H. H. S., K. C. K. Law, et al. (2005). Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation. Proceedings of the 11th International Multimedia Modelling Conference, 2005.

Summary:

Ip et al. created Cyber Composer, which is a music generation program controlled via hand gestures. The author's motivation is to inspire both musicians and casual listeners to experience music in a new way.

The authors split music composition into three parts: melody, rhythm, and tone. The melody is the "main" part of the music and mainly includes the treble parts, such as the singer. The rhythm keeps the beat of the music and is played by the drums and bass. Tonal accompaniment involves creating harmony across all parts.

In order to keep the tone (harmony) of the music interesting and flowing, the authors create a small "chord affinity" matrix that describes certain chord lead/following strengths. During music composition, chords are automatically chosen with high affinity. Melody notes are also chosen automatically to create musical "tension".

The system was implementing using two 22 sensor CyberGloves and two Polhemus positioning receivers. MIDI was used to produce the musical notes.

The seven gestures used in the system include rhythm, pitch, pitch-shifting, dynamics, volume, dual-instrument mode, and cadence. Rhythm is controlled by the flexing of the right wrist. Pitch is controlled by right-hand height, and it is reset at the beginning of each bar. The user can also "shift" the pitch by performing a similar gesture. Note dynamics and volume are controlled by the right-hand finger flex, with fully flexed fingers forcing forte notes. Dual-instrument mode allows a harmony melody or unison melody to be played along with the main instrument; this mode is activated using the left hand. To end the piece, the left-hand fingers are closed.

There are no results.


Discussion:


This paper aroused me. Some of the gestures they defined were intuitive, such as opening and closing of the fingers for volume and moving the hand up and down for notes. Other gestures just seem awkward, such as the ambiguous dual-instrument mode and constantly flapping your wrist (ouch?) to drive the melody.

I'm familiar with building music composition programs (including "smart" programs that use musical theory to assist composition), and I think this program was trying to market itself as something that it could never become. A music tool has to be either robust to allow experts to use it, sacrifice some features to become simple for novices, or fun for just the casual listener. In the expert category I would place Finale, and on the casual end I would place music games such as Guitar Hero. Novice programs are harder to come by, and the tool I worked on was ImproVisor--a system that used intelligent databases to analyze input notes and determine if the notes "sounded good".

CyberGlove is trying to do everything at once and failing. The lack of any results, even the casual comment by an offhand user, tells me that the system is rather convoluted to use or poor for composition. The hand gestures cannot really control notes in a way that experts would use the system, novices will not understand the theory behind why their hand waving sounds good or bad, and casual musicians will probably have no idea what is going on.


Sunday, February 3, 2008

A Multi-Class Pattern Recognition System for Practical Finger Spelling Translation

Hernandez-Rebollar, J. L., R. W. Lindeman, et al. (2002). A multi-class pattern recognition system for practical finger spelling translation. Multimodal Interfaces, 2002. Proceedings. Fourth IEEE International Conference on.

Summary:

Hernandez-Rebollar et al. have a two part paper: they present a glove (AcceleGlove) and they have a test platform for the glove that uses decision trees.

The AcceleGlove contains 5 accelerometer sensors placed at the middle joint of fingers. Each accelerometer has x and y angles that can be measured, ending in a total of 10 sensor readings every 10 milliseconds. The raw data matrix consisting of just the x and y values is transformed into a separate Xg, Yg, and yi values. Xg (x global) measures the finger orientation, roll, and spread. Yg measures the finger bentness of the hand. The third component classifies the hand into three values: closed, horizontal, and vertical. This third component is actually only the index finger's y-component (only in the ASL letters 'F' and 'D' is the index finger not accurate for this measurement).

To classify a posture/gesture, the decision tree first breaks up the letters into vertical, horizontal, and closed. Then the gestures are classified further as rolled, flat, pinky up, and these sections then recognize between the actual letters.

They mention a 100% recognition rate for 21 gestures, with 78% being the worst gesture accuracy.


Discussion:

I like this paper for 2 main reasons:
  1. There are no HMMs
  2. They did not use a CyberGlove
The paper's results and decision tree theory are a bit lacking, but I think that the ideas behind the paper were good and refreshingly different from the ochoish other papers we've read.

I'm curious as to how well the glove they designed can work with gestures instead of postures. The glove polls each accelerometer sequentially, which could be a problem with very quick gestures. This issue is probably not too important, but it might provide slightly more error than a batch poll.

I'm also curious as to how they designed their decision tree. The intuition behind the partitioning is not made clear, except for the main partition of open/close/horizontal.

Wednesday, January 23, 2008

American Sign Language Finger Spelling Recognition System

Allen, J., Pierre, K., and Foulds, R. American Sign Language Finger Spelling Recognition System. (2003) IEEE.

Summary:

Allen et al.'s created an ASL recognition system using neural networks and an 18-sensor CyberGlove. The authors propose that a wearable glove recognition system can help translate ASL into English and assist deaf (and even blind) people by allowing them to converse with the hearing unimpaired.

The authors used a character set of 24 letters, omitting 'J' and 'Z' due to their usage of arm motions. Instead, the remaining 24 characters use only hand positions. Data from the CyberGlove was collected and recognized in Matlab program, and a second program called Labview would output the corresponding audio for a recognized character.

The recognition system for ASLFSR is a perceptron network with an input of 18x24 (18 sensors, 24 characters) and a desired output of 24x24 (identity matrix for the recognized symbols). The network was trained with an "adapt" function.

The system worked well for a single user and had results up to 90%.


Discussion:

The authors claim that they can achieve a better level of accuracy by training the network on data from multiple subjects, but I completely disagree. That's like saying a hand-tailored suit fits alright, but the pin-stripe at the blue light special is better since it has been designed for the average Joe.

To improve their accuracy they should improve their model. Perceptrons are not that powerful since they clobber values, and using some different neurons (Adalines?) might improve their results. Also, neural networks sometimes work better with more than just 2 layers, and data from 18 non-distinct inputs would probably benefit from even a 3-layer NN . Multiple layer NNs are notoriously tricky to design "well" (i.e. guess and check).