Brain Mechanisms of Vision

Hubel DH, Wiesel TN. Brain Mechanisms of Vision. Scientific American. 1979 Sep; 241(3):150-62

Summary:

The brain's primary visual cortex processes images in a modular, distorted way. The rods and cones in the eyes send messages from the retina to the geniculate cells in the brain, which then relay the message to the visual cortex. These geniculate cells are in a layer called Layer IV, are unsophisticated, and receive the bulk of the visual input.

Cells outside of layer IV have "orientation specificity", where a bar of light falling in a certain orientation will activate some cells and have no affect on others. The response for each cell appears to be around 10-20 degrees, at which point the response is lessened or abolished.

At the time (1979), there was no evidence to support that the orientation specific cells had anything to do with visual perception.

As electric signals moved into more complex layers of the visual cortex, some patterns emerged. Cells close to one another often have the same optimal stimulus orientation. Changes in orientation happen in small increments, such as 25-50 micrometers between cell groups mapping to a change of 10 degrees with varying direction reversals.


Discussion:

Really interesting information on how the structure and hierarchy of the primary visual cortex. Although the orientation information did not prove that the brain recognizes shapes using features such as line orientations, other papers citing this one might. I'll have to find some...

Invariant features for 3-D gesture recognition

Summary:

Campbell et al. use HMMs and a list of features to find a good recognition rate for a set of T'ai Chi gestures that are performed by users in a swivel chair; a hand gesture's change in polar coordinates provided the highest recognition for the 18 gestures tested.


Discussion:

Performing T'ai Chi in a chair kind of defeats the purpose of T'ai Chi. That's like trying to study race car drivers by observing people who take the bus.

Interacting with human physiology

Summary:

The authors Pavlidis et al. propose a system to monitor humans for stress levels and altered psychological states using high-end infrared cameras. This system could then be used for a variety of purposes such as stress management of UIs, illness detection, or lie detection.

The system tracks the user's face through tandem tracking to track a small, keys section of the face. These sections include the nose, forehead, and temporal regions. The tracker models each region by its center of mass and orientation. Blood flow is tracked in the face through a perfusion model and directional model. The model involves a differential equation set to measure the "volumetric metabolic heat" flow in the face. Other measurements tracked include pulse, heat transfer in areas, and breathing rate.


Discussion:

The ideas behind this system are great, although talking with Pavlidis showed us that there are issues with the current system's usability. Sweat and minor body temperature fluctuations can alter the system's reliability (since the system is trying to measure minor fluctuations). Unfortunately, the cost for one of these high-end cameras is $60k, so we won't be seeing this any time soon.

Discourse Topic and Gestural Form

Summary:

Eisenstein et al. applied an unsupervised learning technique and a Bayesian network model to study the correlation between gestures and presentation topics.

Their system looks at "interest points" within a video image, where each interest point is said to have come from a mixture model. Interest points from a similar model are clustered together to create a codebook. A hidden variable determines whether the observation gesture codeword is from a topic-specific or a speaker-specific distribution. The authors use a Bayesian model to learn what distribution each gesture belongs to, based off Gaussians of feature vectors.

The system was tested with fifteen users giving 33 presentations picked from five topics. The experiments show that with correct labels, the topic-specific gestures account for 12% of the gestures, whereas corrupting these labels drops the average to 3%.


Discussion:

This paper is a good start to a longer study on how to incorporate topic-specific gestures into recognition systems. Finding these gestures can help computers understand what topics might be presented, as well as what speakers are presenting a topic or if a speaker is veering off-topic. The system can then be used for speech training, presentation classification, or assistance (Clippy).

Activity Recognition using Visual Tracking and RFID

Summary:

Krahnstoever et al. use RFID tags in conjunction with computer vision tracking to interpret what is happening within a scene.

A person model tracks a human's movement through their head and hands. The head is a 3D cartesian coordinate location, and each hand is described in spherical coordinates (r, phi, theta) with respect to the head. The models for where the head will be, p(X^t_h, X^t-1_h), and the hands p(X^t_q, X^t-1_h) had to be learned. The priors p(X_q | X_h) also had to be learned. Both hands and head are segmented using skin color.

Each pixel within a given image frame can belong to either the background or the foreground (body part). The likelihood for an image given the observations is taken to be the Improved Iterative Scaling (IIS) of the image section and bounding box of a body part, summed over the parts and sections. I have no idea how IIS works.

RFID tags provide movement and orientation information in 3D spaces. The amount of charge the RFID tag receives depends on its angle to the wave source, where a perpendicular angle receives no energy and a parallel angle is the greatest. The tag then outputs the tag's ID, orientation, and field strength to the signal.

The authors use the RFID information along with the hand and head positions to interpret what is happening in a scene. Agents are somehow used to do this.


Discussion:

The RFID information looks like it helps recognize what is happening within a scene, but I would have liked to have seen a comparison between a pure vision system and a system with the RFID. This could be a bit difficult, but it might help the strength of the paper.

I also would have liked an actual description of the activity agent system.

Taiwan sign language (TSL) recognition based on 3D data and neural networks

Summary:

Lee and Tsai implemented a vision-based hand gesture recognition system to classify 20 hand TSL signs. The system used hand features based on visual distances, and 8 reflective markers were placed on the hand to assist in these readings. The features are then sent into a back-propogation neural network (BPNN) that had 15 features as inputs and the 20 gesture probabilities as outputs.

The features used include the distances between a wrist point and the finger tips, and the distances between each finger pair (spread).

10 students tested the system and produced 2788 gestures, of which half went to training and the other half to testing. The authors tested on neural networks with 2 hidden layers varying in size from 25 x 25 to 250 x 250. The best results were with the BPNN with 250 x 250 hidden nodes, with a testing accuracy of 94.65%. Two gestures were heavily confused because the only difference was the length of the finger shown (i.e., the fingers were bent in one gesture).


Discussion:

This was a pretty decent use of neural nets, and I'm glad that they gave the results at different hidden layers and the recognition rates for each gesture. In fact, now that I think about it, I'm just glad they gave results. These are definitely the best results I've seen and quite promising: one of their main issues was a good feature to distinguish between bent fingers and non-bent fingers.

The differences between 150x150 and 250x250 are statistically insignificant, but they might be more significant when more gestures are added. I especially like that there is little discrepancy between training and testing sets, which hopefully indicates that their approach works for the general user.

Articulated Hand Tracking by PCA-ICA Approach

Summary:

Kato et al. used Independent Component Analysis (ICA) to find basis vectors for hand motion features. The authors first use Principle Components Analysis to reduce the dimensionality of their system, and then they use ICA to find a set of vectors that are statistically independent from each other (i.e., basis vectors).

Data on 20 angles was collected with a glove. The authors then sandwiched all of the data for the 20 sensors together into one large vector; each sensor was sampled across 100 time points, and the data from all 20 sensors was merged into a 2000-dimension vector.

ICA is used to find the basis vectors for a hand such that a linear combination of these vectors will produce a desired hand movement. The basis vectors U are found through a weight matrix W and a sample of motion data X (where X is a matrix of hand motions). A neural learning algorithm (in this case, gradient descent) is used to calculate the weights. The resulting 5 basis vectors are the movement of each finger individually.

The authors then deviated from their abstract and discussed actually tracking a hand using particle filtering. A hand's current position can be estimated from its prior positions, so each basis vector can estimate where it believes the finger will be given its prior positions. The authors also segment a hand out of an image by doing some thresholding on an image and overlaying a hand model in the image to find the hand location.

There are no results.

Discussion:

There are no results.
The basis vectors seem obvious, but I'm glad that ICA found them.
There are no results.

American Sign Language Recognition in Game Development for Deaf Children

Summary:

Brashear et al. use GT2k to create an American Sign Language game for deaf children. The system, called CopyCat, teaches language skills to children by having them sign various sentences to interact with a game environment.

A Wizard of Oz study was used to gather data and design their interface. A desk, mouse, and chair was used in the study, along with a pink glove. The students pushed a button and then signed a gesture, and the data was collected using the glove and an IEEE 1394 video camera. The users were 9- to 11-year-olds.

The hand is pulled from the video image by its bright color. The image pixel data is converted to a HSV color space histogram, which is used to binarize the data and find the hand. Accelerometers are also used to track hand movement in x, y, and z positions.

The data from five children was analyzed for user-dependent and -independent models. User-dependence was validated in a 90/10 (training/testing) split, with word accuracy in the low 90s and and sentence accuracy around 70%. The standard deviation for the sentence accuracy is very high, with approximately at 12% deviation.

User-independent models were lower with an average word accuracy of 86.6% and a sentence accuracy of 50.64%.


Discussion:

I like the author's user study with the Wizard of Oz to collect real-world data from children. The system's performance (in essence, GT2k's performance) was very low with sentences, which indicates that segmentation is the largest issue with the toolkit. I'm also worried about the 90/10 split for the user dependent models. That is a huge ratio of training to testing data, and it might be skewing the results to show higher than normal accuracy.

Computer Vision-Based Gesture Recognition For An Augmented Realtiy Interface

Summary:

Storring et al. from Aalborg University created an augmented reality system to create a "less obtrusive and more intuitive" interface.

The gestures used in the system are the mapped to the hand signs for 0-6, i.e. no fist, index finger, index and middle, etc. This gesture set can be recognizable in a 2D plane with a camera. In order for these gestures to work, the hand needs to be segmented from the image. The authors use normalized RGB values, called chromaticities, to minimize the variance of the color intensity. The distributions for the background and skin chromaticities are found and are modeled as 2D Gaussians. The hands are assumed to be a minimum number and maximum number of pixels.

Gestures are found by counting the number of fingers. A polar transformation counts the number of spikes (fingers) on currently shown on the hand. Click gestures can be found by checking the bounding box width of the hand between the regular index finger gesture and a "thumb click" addition.


Discussion:

For a system that is supposed to be less obtrusive and more intuitive than current interfaces, virtual reality with unintuitive gestures does not seem like a good solution. Using "finger numbers" is a poor choice, and having a gigantic head-mounted display with cameras is probably less comfortable than looking at a computer screen. Furthermore, if the authors are focusing on using head equipment, why not just use gloves to increase the gesture possibilities?

3D Visual Detections of Correct NGT Sign Production

Summary:

Lichtenauer et al. created an interactive Dutch sign language system that would help train children to use the correct gesture. Their system has various requirements including: working under mixed lighting, being user independent, having immediate response, adaptive to skill level, and invariance to valid signs.

The authors' system uses two cameras to digitally track a person's head and hands, and a touch screen is placed in front of the user for software interactivity. The skin color of the person is first determined by finding the face, which is done by having a system's operator press a pixel inside of the face and a pixel around the outside of the head. These pixels than provide a way to train the skin color model of the system, which is a a Gaussian perpendicular in RGB space. The face and hands are separated into a Left and Right RGB distribution; the authors feel that a light source will typically coming from one direction, such as an open window. Hands are detected through their number of skin pixels, and the motion of a hand starts the tracking.

The system uses fifty 2D and 3D properties (features) related to hand location and movement. These properties are assumed to be independent, and base classifiers for each figure are computed and summed together to get a total classification value. These base classifiers use Dynamic Time Warping (DTW) to find the correspondence between two feature signals over time. These classifiers are trained with the "best" 50% of the training set for each feature. A sign is classified as correct if the average classifier probability for a class is above a threshold.

The results from the authors mention that they achieve "95% true positives" of the data.


Discussion:

In class, we have already discussed the issue of having a 95% positive rate, since the system is set up so that each symbol is known and the user is supposed to gesture the correct system. Always returning true will produce 100% accuracy.

I think the larger issue is that the classifier itself needs to be tested independent of the system. Theoretically, a separate classifier can be fine tuned for each gesture so that it can correctly recognize a single gesture 100% of the time. The issues involved with using a generic classifier will then be avoided.

Television Control by Hand Gesture

Summary:

Freeman and Weissman devised a way to control a TV with hand gestures using computer vision. In their system, the user's hand acts as a mouse. The user moves their open hand in front of the camera, palm facing toward the television, and the computer detects their hand and maps it to an on-screen mouse. When the user holds their hand over a control for a brief time period, the control is executed. Closing their hand or moving it out of the computer's vision deactivates the mouse.

The hand movement is detected by checking the angle difference between two vectors of pixels, where the pixels correspond to the pixels in an image frame and its offset. The dx and dy information is calculated for the image gradient, and this provides an orientation that can be handled in different lighting scenarios.

Discussion:

This paper was quaint. The actual algorithms used were rather simple, but the concept of controlling a TV via hand waving intrigued me. My main concern is that this application would train people watching a TV to not make any sudden movements so that the on-screen menu would not appear. Also, it forces people to walk through a living room slowly so that the TV does not catch their hand in any rapid movements. Some better gestures would benefit this system, such as twisting motions for channel or volume control.

Gestures without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes

Summary:

Wobbrock et al.'s $1 recognizer is a system designed to be a simple recognizer that does not rely on any mathematical background. With this recognizer, novices to sketch recognition can have an use simple gestures in their interfaces.

The $1 recognizer has four steps: (1) resampling, (2) rotation, (3) scaling, and (4) classification. Points in a gesture are resampled into N equidistant points defined by the developer. The gesture is then rotated so that the line between the center of the gesture and the starting point is at the 0 degree position, i.e. the center-start point axis is at 3 o'clock. The gesture is then scaled to fit within a square of some size and translated so that the center of the gesture is at the (0,0) origin.

Finally, the gesture is classified by first calculating the average distance between points in the gesture and the points in all templates (known gestures). This is called the path-distance by the authors. A score is determined directly from this path distance and the scaled square's size.

Overall, the recognizer has good results with around 98% accuracy for simple gestures. The classification relies heavily on the number of templates in the system.

Discussion:

This recognizer really does not introduce anything new to sketch recognition, but it does wrap up some basic vision and sketch recognition techniques into a simple (and given) algorithm. The technique used by $1 is not much different then general template/bitmap comparison, except using the points is a bit nicer when working with single strokes. On the other hand, bitmap comparison allows for multiple strokes in any order. This technique also relies highly on visual differences in gestures, and it does not allow for the same gesture at different scales or rotations. For instance, a forward slash gesture with a mouse or pen is a common gesture to indicate move forward in a web browser. Similarly, a backwards slash/dash indicates "page back". This recognizer cannot handle these cases.

Envisioning Sketch Recognition: A Local Feature Based Approach to Recognizing Informal Sketches

Summary:

Oltman's Ph.D. thesis uses computer vision techniques to recognize freely-drawn symbols and sketches. Freely-drawn sketches do not constrain the user in drawing style, so many issues need to be taken into accounted. For instance, stroke overtracing is a large problem in free-form sketches, especially when doodles or notes are involved. Also, noise is more prevalent, and temporal data cannot be utilized because strokes can be drawn in any order.

To combat these issues, Oltman uses computer-vision based techniques that lessen the issues from overtracing and noise while ignoring any temporal features of the sketch. The technique used is dubbed "bullseye" by Oltman, and consists of a radial partition of space around a point. The radial partitions corresponds to a histogram that keeps track of the number of stroke points within that section/bucket/slice. Stroke points are preprocessed to be relatively equidistant from one another. Each symbol has a corresponding set of bullseye patterns stored in the "codebook" for the system. Matching a series of found histograms with the trained codebook patterns allows for symbol classification. The system is trained using a SVM on known data.

To find a symbol within an entire sketch, though, is rather difficult. Oltman tries to find "candidate regions" for a symbol by looking at small, overlapping windows of points and the combining these regions into a large regions. The large regions are found using EM clustering techniques to group the smaller regions together. Any cluster that is considered too large is split into smaller clusters. The bounding box of the cluster is then taken to be the symbol's region, and the bullseye histogram is taken for the ink within the box.

Overall, the system works well for individual symbols (94.4%), especially when compared to existing systems for noisy data. The system faired slightly worse when taking the entire sketch into account, achieving an accuracy of 92.3%.

Discussion:

The use of vision techniques to classify freely-drawn symbols is a good idea because stroke data has so much noise. Using vision mapping, whether it is histogram or simple pixel overlay, tends to avoid corner segmentation and overtracing issues.

I find the results for Oltman's full sketch tests slightly skewed. The vast majority of the shapes tested in the full sketch were wire segments and resistors. Since the wires are broken down into smaller segments, there were roughly 9,000 wires, and there were only approximately 14,000-15,000 shapes total. The accuracy for resistors (2000 shapes) was also extremely high, but the accuracy for the rest of the shapes was around 60-80%. So over 2/3 of the shapes were easy to detect with high accuracy because they were either a straight line (wire) or unique (resistor). It seems like the system has a very hard time distinguishing between similar shapes, such as batteries vs. capacitors, mainly because the histograms are just not accurate enough to catch small variations with noisy data.

I'm also interested to see how the histograms can be tweaked depending on the scale of the image. The histograms used were hard-coded to 40 pixel radii, and possibly having a variable histogram size would help.