Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation

Ip, H. H. S., K. C. K. Law, et al. (2005). Cyber Composer: Hand Gesture-Driven Intelligent Music Composition and Generation. Proceedings of the 11th International Multimedia Modelling Conference, 2005.

Summary:

Ip et al. created Cyber Composer, which is a music generation program controlled via hand gestures. The author's motivation is to inspire both musicians and casual listeners to experience music in a new way.

The authors split music composition into three parts: melody, rhythm, and tone. The melody is the "main" part of the music and mainly includes the treble parts, such as the singer. The rhythm keeps the beat of the music and is played by the drums and bass. Tonal accompaniment involves creating harmony across all parts.

In order to keep the tone (harmony) of the music interesting and flowing, the authors create a small "chord affinity" matrix that describes certain chord lead/following strengths. During music composition, chords are automatically chosen with high affinity. Melody notes are also chosen automatically to create musical "tension".

The system was implementing using two 22 sensor CyberGloves and two Polhemus positioning receivers. MIDI was used to produce the musical notes.

The seven gestures used in the system include rhythm, pitch, pitch-shifting, dynamics, volume, dual-instrument mode, and cadence. Rhythm is controlled by the flexing of the right wrist. Pitch is controlled by right-hand height, and it is reset at the beginning of each bar. The user can also "shift" the pitch by performing a similar gesture. Note dynamics and volume are controlled by the right-hand finger flex, with fully flexed fingers forcing forte notes. Dual-instrument mode allows a harmony melody or unison melody to be played along with the main instrument; this mode is activated using the left hand. To end the piece, the left-hand fingers are closed.

There are no results.


Discussion:

This paper aroused me. Some of the gestures they defined were intuitive, such as opening and closing of the fingers for volume and moving the hand up and down for notes. Other gestures just seem awkward, such as the ambiguous dual-instrument mode and constantly flapping your wrist (ouch?) to drive the melody.

I'm familiar with building music composition programs (including "smart" programs that use musical theory to assist composition), and I think this program was trying to market itself as something that it could never become. A music tool has to be either robust to allow experts to use it, sacrifice some features to become simple for novices, or fun for just the casual listener. In the expert category I would place Finale, and on the casual end I would place music games such as Guitar Hero. Novice programs are harder to come by, and the tool I worked on was ImproVisor--a system that used intelligent databases to analyze input notes and determine if the notes "sounded good".

CyberGlove is trying to do everything at once and failing. The lack of any results, even the casual comment by an offhand user, tells me that the system is rather convoluted to use or poor for composition. The hand gestures cannot really control notes in a way that experts would use the system, novices will not understand the theory behind why their hand waving sounds good or bad, and casual musicians will probably have no idea what is going on.