Final++ . Lenny Burdette . Programming Media II . UCLA D|MA . Spring 05

Assignment

Implement a concept of your choice within one of the following areas:

Mobile Phone Software
Camera Input / Computer Vision
Generative Animation
Generative Music
Artificial Life
Immersive Software Environment
Information Visualization

Research

Find past examples and uses of your chosen concept.

CAVE (Cave Automatic Virtual Environment)
- Wikipedia
- Electronic Visualization Library
- The CAVE at NCSA
- Fakespace Systems
- City Cluster (Ars Electronica 2003)
Random projects found on the web
- Afterwords (Nice construction diagram)
- I Am Driving Through Sound Space (Immersive sound)
- Run Motherfucker Run (The artist's site sucks)
- Isophone (Immersive or Submersive?)
Text
- Immersion and Interactivity
  A good immersion is supposed to get the user's brain closer to the default state, namely into a state where he is almost unconscious and forgets himself and just is. Interactivity asks for consciousness and that's why I think that interactivity and immersion don't go together and the impact of the piece can be stronger if the artist chooses either the one or the other.
- Myron Krueger Interview
Immersive Video Games
- Immersive Gaming.com
- 4orty2wo Entertainment (The Beast)
Random Terms and Thoughts
- "Immersive" isn't in the Merriam-Webster Dictionary
- Technologies for immersion:
  - Widescreen/Wall-length/Multi-wall/IMAX Screens or Projectors
  - Surround Sound/Hypersonic Sound/Headphones
  - Water/Submersion

Concept Proposal

In my research into immersive software environments, I quickly became disillusioned with the CAVE-type projected environments and similar projects, both for their effectiveness and the practicality of creating one for this assignment. Instead I chose to focus on immersive activity, such as watching movies, reading novels, or playing games.

Internet games are something many people lose themselves into daily as an escape from work and active thinking. Ferry Halim has some great, simple, innovative Flash games that I can play for hours. I started wondering if these games could be more engaging taken away from the traditional mouse/keyboard/monitor interface.

Because of my previous work with gesture, I chose to recreate Halim's Floats in Processing and create a physical interface for it. My idea for the interface grew from the game concept, but was also heavily inspired by Glen Murphy's System B1.

I presented this quick sketch of my ideas on May 16th, 2005

Infrastructure

I put together these small sketches separate so that I could piece them together for the prototype:

Gesture and closed shape finding
Color-based hit test
Camera blob detection (Cannot run in-browser)
Mouse controlled game prototype (web version is unusably slow. download the OpenGl version here)

The AI of the creatures in the game is derived from the steering behaviors of Craig Reynolds and the Processing adaptation by Daniel Shiffman.

A few Processing-related problems I encountered:

When using OpenGL and get(), you have to flip the Y-position (height-y).
When using OpenGL (on many, but not all computers), and comparing a color retrieved by get(), you must add 1 to each color component (up to 255 or your colorMode settings): red(color) = red(getColor)+1.
Capture.settings() does not work in Mac OS X.

I also started collecting the pieces for the physical interface:

Various "Superbrite" red and green LEDs, 9-volt batteries, battery connectors, and velcro.
A 5x5 foot screen made from opal white plexiglass and 2x4s, generously donated by Jay.
A projector from the D|MA labs.
A Logitech webcam.
A cart, and table, and a chair from the labs.

Prototype

Because at this step I ran into most of my technical problems, I have very little documentation.

I made two LED rings, one green and one red, with four LEDs wired in series and arranged in a square. The player holds the battery in his or her fist.

I had to alter my original hardware setup because the highlight from the projector lens made reliable blob tracking impossible.

View Original Setup View final setup

When tracking colored LEDs with a generic webcam, two problems arise:

The camera reads the center of both colored blobs as white, so comparing their hues is difficult.
The jitter from the webcam image will cause the blob detection to jump to other spots on the image with even low levels of that color component. So when tracking a red LED, you cannot project any red pixels onto the screen. Even brighter grays will be picked up as red.

Using Infrared LEDs was always an option, and would have eliminated the latter problem, but making the game two-player became important to me the more I played with my early prototypes. Using colored LEDs was my only option in this case, and after much trial and error I was able to differentiate between the red and green spots on the image. Unfortunately, I was reduced to using only medium gray and blue for the game graphics.

Refinement and Conclusions

I set up the installation in the EDA in Kinross on June 7th, 2005.

View movie of one person playing the game. (Quicktime)

At this point, the game play works as follows:

There are two players:
- The player with the red LED ring draws a gray line and tries to capture gray creatures.
- The player with the green LED ring draws a blue line and tries to capture blue creatures.
To capture creatures, draw a shape around two of them. The shape cannot contain a creature of the other color.
When you capture two or more creatures, they join together and spawn a creature of the other color. This is effective because the more creatures of the other color, the harder it is to capture your own.
When a player combines all his creatures to one, it spawns 10 creatures for the other player, letting the winner gloat while the loser scrambles to combine all his creatures. More creatures are created and game play continues indefinitely.

Technical Conclusions

I tried to design my concept so that I could get away from the traditional human-computer interaction but still focus mostly on programming and not hardware. It was a nice thought, but using computer vision for a game-like situation is a difficult task, and I spent most of my time working with the webcam image.

Overall, I'm pretty happy with the players' ability to physically interact with the system. Considering it is mostly parts scrounged together from around the D|MA labs (I only spent about $15 on the LEDs, batteries, and miscellaneous electronics), the system has a workable response time. A few hours more work into callibrating the system would help, but it is easy to tell how the LEDs and the response from the system align.

The worst part of the system, in terms of game play, is the inaccuracy of the shape recognition. If you complete a shape, only to have it draw unexpectedly and not capture the creatures, it can be incredibly frustrating. This takes the player far from an immersive experience. This can be witnessed in the videos.

Other technical steps needed for a better functioning system:

A better camera: The framerate and resolution of the webcam were more than sufficient, but the color accuracy and jitteriness of the image make color tracking very difficult.
Keystoneing: Because the projector is situated at an angle, the image is distorted. A project that can correct that distortion would help greatly.
LED rings: These just need to be assembled better, and some work needs to be done to equal out the brightness between the red and green (currently I am using a piece of hot glue stick to diffuse the red LEDs).

Conceptual Conclusions

I am a bit afraid that over the course of the five weeks, I lost the immersive gaming experience aspect of the project and that it instead became a physical interface project. The game became just a metric for judging the success of the interface; if it's good enough to play a game with, it's good enough for a multitude of other applications. The guest critics brought up many great ideas for what this system could be used for, including drawing programs and Golan Levin-like sound creation programs.

More than anything, it was exciting to see how well all these pieces could be put together in such a short time. With many of the technical issues overcome, ideas for more interesting, engaging, and immersive applications are much easier to come by. As a programming project, I wish it was a little more successful, but as my first installation piece, I am happy with my accomplishments and encouraged about future work.

Summary

Images

Code

Gesture and closed shape finding
Color-based hit test
Camera blob detection (Cannot run in-browser)
Mouse controlled game prototype (web version is unusably slow. download the OpenGl version here)

Video