The amazing invisible world of the Kinect

It has been both a pleasure and a challenge to work with the Kinect as material. The level of abstraction is a lot higher with the Kinect compared to sketches made with Arduino. We have to think very different when designing for a Kinect because the translation of the input is so abstract compared to the Arduino designs, where you mostly work with very haptic designs and due to this the inputs and outputs are mostly easier to map.

U N C A N N Y – P L A Y G R O U N D from Jette Forstholm on Vimeo.

With the Kinect we have to work with the body as the interface or at least this is what the traditional Kinect is known for. But with the processing hacking of the Kinect’s it was possible to use the Kinect’s in a very different way. Now it was possible to fx place invisible buttons with sound effects. This is a very different way to interact with a piece of art, since the haptic interaction is not necessary at all.

We started of with a projector exercise where we wanted to understand how we could place these non-haptic buttons in mit air so they could trace a person walking up a staircase. Every step had color attached to it so a person walking up the stairs would be changing the colors of the staircase.This gave us lots of ideas of have to design this; one of them was where we had projection up under every single box in ITU’s atrium, and when a person walked up the staircase, the steps would individually be connected to one of the projectors. This would be best experienced in a dark surrounding.

We continued working with Kinect , staircases and projections in our next idea: We wanted to create an illusion where a person walking up the staircase would have a feeling of separating a river or ocean. This design was somewhat sublime.The feeling of both walking against the stream AND having the power to separate it with ones mere presence would at least to some extent evoke a feeling of the sublime (Kant) perhaps a feeling of dynamic sublime where an overwhelming feeling of a forceful element. We didn’t go with this idea both because of some practical issues but also because we didn’t believe that the look and feel of it would be as nice in real life as in theory.

This was mostly due to the problem of the shadow that the person on the staircase would to some extent ruin the illusion. Furthermore, it would also take a lot of time to set up, thus we would have shorter timer to play with the design.

With the experience of the projector in fresh in mind we quickly decided that sound would be a much more satisfying output, because the audience couldn’t disturb it with their physical presence. We took some of the experience we had from our first design and worked with buttons floating in mid air containing sound. At some point we got the idea to place the sound containing buttons in a height where they couldn’t be trigged by a human alone: the participant had to use a ball or balloon to throw up in the air and this would then trig the buttons. In this way our final Kinect design invites to a playing and a social attitude with an experience for both a single participant and several participants throwing the ball or balloon to each other. The design can evoke a feeling of wonder, because since the buttons are invisible, where else in this space above my head and around me does the air contain sound? And what else could be hidden here in the invisible digital layer mixed with our physical haptic world? If the audience didn’t know of the Kinect, this would enhance this wonder even more. We all try to map how this and that works, but if there is nothing to be seen, then it can give a really magically experience. In this way this design is more in relation to the transcendental sublime, in which the digital layer is beyond our human reception we need technology to access this. In some designs maybe we need an understanding of the technology too to be able to understand how to interact with it in a way, which makes sense to us. But at the same time I believe this design would have worked without a full mapping of the experience. Through the interface, which is objectified, we invited to a interaction, which is involving for both the audience and the interacted. Furthermore, it had a sense of haptic design, because the input came through the balloon or the ball being thrown. This made the interaction with it very appealing, because everyone can throw a ball. The choice of music added to the sublime feeling of the installation, because it was in sharp contrast to our childhood memories of both a floating balloon and playing with a ball throwing it to each other. The music was uncanny, dark and created a universe of uncertainty in which the playing part of the design woke our perception of the installation more than if it had been in total symmetry with our expectations of childhood memories of play. It was in category with some of the latest games for smartphones and tablets for adults (e.g. Yearwalk).

I really like designing with the Kinect because it has this amazing possibility to create wonder due to the limitlessness it evokes. We have worked with this limitlessness through the feeling of sound placed in invisible layers above us. I really like the thought of how these invisible digital layers is all around us in our physical world, not just in relation with the Kinect but also through e.g. geotagging sounds (Recho). It occurs boundless to our mind if the kinect is hidden more, or if you do not understand the nature of the digital medium: It’s not obvious. I can’t help be imaging all these invisible threads of digital signals we are all walking through every single day: Like ghosts spreading across our physical space. It’s amazing!