Interactive Installation, Visual Programming, Speech Interactive Interactive Shadow Installation An audio interactive shadow that responds and sings songs for you
About
Duration
Role
Tools
Sound & Visual Interactive Media Art Digital Installation
March - April 2022
Personal project (Research Prototype
Touch Designer/ Microsoft Kinect/ Voice Tracking/ MaxMsp/Projector/ Miro
My responsibility
Ideation, prototype and present a real-time interaction performance This individual project is developed based on my personal interest to have a responding shadow.
Design process
Overview
About Shadow
The project uses both visual and verbal interaction shows the possibility of amity between the shadow and human through dialogues, expressions and vocabularies.
Question
Imagine talking to my shadow is one superpower that I have obtained, how can I communicate with and understand my shadow and what kind of relationship can we maintain? Can shadow becomes an extended interface of the body?
Goal
Create a shadow that can respond to my request, and sing songs for me.
Design Preview
Responsive shadow
“My body shadow is a friend of mine. Naturally, people might assume that their shadows are insubstantial compared to the body and consciousness. But I disagree. As I spent more time observing shadows and, one day, I found out that I obtained the superpower to communicate with my shadow using the shadow language. My shadow is a good singer, and has the singing skill one could never achieve. The shadow helped and supported singing performances whenever I asked.”
Concept The shadow has become an extended interactive interface or space of the main body. The shadow on the wall reveals a possible connection between the body and it as it moves according to the body motion. Yet, the relationship between human and shadow is blurred as perception, self-recognition and ownership is questioned through the performance.
Have you talked to your shadow? Are shadows presence or absence?
Ideation and Design
Interaction Mapping
The project is consist of two interactions: physical and speech based. In the physical interaction, audience can see a real-time interactive shadow that follows body movements. In the speech interaction, audience can verbally communicate with the shadow and ask the shadow to sing different songs for them.
Prototyping - Visual
Using Touch Designer, I am able to create the visual representation of the shadow.
Rather than having a light source going past the edge of the body, a projector is used as a window showing the computerized figure using camera capture to mimic the body shadow. The figure of shadow camouflages itself in designed shape and movements. The projector becomes the magnification of the illusory, which further questions the boundary between real and projective.
Shadow Mode I (left): Colorful shadow represent the active mode: the shadow is responding to the speech
Shadow Mode II (right): Colorless shadow represent the inactive mode: there is no speech/request, the shadow remains inactive
A time lag used to enhance the relationship between me and the show, showing the shadow as a follower of the body movement.
Prototyping - Audio
I used machine learning speech recognition technique to track my speaking, and send signals to Maxmsp to activate different responses based on different requests.
I used p5.speech and socket.io to achieve speech recognition and use words as a trigger to activate music in Maxmsp.
Screen hots from Maxmsp and Visual Studio
a live recorded experiment of using words to change the song the shadow sing
Audio and Visual Interactive
The visual and verbal interaction shows the possibility of amity between the shadow and human. It is constructed through dialogues, expressions and vocabularies.
When words are said, it triggers the music and also color changes of the shadow from grey to colorful.
.jpg)
.jpg)
.jpg)
.jpg)
.jpg)
.gif)
.jpg)
.gif)
.gif)
.gif)
