Posts Tagged ‘computer vision’
Aural Aura
Aural Aura in Development
Harvestworks July 09
Current developments I am currently in the process of replacing the flex sensors with hand made sensors made from neoprene and velostat. I am also examining the possibility of eliminating body sensors in favor of a more sophisticated motion tracking system with the camera, which allows the precise identification of specific moves. At the same time I am implementing RGB LEDs mapped to the chakra points on the body suit, which are illuminated through the astral body dress at the precise moment when each chakra is triggered.
Aural Aura
Aural Aura is a computer vision and sound installation/performance that visualizes color frequencies and sonifies resonances of key energy points of the body, generated through movement and triggered using sensors. The project aspires to enhance energy management through a mapping of color, sound and movements based on the Shaolin 18 Lohan hands, the fundamental chi Kung exercises used for meditation since 527 BCE in China. Active and/or passive bodily movements trigger specific sound frequencies and colors mapped to the chakras. The goal of Aural Aura is to to unite sound, color and dance in a fluidly attuned communion that transcends the performer, and ideally the audience into a hyper reality that approaches a spiritual experience.
The Structure
The nature of Aural Aura is fivefold involving computer vision, a wearable interface, an astral body projection, musical compositions and sensors.
The Bodysuit
The bodysuit is the controller and generator of sounds and visual effects according to the readings of the attached sensors. A choreography of 8 moves measured by the sensors, determines the triggering of sounds, and images. The sensor bodysuit acts as an instrument capable of playing the 8 tone scale resembling the lydian mode scale, that is, the white keys of the piano. Choosing a wearable interface as an event based generator for mapping gestures was necessary in order to identify and associate chakra to sound and color.
A very special thanks to Andrea Lauer whose contribution made the execution of the design of the body suit into a costume possible.
The Gloves
A pair of gloves use two Flex sensors, one soft potentiometer, one FSR and soft switches, to provide continuous controllers and switches affecting rhythm, pitch, reverb and delay. The gloves went to multiple design variations. In the final design of the glove I used soft switches made of conductive fabric and tightly swed the remaining wire wrap onto the surface of the glove. For an update please check the link Musical Gloves.
The Astral Body
An exoskeleton wearable layer, the astral body dress, physically extends outwards from the central bodysuit, representing the aura and serving as a canvas for projecting images. The astral body dress is used for projection to visualize the aura in three dimensional space rather than on a screen. Visual effects projected on screen merge with the astral body dress color effects to simulate the aura.
Computer Vision
As computer vision tracks movement, the screen/body projection reflects chakra points being triggered by glowing the equivalent colors and/or symbols, while corresponding sounds are generated in realtime.
The Sound
The musical compositions attempt to capture the ethereal aspect prevalent in the visual representation of the aura. The estoreric aspects of music inform the process of sound design as different energy points are mapped to distinct instrumental sounds and musical intervals based on the Pythagorean tuning system. Sounds convolve harmoniously into each other as gestural movements trigger different energy points (chakras). Pure oscillators are synthesized into alpha and delta vibrations, intervals and resonances that accompany the choreography of movements. The low vibrational frequencies used help induce meditative states. Sound does not accompany the dance but is instead created by the dancer.
The Mapping
Aural Aura relies on a mapping between energy point (chakra), color, sound frequency and movement, to produce a synesthetic experience that physically manifests the invisible and inaudible nature of the aura. The mapping of color to sound is based on “Rainbow Resonance”, a project I developed in Spring 2008, which assigns sound frequencies to colors in an intelligible manner by transposing light wavelengths 40 octaves below their spectrum to match the audible spectrum of the electromagnetic field. Aural Aura adds the element of movement to achieve an immersive synaesthetic experience.
The Moves
When the 8 shaolin moves are performed the chakra centers are activated allowing for a connectedness between body, mind and spirit, that enables a “balancing” of the energies to occur.
Why a Performance?
Performance art is an unprecedented expressive experience. Inherent in its nature is the capacity to involve the audience so that they in turn may become a part of the performance. Interactive art is a performative art that invites the audience to not only participate but in essence create the art. In this performance the dancer is not merely interpreting an aural event, but “invents” the soundscape through the dance. Through a tight mapping between color, sound and movement that allows a connectedness between body, mind and spirit, Aural Aura seeks to transcend the performer, and ultimately the audience into a “healing” experience.
Healing
The healing element of Aural Aura is implicit. The performance seeks a balancing of the energies of the body, through an alignment of the chakras using sound vibrations in conjunction with the corresponding colors, as chi kung movements activate the chakra centers.
The Future
Aural Aura can be expanded to integrate movement in three dimensional space through a mapping of the physical location in which the performance takes place. Sensors that measure parameters such as distance, vibration and force, can be coupled with multiple motion tracking cameras to enrich the performance. Furthermore, a performance could include a group of dancers who in an ideal user scenario compose symphonies through their dance, using a combination of wearable sensors, and tracking movement in 3D space through computer vision and sensors placed on location.
Inspiration
Aural Aura is a natural development of my previous project Rainbow Resonance which acted as a computer vision musical instrument. In this project the idea of producing sound through dance is further explored as the performer generates sound intervals through a mapping of her movements. The content of this project sprang from the desire to visualize the aura that extends from the physical human body, and to sonify the associated colors, using a mapping based on mathematics.
Symbolisms in Aural Aura
- Star tetrahedron
The metaphysical nature of numbers as they interact to construct the shapes of the five pointed and six pointed star is an underlying theme conveyed through all aspects of this project. Attuned to the immaterial reality of sound and color, Aural Aura attempts to physically manifest the immaterial dimensions of being. 
Wearable Design
In designing the form of the astral body dress I was inspired by Issey Miyake’s designs extending from the body in unique flower forms, and also Philips Design Probe Skin Dresses. My intention was to illuminate the astral body dress “from within” so that I would achieve a “glowing effect” in the areas of the chakras. Miyake’s illuminated dresses and Hussein Chalayan’s video dresses served as an inspiration. Laurie Anderson, Pamela Z, Letitia Sonami, Luibo Borissov inspired me with the use of a wearable interface and sensors to produce sound and images. Arleen Schloss’s fibre optic dress by Maurice Daniel designed in the mid 80s was also one of my influences.
Arleen Schloss performing in a fibre optic dress by Maurice Daniel 
http://www.sofiart.com/wp-content/uploads/2009/06/Asdressw.mov
Probe Skin Dress by Philips Design
http://www.youtube.com/watch?v=WRX-3DDBow0
Issey Miyake
[youtube=http://www.youtube.com/watch?v=YDVhDddnta4]
Liubo Borissov – Autopoiesis
http://2rem.net/projects/reel/
Laurie Anderson – Zero and One
http://www.youtube.com/watch?v=r3l9wwp_srg&feature=related http://www.laurieanderson.com/
http://www.youtube.com/watch?v=gR969oeASVI http://www.pamelaz.com/
Toni Dove
http://www.troikaranch.org/vid-loopDiver.html
http://www.sonami.net/ http://www.sonami.net/lady_glove2.htm
The OpenEnded Group - Paul Kaiser
http://www.openendedgroup.com/index.php/artworks/a-to-b/
Dance
The Taiwanese dance group Cloudgate who use tai chi in their performance Moonwater was one of the most important inspirations that drove me to implement Chi Kung as a dance in Aural Aura. A beautiful solo performance by Yang LiPing entitled Moon that uses Kung Fu moves in the dance was another of my inspirations in designing Aural Aura.
Cloud Gate – Moonwater
http://www.youtube.com/watch?v=fmzMCetOM6w
Moon – Solo Dance by Yang LiPing
http://www.youtube.com/watch?v=ZkLrFpo0lHA&feature=related
Thanks To
Brian Gruber
The Trickster – Interactive Performance
Thursday, October 22 at 6PM – New York Public Library – Hamilton Fish Park
Sofia Paraskeva and Elodie Lauten: The Trickster
The Trickster is a collaborative sound and interactive installation with choreography by Khoi Bao Le and visual interactivity by Sofia Paraskeva. This performance is an offshoot from Lauten’s new work The Two-Cents Opera, involving interactivity between movement, music, and image projections triggered by dance and wearable components.
Khoi Bao Le performing for The Trickster at the New York Public Library – Hamilton Fish Park, Oct 22nd 2009. Created by Sofia Paraskeva and Elodie Lauten.
The Trickster – An Overview
A playful character performed by Khoi Bao Le (a young choreographer from Viet Nam in training at Martha Graham School of Contemporary Dance), unexpectedly triggers sounds from his own movement and costume details, while his image is projected on screen. Within a sound environment of pre-recorded electro-acoustic music by Elodie Lauten, the performer initiates with his movement a new layer of interactive sound samples in an improvisational framework programmed in MAX/MSP by Sofia Paraskeva. Elodie Lauten is a classical and experimental composer whose work has been presented by the Lincoln Center Festival, the New York City Opera, WNYC, The Kitchen, the Performing Garage, the Dance Theater Workshop, La Mama, the Soho Baroque Opera, Downtown Music Productions, AFMM, Interpretations, the SEM Ensemble, The Whitney Museum, and at the Paris Museum of Modern Art. Her current discography includes 27 titles to-date, on both independent and major labels. She has taught on the composition faculty at New York University.
Images from the Trickster at the Op On Screen Festival 2009
New York Public Library – Hamilton Fish Park, Oct 22nd 2009. Created by Sofia Paraskeva and Elodie Lauten.
Screenshot from The Trickster program – MAX/MSP
Elodie Lauten, Khoi Bao Le and Sofia Paraskeva working together for The Trickster
Rainbow Resonance – Interactive Installation
The Op on Screen Interactive Festival
Thursday, October 15 at 6PM – New York Public Library – Hamilton Fish Park
Sofia Paraskeva: Rainbow Resonance
Rainbow Resonance is a computer vision installation that generates colors and musical sounds of the equivalent sound frequencies in the electromagnetic spectrum, according to the motion of the participator. The audience is encouraged to participate in a playful performance that engages the body to produce colorful images and sound resonances through simple movements.
Rainbow Resonance at the New York Public Library – Hamilton Fish Park, October 15, 2009. Created by Sofia Paraskeva
In Rainbow Resonance anyone can potentially create a personal experience within the installation. The color scheme is based on the visible light spectrum and is also inspired by the chakras. A video camera is used as a sensor to detect motion, and motion tracking software technology is used to map the vertical and horizontal “zones” of movement in the space. The “zone” with the most vivid motion determines the predominant sound generated by the participant. This piece was shown at the Interactive Telecommunications Program Spring Show 2008 at NYU and was exhibited at the New York Hall of Science in October 2008 and February – April 2009. Sofia Paraskeva is an artist who experiments with interactive media, film and video. She explores sound and visuals in the context of leading edge technology. She recently graduated from the Interactive Telecommunications Program (ITP) New York University, developing computer vision projects and wearable musical instruments such as a musical glove.
Images from Rainbow Resonance at the Op On Screen Festival 2009
More inforomation on Rainbow Resonance.
Rainbow Resonance – Computer Vision Performance
Rainbow Resonance is a computer vision installation that generates colors and musical compositions of the equivalent sound frequencies of the spectrum, according to the motion of the participator.
Rainbow Resonance at Ms Viewpoint Ltd, Nicosia, July 8th 2010.
Click to watch video
The audience is encouraged to participate in a playful performance that engages the body to produce colorful images and sound resonances through simple movements. Anyone can potentially create a personal experience within the installation. The color scheme is based on the visible light spectrum and is also inspired by the chakras (energy points on the body).
Rainbow Resonance at ARTos Cultural and Research Foundation, November 2010.
Click to watch video
Color to Sound Mapping
Rainbow Resonance the New York Hall of Science, October 2008, Spring 2009.
Light wavelengths are transposed 40 octaves below the visible light spectrum, into audible sound frequencies and mapped to an 8 tone musical scale based on the Pythagorean tuning system. My intention was to use natural rhythms as a reference for an alternative intonation system to represent colors. By trying to map colors to sounds I was surprised to discover that a scale analogous to the equal tempered scale was possible to design, matching the light wavelengths in the spectrum, to a single octave in the sound frequency spectrum.The resulting eight tone scale corresponds to the “white keys” on the piano, and represents the seven predominant colors of the light spectrum.
In the color to sound mapping of Rainbow Resonance, a universal correspondence of sound to color is sought in order to produce an ultimate synaesthetic experience that transcends the participant into a meditative state. This interactive performance seeks an alignment of the chakras through sound and light vibrations, while the Chi Kung meditation exercise “Carrying the Moon” activates the chakra energy points to enable the balancing of the energies of the body. Through dance, play and movement, the audience may be entertained, and healed from the stress of the day.
User Experience
Visitors experiencing Rainbow Resonance at Ms Viewpoint Ltd, Nicosia, July 8th 2010.
A lingering sound invites the visitor to enter the space of the installation. Upon entering, the visitor faces a mirror image of him/herself projecting the colors of the rainbow, which are mapped to the figure according to energy points on the body (chakras). Simple movements generate harmonic intervals, while the equivalent colors brighten up. Movement along the vertical axis iterates through the color musical scale, while movement along the horizontal axis adds richness to the sounds with additional chords. An all encompassing composition of the frequencies occurs when the arms are extended upwards towards the sky. The movement is reminiscent of an exercise in Chi Kung (Qigong), an ancient Chinese method of healing and strengthening the body, mind and spirit, called “Carrying the Moon”. “Carrying the Moon” is an exercise that focuses on enhancing youthfulness by keeping the spine young.
Children playing at Rainbow Resonance ARTos Foundation, Nicosia, November 3rd 2010.
More info on Rainbow Resonance at ARTos Foundation, Nicosia
The target audience is everyone who enjoys playing with colors and sounds. This interactive experience can be modified to benefit children with disabilities, and can be used as an educational and recreational tool for movement through play and dance.
Special children enjoying Rainbow Resonance at Ms Viewpoint Ltd, Nicosia, July 13th 2010.
Click to watch video
more info on Rainbow Resonance at MS Viewpoint, Nicosia
Chakra associations
In Rainbow Resonance the chakra energy points are associated with specific colors as well as sounds and are mapped to different organs, vertically along the y-axis of an upright body, beginning with red at the lower abdomen area, orange at the navel area, yellow around the stomach, lime-green at the heart , blue/turquoise at the throat, and purple at the forehead and above.
Chi kung – A healing exercise
Chi Kung began when prehistoric man discovered how to manipulate breath, as a form of cosmic energy, and according to Chinese records by 2700 BCE it had become an important aspect of chinese medicine. The earlier type of Chi Kung was probably a form of meditative dance which encouraged energy balance in the body. Chinese martial arts, especially Shaolin Kungfu make use of Chi Kung to enhance fighting abilities. Carrying the Moon is one of the best exercises in Chi Kung and can be practiced in many styles. A Chinese saying tells us “You need not worry about growing old, so long as your spine is young”. Carrying out the exercise daily enhances youthfulness and health at 60 and beyond and strengthens the kidneys, thus enhancing sexual vitality.
A Musical Interface – The Body as an Instrument
Arianna Economou dancing at Rainbow Resonance, Ms Viewpoint Ltd, Nicosia, July 8th 2010.
Rainbow resonance focuses on a human-centered experience that uses the body as the ultimate instrument of expression through dance, play and movement. It utilizes ambient intelligence technology (computer vision) to empower the participant by enabling them to playfully interact with a software mirror that generates sounds and visuals.
Alexis Vassiliou dancing at Rainbow Resonance, ARTos Foundation, Nicosia, Nov 3rd 2010.
Click to watch video
Technology
A night-vision video camera acts as a sensor to detect motion in space and background subtraction is used to isolate human presence from the environment. Motion tracking is used to map the “zones” of movement in the space and on the screen. The “zone” with the most vivid motion determines the predominant sound generated by the participant. Other hardware components included in the space are infrared lights, a laptop, speakers, a projector and a human-size screen. MAX/MSP/Jitter and OpenGL are used for motion tracking and visual effects, and, Max for Live (M4L) executes musical sounds using samples, AM and FM synthesis.
Esoteric Music Theory
The esoteric aspect of music theory runs deep and wide. In this project I have only attempted to scratch on the surface of this complex field of study. It is admittedly intriguing and inspiring to explore correlations between the physical world and symbolisms that pertain to the metaphysical realm, both part of nature.
Inspiration
“Through Vibration comes Motion
Through Motion comes Color
Through Color comes Tone”
Pythagoras
Pythagoreans believed that the universe is constructed according to the laws of the divine science of mathematics and that mathematics is the bridge between the visible and invisible worlds. In the Pythagorean tradition color is bound by the laws of harmony as all else in nature. Pythagoras was convinced that by aligning the forces of light and sound human beings could be healed from ailments, and he carried out the practice of “Medicinal Music”.
Background
The background for this project starts from an interest in mapping colors to sounds and vice versa, to examine the different emotional, mental or spiritual states they might evoke. I had always wanted to “hear the colors” and “see the sounds”, and wondered whether colors or sound frequencies actually depict or evoke specific emotions or thoughts. The idea of using the chakras as a color reference was inspired by a preoccupation to “see the invisible” by photographing people’s auras. The chakras are associated with specific colors as well as sounds. However, different theories that associate sounds to the chakras and their equivalent colors have not reached a consensus. Today, scientists use sensitive devices such as SQUID (Superconducting Quantum Interference Device) instrument technology to map the ways diseases alter biomagnetic fields around the body. Tests are being carried out to measure the magnetic fields around energy points on certain Qiqong masters.
Rainbow Resonance at ARTos Cultural and Research Foundation, Nicosia
Rainbow Resonance at MS Viewpoint, Nicosia
Rainbow Resonance at Harvestworks, New York
Rainbow Resonance at the ITP Spring Show 2008
Click to watch Rainbow Resonance at the ITP Spring Show 2008.
Preliminary worksheets
