What is soundSense?

soundSense is an exploration of the possibility of representing human identity and motion through sound. The fundamental goal is to understand how computers understand and communicate with people. Imagine that a computer senses “Jane Doe is walking across the room.” It may be hard to program the computer to recognize and articulate this fact. Through soundSense, researchers hope to program computers to communicate complex states to people without directly programming the articulation.

How does soundSense work?

The soundSense experiments take place in a studio instrumented with infrared motion detecting sensors. As people move around the room, the sensors record information such as speed, trajectory, groups, swinging arms, turning heads, moving legs, etc. The sensed information is translated into tones that incorporate many different perspectives. In combination, these tones simultaneously communicate the range of activity.

Why use sound?

Sound is an intriguing method for conveying complex information because our brains routinely process layers of sound all the time (think of a Beethoven symphony). The point of the experiment is to create meaning, however, not music.

soundSense explores the intersection of reality as it is perceived and communicated by humans and computers. "Reality" in this case means the positions and motions of people in the studio. Although computer systems can detect activity in the room, they are unable to convey it a manner that is as rich in nuance as the actual event. By representing electronically detected data as sound, we explore people’s ability to discover and articulate patterns of sound and then associate that with a vision of reality. Through soundSense , researchers hope to autonomously transform the "sentience" of the room into a sound communication understandable to people.

How do experiments such as soundSense help engineering research?

soundSense is helping researchers understand how humans and machines understand geometry and objects. We are used to the idea of "imaging," which means creating something that looks like something else. A human looking at an image understands it by creating an intellectual model for the imaged space. Machines cannot be programmed to "understand" in the same way. In the information spaces studio, Duke engineers are studying non-image sensors that may allow machines to "understand" objects and people by creating simple data structures (of sound in this case) related to the geometry of the space and objects it contains.

Through soundSense, Duke researchers (including electrical, musical and language "engineers") are gaining new perspectives on what it means to understand objects in a space. The primary goal is to create an autonmous system for creating a musical image, where in this case "image" refers to a sound created from a scene that evokes feelings and thoughts about the scene that are similar to feelings and thoughts that a visual image of the same scene might evoke.

How could soundSense be used in a practical application?

Area and perimeter security is a possible application for soundSense technology. For example, the Department of Defense needs large autonomous sensor networks to enable machines to track human activities in secured spaces and to enable machines to present complex distributed sensor data to security personnel. SoundSense techniques could help airport security identify abnormal activity in a crowd of travelers or help parents monitor children from another room.

On a longer time scale, soundSense is an exploration of “information spaces,” which use digital information to enhance common environments. We are currently used to environments with fixed portals into the digital world, typically represented by computer monitors and keyboards. Imagine a space in which lighting, sound, smells, breezes, etc. automatically adjust to human motions. Such information spaces will be based on low-power autonomous sensor networks based on soundSense technology.

soundSense Participants

The soundSense experiments are a collaborative venture by scientists at the Fitzpatrick Center for Photonics and Communications, Duke’s Department of Music and the ISIS (Information Sciences + Information Studies) program.

• David Brady, Fitzpatrick Center for Photonics and Communication Systems, Pratt School of Engineering
• Scott Lindroth, Department of Music
• John Bower, Department of Music
• Joe Donahue, Department of English
• Rachael Brady, Duke Department of Computer Science and the Electrical & Computer Engineering Department, Pratt School of Engineering
• Steve Feller, Fitzpatrick Center for Photonics and Communication Systems, Pratt School of Engineering
• Casey Alt, Information Science + Information Studies
• John Burchett, Electrical & Computer Engineering Department, Pratt School of Engineering
• David Zielinski, Electrical & Computer Engineering Department, Pratt School of Engineering
• Evan Cull, Electrical & Computer Engineering Department, Pratt School of Engineering
• Zach Pogue, Information Science + Information Studies
• Richard Lucic, Department of Computer Science
• Mohan Shankar, Electrical & Computer Engineering Department, Pratt School of Engineering
• Christian Liu, Electrical & Computer Engineering Department, Pratt School of Engineering
• Rhazes Spell, Electrical & Computer Engineering Department, Pratt School of Engineering

Special thanks to:

Anya Beklina
Chuck Catotti
Pricilla Wald
Rob Clark
Rob Sikorski
Kathy Hudson
Cathy Davidson