News & Events The Sound of Science

It all began when Christianna Mariano ’21 took the first-year seminar “The Art of Noticing” and she noticed some art.

Specifically, “Student’s Desk,” then at the Mead Art Museum, the work of Elizabeth Hoak-Doering ’88. Picture a video of a graffiti-marked school desk, which spun from a ceiling, with a pencil dangling from it. When the desk moved, the pencil drew.

“It got me to thinking, OK, a desk is usually written upon, but now it’s writing,” says Mariano. She felt freed up, somehow, seeing this norm inverted: “My brain just started making all these associations. Like, a picture is usually looked at—but what if you listened to it?”

So she went online, found an image of a piece of art and fed it to the software program Audacity. “You can import audio files to Audacity,” she says. “But I wondered what would happen if I just imported a picture.”

What happened was this: the picture file data, which would normally be expressed graphically, was expressed audibly instead, so she could “hear” all those 1s and 0s in its code. “I thought there’d be nothing, no sound,” says Mariano. “But it was so abundant, and such an interesting noise came out.”

What she did is called “sonification,” a word so new that many dictionaries don’t even include it yet. Tech sites define sonification as a sort of coding out loud.

After that, Mariano (who is also a flute player and dancer) couldn’t stop thinking about what other seemingly silent things could be sonified.

She’s a neuroscience and theater and dance major, possibly pre-med, and when it came time to pick her classes last fall, she ran her proposed schedule by her advisor, Josef Trapani, associate professor of biology and chair of neuroscience. One course was “Sound Design,” taught by Jake Meginsky, visiting lecturer in theater and dance.

When Trapani asked why that course appealed, Mariano explained her new passion for sonifying art. He told her there were parallels in his own lab, which explores the auditory and vestibular systems of zebrafish by gauging how their hair cell receptors transform mechanical stimuli—like sound waves, gravity and changes in body acceleration—into sequences of electrical impulses (aka neural coding) that travel to the brain.

In short, Trapani’s research may help us better understand how we hear, and how we can hear better.

“So my brain is just firing,” recalls Mariano, learning of Trapani’s work. She got the idea to sonify the zebrafish larvae data—and this became her final project in “Sound Design.”

But how, exactly, did she pull this off? You can’t import a fish to Audacity. You can’t hook up tiny microphones to tiny swimming creatures.

The first step was to go to the IT department and talk to Andy Anderson, academic technology specialist. “I walked in there and told Andy what I wanted to do, and he got a little spark in his eye and said, ‘This should be interesting.’” After some trial and error, they settled on using the Mathematica platform. “Then we took all this zebrafish data and created this audio file where, every time a neuron activation happened, there would be a sound,” says Mariano. In Mathematica, this sound was a speaker pop.

Meanwhile, at Meginsky’s suggestion, she attached contact microphones, which record vibrations, to the fish tanks and water pipes in Trapani’s lab.

To join the familiar sound of water and the unfamiliar sound of data, she used a sidechain compression technique in Ableton Live music software. Each time the “data” audio played a spike, the water audio would bottom out. Whoosh, whoosh pop pop.

As Mariano puts it: “In essence, the absence of sound represents the presence of an activation, pulling you in this feel of deep tides and trends. I wanted to make it as simple as I could, where I would show the data in the most raw form, while still making aesthetic choices that were very distinct.”

In class, Mariano gave what’s called an “acousmatic” presentation, in which the audience listened to the work by sitting between sets of speakers. You can hear the presentation, titled “Subtle Submergence,” on SoundCloud.

Meginsky was impressed. “Sonification is such a broad-ranging field with no turnkey solutions,” he explains. “It demands flexibility and engagement with ideas. Christianna made a lot of really elegant choices about how this was done. The sound was amazing. It was evocative. It felt musical, but also had the quality of nature: organized, but highly complex and unpredictable.”

Image

Not long after, Wendy Woodson, the Roger C. Holden 1919 Professor of Theater and Dance, reminded Meginsky that the prestigious Diffrazioni Multimedia Festival was upcoming in Florence, Italy (he is a filmmaker himself). Meginsky suggested Mariano apply to present her sonification piece. She did and was accepted.

Zoe Jacobs (in the Center for Community Engagement) steered her toward the Gregory S. Call Student Research Fund, and Emily Jacobson (in the Dean of the Faculty’s Office) helped her apply for the funding. Meginsky received financial support from the theater and dance department to travel with Mariano to the festival.

“I’m really, really grateful for Amherst, giving me this opportunity and funding me to go,” says Mariano. “And for hiring my professor, who is an out-of-this-world teacher.”

So, in March, they headed to Italy, where Mariano—the youngest artist to present—made a splash. Eight huge speakers, a big audience, all listening within the walls of a renovated stone prison from the 1400s. “She was the best one there,” Meginsky told this reporter when the two returned to Amherst. “Not biased at all,” joked Mariano.

“It was amazing to be in Italy, and hear these other artists who are also working in this fertile ground between art and science,” she added. “I don’t know if there’s any way I can do art all day, and I don’t think there’s any way I can do science all day. I really want to find something in my life where I can do both.”