It all began when Christianna Mariano ’21 took the first-year seminar “The Art of Noticing” and she noticed some art.
Specifically, Student’s Desk, then at the Mead Art Museum, the work of Elizabeth Hoak-Doering ’88. Picture a video of a graffiti-marked school desk, which spun from a ceiling, with a pencil dangling from it. When the desk moved, the pencil drew.
“It got me to thinking, OK, a desk is usually written upon, but now it’s writing,” says Mariano. She felt freed up, somehow, seeing this norm inverted: “My brain just started making all these associations. Like, a picture is usually looked at—but what if you listened to it?”
So she went online, found an image of a piece of art and fed it to the software program Audacity. “You can import audio files to Audacity,” she says. “But I wondered what would happen if I just imported a picture.”
What happened was this: the picture file data, which would normally be expressed graphically, was expressed audibly instead, so she could “hear” all those ones and zeros in its code. “I thought there’d be nothing, no sound,” says Mariano. “But it was so abundant, and such an interesting noise came out.”
What she did is called “sonification,” a word so new that many dictionaries don’t even include it yet. Tech sites define sonification as a sort of coding out loud. After that, Mariano (who is also a flute player and dancer) couldn’t stop thinking about what other seemingly silent things could be sonified.
She’s a neuroscience and theater and dance major, possibly pre-med, and when it came time to pick her classes last fall, she ran her proposed schedule by her adviser, Josef Trapani, associate professor of biology and chair of neuroscience. One course was “Sound Design,” taught by Jake Meginsky, visiting lecturer in theater and dance.
When Trapani asked why that course appealed, Mariano explained her new passion for sonifying art. He told her there were parallels in his own lab, which explores the auditory and vestibular systems of zebrafish by gauging how their hair cell receptors transform mechanical stimuli—like sound waves, gravity and changes in body acceleration—into sequences of electrical impulses (a.k.a. neural coding) that travel to the brain.
In short, Trapani’s research may help us better understand how we hear, and how we can hear better.
“So my brain is just firing,” recalls Mariano, learning of Trapani’s work. She got the idea to sonify the zebrafish larvae data—and this became her final project in “Sound Design.”
But how, exactly, did she pull this off? You can’t import a fish to Audacity. You can’t hook up tiny microphones to tiny swimming creatures.
The first step was to go to the IT department and talk to Andy Anderson, academic technology specialist. “I walked in and told Andy what I wanted to do, and he got a little spark in his eye and said, ‘This should be interesting.’” After some trial and error, they settled on using the Mathematica platform. “Then we took all this zebrafish data and created this audio file where, every time a neuron activation happened, there would be a sound,” says Mariano. In Mathematica, this sound was a speaker pop.
I thought there’d be nothing, no sound,” says Mariano, a neuroscience and theater & dance major. “But it was so abundant, and such an interesting noise came out.”
Meanwhile, at Meginsky’s suggestion, she attached contact microphones, which record vibrations, to the fish tanks and water pipes in Trapani’s lab.
To join the familiar sound of water and the unfamiliar sound of data, she used a sidechain compression technique in Ableton Live music software. Each time the “data” audio played a spike, the water audio would bottom out. Whoosh, whoosh, pop.
As Mariano puts it: “In essence, the absence of sound represents the presence of an activation, pulling you in this feel of deep tides and trends. I wanted to make it as simple as I could, where I would show the data in the most raw form, while still making aesthetic choices that were very distinct.”
In class, Mariano gave what’s called an “acousmatic” presentation, in which the audience listened to the work by sitting between sets of speakers. “It’s almost like the fish is swimming around you,” she explains. “And it gets faster and faster and faster and faster, and then it’s on all four speakers, and it’s just the data. So it’s kind of like this whirling transformation through the water into the world of science.” (Go to amherst.edu/magazine to hear the presentation, titled “Subtle Submergence,” on SoundCloud.)
Mariano traveled to the festival with Visiting Lecturer Jake
Meginsky. “The sound was amazing,” he says of her work.
“It was evocative.” She was the youngest artist to present.
Photo courtesy of Jake Meginsky
Meginsky was impressed. “Sonification is such a broad-ranging field with no turnkey solutions,” he explains. “It demands flexibility and engagement with ideas. Christianna made a lot of really elegant choices about how this was done. The sound was amazing. It was evocative. It felt musical but also had the quality of nature: organized, but highly complex and unpredictable.”
Not long after, Wendy Woodson, the Roger C. Holden 1919 Professor of Theater and Dance, reminded Meginsky that the prestigious Diffrazioni Multimedia Festival was upcoming in Florence, Italy (he is a filmmaker himself). Meginsky suggested Mariano apply to present her sonification piece. She did and was accepted.
Zoë Jacobs (in Amherst’s Center for Community Engagement) steered her toward the Gregory S. Call Student Research Fund, and Emily Jacobson (in the Dean of the Faculty’s Office) helped her apply for the funding. Meginsky received financial support from the theater and dance department to travel with Mariano to the festival.
“I’m really, really grateful for Amherst, giving me this opportunity and funding me to go,” says Mariano. “And
for hiring my professor, who is an out-of-this-world teacher.”
So, in March, they headed to Italy—where Mariano feels right at home, having gone often to visit her dad’s side of the family. Her parents met in Tuscany (“at a disco on the beach!” she marvels). Her American mother, who had taken a job as an au pair, fell for her father, then in the Italian Air Force. The couple moved to the Atlanta area, where Massimo Mariano is a chef and Rachel Mariano is director of student life at Gwinnett Technical College. Christianna is the oldest of three children.
At La Diffrazioni, Mariano—the youngest artist to present—made a splash. Eight huge speakers, a big audience, all listening within the walls of a renovated stone prison from the 1400s. Several members of her Italian extended family came to Florence to cheer on her performance.
“She was the best one there,” Meginsky told me when the two returned to Amherst.
“Not biased at all,” joked Mariano.
“It was amazing to be in Italy, and hear these other artists who are also working in this fertile ground between art and science,” she added. “I don’t know if there’s any way I can do art all day, and I don’t think there’s any way I can do science all day. I really want to find something in my life where I can do both.”
Katharine Whittemore is Amherst magazine’s senior writer.
Photos by Maria Stenzel