When you think about Mozart, you probably don't imagine a computer. But artists Marc Downie, Paul Kaiser and Shelley Eshakar hope you'll do just that: Their installation Enlightenment, running as part of Lincoln Center's 40th annual Mostly Mozart Festival, is an analysis, reconstruction and performance of the five-part fugue of Mozart's last symphony – all performed completely by computer.
Downie, who earned his Ph.D. in artificial intelligence (AI) last September, explains that Enlightenment is an algorithm that allows AI to analyze the coda and figure out how it's put together, using methods similar to DNA sequencing and Google searching. As the ten computers – each one representing a section of the orchestra – start from scratch and work toward the solution, they play and illustrate their guesswork live via speakers and high-resolution television screens. While monkeys locked in a room might eventually figure out the same piece of music, the team of computers reconstructs the 30 second coda—widely renowned as one of most complicated pieces of music ever written—in a barely a beat. In the installation, the sequence, which is completed from scratch each time, is slowed down to 35 minutes to allow the viewer to see the complexities of Mozart's composition.
"The musical knowledge that the computers have is extremely limited" when they begin their task, says Downie. "They know some of the fundamentals of how far apart pitches are. It's very generic." Even with just a few base rules, the computers are able to build their knowledge of harmony, fugal structure and counterpoint, which ultimately help them reconstruct the coda.
The program's 265,000 lines of code perform a process Downie likens to a series of games the computers must play, first individually, then collaboratively. "The first task [the computers have]," he explains, "is to, given all the notes an instrument can play, figure out which ones are used in the piece." The computer illustrates and plays the notes as it tests them, moving from "random notes, sort of the most brutal early 20th century atonality" to the correct key of the piece, with "snippets of Mozart coming in through the desire to get the intervals right," Downie says. If the necessary notes or durations aren't available, the computers will change the pitch or stretch the notes that they do have. "There's a little bit of live sample manipulation," Downie says, "enough to fill in the gaps for all the notes that don't match the current piece of music."
More From Popular Mechanics
Next, the computers move on to the second game – finding the themes of the piece. The coda of "Jupiter" is considered one of the most intricate pieces of music ever written, and each computer must discover the five themes using a process of trial and error. The computers illustrate their search via pop-up boxes and show their progress using two bars of notation: one bar shows the re-editing of the sequence with accompanying explanations for why the changes were made; the other bar tracks the audio playback of the new sequence.
Once each screen has discovered the five themes, they begin to collaborate with each other in order to play game three, where they must determine where each theme belongs in time. "When they get to an alignment between themselves and another screen which is satisfactory and statistically likely given the piece, they call that a solved problem, and it gets saved to tape," Downie explains. "They move on to another area, and eventually, when they fill the tape in, they can take it from the top." Visually, the instrumental parts rearrange themselves in a repeating loop, which eventually grows to the full 32 measures of the final coda.
That's when the fourth game begins: the actual performance of the coda, framed with the introduction and finale. The instruments, in abstract form, are now fully visible on-screen. At the end of the cycle, the artwork discards its results and starts over. Because the generated art is so large – at the end of its run, it will have animated 41 million pixels in real time generating 125 million frames of animation, which amounts to more than 18 petabytes of video data – it is impossible to store even a single performance.
Each performance of Enlightenment is totally unique. Although the artists have aesthetic responsibility for the visualizations created – "the colors are ours, and the line thicknesses are ours," Downie explains – they don't craft each frame. "The images are surprising to us, rather than designed by us," he says. "They're discovered." And, says Downie, he and his colleagues like the fact that the images generated are different to each person who sees them (this reporter, for example, found that the third phase reminded her simultaneously of an etch-a-sketch drawing and an architectural blueprint). "That was one of the things we were looking for, that uncanny familiarity to something," he says.
If there's anything Downie hopes the piece accomplishes, it's that it helps people see Mozart without all the typical biographical trappings attributed to him. "I don't think that's his legacy," Downie says. "It's not some intrigue about his death and his requiem. It's not about the powdered wig or people getting dressed up to perform him. That's not what he left behind. He left behind the music. This piece is about the music, the answer is in the music. What makes it special was not that he was precocious or young, or anything like that. It was the music. That's why we remember him."—Erin McCarthy
Enlightenment runs 24 hours a day until August 28 in the colonnade of Avery Fisher Hall at Lincoln Center. For more information on the Mostly Mozart Festival, visit the Lincoln Center website.
Automatic Notation: Illustrated by lines, the notes weave together as the computer tries out possible combinations.
Stage Two: Find the themes. The computers' progress is shown using two bars of notation and a search box.
Stage Three: Solving the Fugue. The screens work together to place the themes in time, illustrated by a repeating loop.
Stage Four: Perform the Coda. Abstract representations of the instruments continually adjust themselves as the computers play the fugue.