Virtual reality provides a sensory experience with the help of sight, sound, and touch for an individual—but for assistant professor of performing arts technology Anıl Çamci it’s an opportunity to create a more immersive experience for large audiences.
In this episode of Michigan Minds, Çamci discusses his project Bringing Down the Fifth Wall, a pilot project funded by the U-M Arts initiative.
“The Fifth Wall project started out with the goal of bringing virtual reality music performances to large audiences,” he says. “To break that down a little further, virtual reality music performance can take several forms. Some of the musical applications we find in VR today are more consumer-facing applications, where anyone could go into VR and play with a musical system in the style of playing a game.
“But then there are VR performances where an artist or performer goes into VR and leverages this medium as a musical instrument in front of an audience. And when I mention a VR instrument, I don’t mean a virtual violin that you would play in a simulated space. I think that does injustice to both violins and VR.”
Çamci explains that virtual reality makes it possible to create instruments that would otherwise not be able to exist—instruments that defy gravity or the laws of physics.
“Maybe it’s an instrument where your body can pass through the instrument, or maybe it’s an instrument that is blown up to the scale of a planet. So these would be instances where the use of the virtual medium in this musical application is truly justified. But nevertheless, with current VR technology and consumer adoption, it is not yet quite possible to bring an entire concert audience into VR along with the performer.”
He also noted that audiences see many virtual reality performances through the performer’s headset, displayed on a projection screen, the image doesn’t come through in one clean and practiced motion and, in turn, can cause a layer of disengagement between audience and musician engaging. To mitigate this, VR performances can also be set up to showcase feeds on separate screens, but this also has its drawbacks.
“When you take the feed from someone’s headset and project it on a large screen, those minute movements become amplified and display a very jittery representation of the virtual space, so it doesn’t look very good,” Çamci says. “Another option for showcasing VR performances is to set up virtual cameras in VR and show the feed from those cameras to the audience. I’ve done that for many years, and it’s a very tedious process that requires planning every shot.”
Çamci touched on the subject of arts research and its importance in the university community.
“Research is a desire to understand or discover things that we have yet to understand or discover, artistic research can be viewed as a way to reflect on creativity and understand it. If art, as an introspective process, is one of the defining qualities of humans as species, understanding the nature of and the mechanisms that underlie artistic expression is a very important and rewarding process or pursuit. Artistic research often adopts interdisciplinary methodology in that pursuit and interdisciplinarity is not new to the arts by any means.”
Anil Camci
He notes that the overlap of art and science is not new, and that similar intersections exist throughout all art forms, but that the connection is more explicit because of advances in technology.
“The overlap between the arts, science, and technology underlies many artistic and academic practices today. Artistic research can stem from hypotheses just like in scientific research, or it can originate from more exploratory questions, like what does immersive media mean for the future of musical expression? And what unique possibilities do such media offer us in that regard?”
Much of Camci’s work had to be reimagined because of the COVID-19 pandemic: the initial project proposal was submitted in March 2020, when nobody had any idea what would happen in the coming months.
“The project came from a very practical need to realize these large-scale virtual reality performances with in-person audiences. And navigating a pandemic was not at all part of the initial proposal,” Çamci says. “The Arts Initiative organizers did a great thing there: they reached back out to the primary investigators and asked how their projects could or should be reimagined in light of the pandemic.
“We immediately started thinking about how some of the pieces we were working on could be fashioned into network performances and streaming performances,” he continues. “Some of the research plans, especially those involving user studies, had to be deferred….So although we preserved the core idea of the project as it was conceived, we had to adjust how we approached the idea and what kinds of outlets and applications it could lead to. Since then, we’ve carried out numerous network performances at conferences and virtual venues. And the project evolved in constructive ways.”
While Çamci led the way, he didn’t work alone. Two faculty collaborators, Erik Santos and Amy Porter, as well as many student researchers from the School of Music, Theatre & Dance, Michigan Engineering, and School of Information, came together to create the cinematography system that utilizes audio and user engagement through applying music information retrieval algorithms to real-time audio, which then identifies musical features to in turn control the camera system.
“When we think about the relationship between music and accompanying visuals or cinematography, there are some low-hanging fruits, right? The visuals can fade in or out along with the music. The cuts between the cameras can follow the rhythm of the music. But then there are more nuanced questions that are still ongoing aspects of our research,” he says.
“We are analyzing a host of musical features like noisiness, spectral centroid, and perceived sharpness. And we are asking, okay, how could those be mapped to the camera system? And on top of this, through the tracking technologies that are built into modern VR systems, we have access to data on where the performer is in virtual space, what they’re looking at, and what they’re holding or reaching out to. And all of these also serve as interaction cues.”
One of the main goals of the pilot project is to make the concert experience feel more natural even through the camera systems. Çamci touches on the performances that will be held showcasing the finalized project, offering audience members an engaging musical performance from an unusual perspective
“Erik’s [Santos] performance is set in this abstract outer space environment. We then fly from there down to Earth, and we were discussing concepts for these environments with Erik and Amy, and Amy expressed interest in performing at this place in Michigan she has a connection to. It’s Whitefish Point in Paradise, Michigan,” says Çamci. “So we took that as an inspiration, and we did not want to just recreate it because again, VR needs to go beyond what’s already there. So from outer space, we land on this island that has references to Whitefish Point, but then adds some surreal elements on top of it that react to Amy’s performance.
“Then we start moving into the ocean, the weather begins to take a turn, and we arrive at this station in the middle of the sea where I perform with a virtualization of my modular synthesizer, that’s my instrument, which is a hardware instrument. But the virtualization actually controls the real thing. So there’s this connection between virtual and physical spaces.”
As the project itself continues to evolve and Camci dives deeper into research in this space, he shares his goals for the project.
“As the project developed, we discovered educational applications. The problem of presenting a single-user VR experience to an audience is not exclusive to music performances. And with the growing use of VR in our classrooms across meta-disciplines, we run into that same problem. So for instance, in my immersive media class, there are times when I, as the instructor, need to go into VR to demonstrate a concept or technique that we are exploring.
“I think this project is a good example of how technology presents an opportunity for artistic expression in the beginning, in this case, virtual reality technology enabling a new kind of performance practice, but then through the interactions between these two domains, the arts begin to prompt a need for new technology.”
Anil Camci
Çamci suggested that students register for his course if they’re interested in learning more, and encouraged people to read about his work on his website. He also noted that this project would not have been possible without the support of the Arts Initiative, which offers funding opportunities for faculty and staff.
“I would encourage faculty and researchers at Michigan to engage in the Arts Initiative. There are already some really inspiring projects that grew out of the initial phase of the initiative, and they are not only supporting ambitious artistic projects, but also instructional activities. So I would highly recommend checking out the Initiative’s website and looking at the opportunities they offer.”
