Artist Profile: Matteo Zamagni Uses VR to Hack the Senses

What is information? What is its relationship to meaning and structure? Does information exist apart from the material or medium by which it is transmitted? Matteo Zamagni’s VR experience Nature Abstraction offers a fascinating exploration of these questions. Like VR itself, the work challenges conventional notions of human perception and knowledge.

Divided into three sections, to which Zamagni gives epic/spiritual overtones by calling them “Birth,” “Communion,” and “Aether,” Nature Abstraction literally “makes visible, invisible structure.” First, the artist uses mathematical equations to generate fractal geometries. He then feeds these images into Google’s Deep Dream , an artificially intelligent neural network that analyzes and enhances patterns in visual images. In this way, Zamagni procedurally recreates the process by which consciousness moves from mathematics to visual abstraction to recognizable representations. The overall effect is both familiar and strange. 

“There is no absolute truth in the way we see, and more in generally, in the way we perceive. Our experiences are always biased from our sensory perceptions, our tools to relate to the world. In the case of DeepDream it was meant to be another type or level of perception of the same form.”

Matteo Zamagni

In the first “world,” Zamagni moves the user through a highly organized space of metallic cubes or cells which resemble Cantor spaces . Thus, “Birth” is associated both with a regular structure and an uncountable, infinite set.  This is a key idea operating in Nature Abstraction. Although the structure of physical bodies consists of millions of identical cells, on a macro level these self-organize to create the unique, singular form of a perceiving individual. 

The second world, “Communion,”  offers more organic, rounded structures that suggest  snow flakes or other naturally occurring decorative patterns. In this space, the user feels her body relax. It is as if she is floating through coral reefs under the ocean.

The final world is even more fluid. It  suggests a return to the flow as occurs with death and the dissolution of embodied consciousness. In all of these, the user finds no way of orienting herself— no horizon, no ground, and no way of determining scale. Everything is depth and surface together.  Depth is a dimension, after all, which is not merely relative, but artifactual. It results in part from the left and right eyes having slightly divergent views. In other words, the dimension of depth largely depends upon a body that sees in this peculiar manner. 

The primary role of the body in perception and epistemology , which philosophers such as Charles Peirce , Henri Bergson , and Merleau-Ponty have described, is being born out by neuroscience. The sense of presence , that feeling of actually being in a space, which is so critical to VR, is mediated via a body’s ability to move through an environment. Thus, presence, itself, is intimately related to how we see and organize the world. Moreover, VR makes apparent the role of the mobile body in determining visibility and invisibility, which is only hinted at in two dimensional media such as painting and photography. In Nature Abstraction, as the user’s body is moved through the space, previously invisible layers of structure appear and morph. Using a cleverly timed process of animation, Zamagni makes it seem as if user movement through the world actually co-produces it.

Zamagni uses DeepDream to extend this process of perception to machines. The DeepDream software finds patterns. It makes other surfaces visible. Where previously, we saw “empty” space, now we perceive what the AI sees—more patterns. In “Birth.” it sees towers or dwellings of some sort. In “Communion,” faces appear, in “Aether,” concentric rings like orbits or the eyes of peacock feathers. This is how humans see—we are trained to see and seek meaningful patterns just as the Google’s neural network is. Our own process of pareidolia is largely subconscious. 

Importantly, Zamagni further complicates these images. He opens up yet another surface or dimension by adding glitch .

“Glitch in my works is about destroying the image in a way that gives the audience another point of view, another perspective of the work. In Nature Abstraction, the slow mindful pace is suddenly broken by these glitches which then introduce the Deep Dream patterns…another, deeper level of reality exist beneath, but you could have not seen before if not for that shift of perspective.”

Matteo Zamagni

Modern information theory began with the Macy Conferences on Cybernetics . Claude Shannon defined information in terms of the efficiency of transmission of the signal, not meaning or structure. Donald MacKay , another conference participant, offered an alternative view. He argued that although it made information difficult to quantify, information theory had to take into account meaning—a metamessage that told the recipient how to read the message. Because it was seen as mathematically quantifiable and “scientific,” Shannon’s version won the day. According to N. Katherine Hayles , the unfortunate legacy of Shannon’s information theory is that it “constructs information as the site of mastery and control over the material world.”

By relying on both the viewer and machine perception Nature Abstraction inserts materiality back into the “pure” notation of mathematics. The code may be abstract and universally legible to computer systems, but the meaningful output is not. This is a very different concept of abstraction than one put forth a century earlier by Malevich and others who sought a universal language through the removal of representation. Zamagni explains:  

“Since fractals, resemble very familiar shapes although purely abstract. Information, in this case numbers, become concrete and speculative. The viewer’s interpretation, the meaning, the attachment and the feeling related to the experience can have multiple facets based on who is observing, but also factors like the environment where the installation is located and people’s presence around it can change influence drastically.”

 Zamagni’s prior video work, most notably, Mass Production, illustrates some of the themes in Nature Abstraction. Here, Zamagni takes found footage and “destroys” its integrity not only through filmic cutting and suturing, but also via analog and digital processes including feedback loops. While a woman intones “simulation, orientation, immersions, tensions” to a rhythmic synth beat, Zamagni cuts quickly, often along with the beat of the music, between scenes of mass movement, mass transit and mass production.

Unlike Bruce Conners’ music videos which also privilege found footage and editing over content, Mass Production contains no vestiges of conventional narrative. Rather, meaning making occurs via the processes of immersion, tension, orientation and simulation. The film is not about anything except perhaps the effects of information overload on the human nervous system and memory which the video seeks to activate. As Walter Benjamin notes, “the technique of reproduction detaches the reproduced object from the domain of tradition. By making many reproductions it substitutes a plurality of copies for a unique existence. And in permitting the reproduction to meet the beholder or listener in his own particular situation, reactivates the object reproduced.” In a similar way, the images in Mass Production act as cues for viewers, each of whom will experience the piece differently, not because the work itself is overtly “artistic,” but because each viewer brings to it a unique set of prior experiences.

In addition to creating more fractal worlds for Nature Abstraction, Zamagni plans to continue his exploration of human perception of reality and its relationship to our embodied existence with a new immersive VR experience. Although his materials consist of computers, software, and graphic image processing techniques, the description he offers of this project speaks to the revelatory potential of art in general.  

“I am researching ways to ‘hack senses’ and trick the brain to produce out of body inductions . The piece is related to my quest into subjective experience and consciousness— how we construct our own realities as we perceive them, not the other way around. The outside wold is only an illusion as there is no physical boundary where everything is in continuous movement, a flow. The way we perceive is only one in a million. By giving people different perspectives or ways of perceiving through my art, I want to challenge those notions of subjective realities and the existence of consciousness.” 

Matteo Zamagni

Matteo Zamagni would like to thank the following people for their contribution to Nature Abstraction. 

Sound design : Daniel Ben hur
Technical assistance DeepDream : David li
And a big thanks to the amazing fractal community out there.
Without their suggestions and contributions it would had not been possible.