Consumption and the machine

Appropriation in the Age of AI

BY BEN BOGART

AS AN ARTIST drawn to generative processes, I am not concerned with compositions themselves but with constructing systems that generate compositions; this approach allows space for indeterminacy and surprise. My attraction to systems leads my work to an emphasis on autonomy and machines that execute systems of rules (computers and code). Computers are empty vessels whose meaning is entirely determined by the agents that program and use them. Given the formalization of code and the flexibility of meaning, computers are invaluable as tools we can use to understand ourselves. Artificial intelligence is both a discipline of understanding how minds and intelligence may work, and the application of this knowledge to offload to machines certain tasks normally completed by humans. AI is thus an ideal site to examine how we conceive of minds, brains, and ourselves. Are we conscious agents with free will, or are we complex algorithmic machines trapped in the illusion that consciousness precedes action?

A central theme of my work is the relation between humanity (in particular brains and mental processes) and computational systems that we increasingly use to model and extend our intellectual capacities. I am interested in an examination of the technical and cultural implications of seeing ourselves through the lens of computation as we increasingly reallocate work to machines.

A key component of AI, machine learning is concerned with automating the construction of statistical models. Models are built by presenting an algorithm with a corpus of data (collections of measurable properties) referred to as training data. The algorithm ‘learns’ incrementally by creating an abstraction (the model) that represents statistical properties of the training data. I’ve recently been thinking about this process of modelling as a process of consumption or decomposition. That which is to be modelled is reduced to a series of measurements, breaking the world into simple components. These components are the raw material consumed by the machine; quantitative meaning results from the processing, filtering, recombination, and restructuring of that material. The model is built from the bottom up, such that its structure is dependent on both the raw material and the ‘learning’ process.

The core of my artistic inquiry into machine learning is a consideration of the relation between the training data used to construct these models and the structure of the models themselves. In this text I will focus on recent and in-progress bodies of work that embody the theme of consumption. In some of this work, such as the Watchers series discussed below, the concept of consumption is expanded to include the machine’s act of pop-culture appropriation. Influential science fiction films are broken down into fragments that serve as raw material used to reconstruct a resemblance of the original. The machine’s reconstruction holds statistical validity, but in no way could be mistaken for the film itself. The tuning of the parameters and the methods of deconstruction distance the machine’s model from the original. Reconstructions are abstractions that reflect the biases of the algorithm (as well as my own biases embedded in the software) regarding what aspects of the original are most important and should be preserved. This text will also introduce a new body of work currently in development titled Machines of the Present Consume the Imagination of the Past. This work is situated in painting history. Constituent works point to a near future in which AI could enable the automatic generation of artworks following the whims and fads of the market.

Subjective Machines

While I consider the primary material of my work computational processes, I have also come to accept myself as an image-maker; my work emphasizes the visual. Images manifest thought; they betray those mechanisms that allow us to generate internal and external representations of the world. Representational images allow us to reflect not only on how we attend to the world, but also how we categorize and conceptualize every unique moment of embodied life.

Just as my inquiry into machine learning is concerned with the model in relation to the data fed to it, my interest in representational images is concerned with the image as imprint of thought in relation to the external world it depicts. At the root of my practice is an epistemological position where subjects and objects are considered mutually constructive. As subjects, we read into the world and ignore variation to focus on the abstract and quintessential aspects of objects; these aspects are as much a function of our imagination as they are of the world independent of us.

We unconsciously build mental models of the world that mirror our constructed culture and facilitate sensory perception. We require such models to resist the constant barrage of sensation, less we recoil into the abyss of constant flux, randomness, and noise of the world as independent of our perception of it. This tension between subjects and objects is the very core of our nature. We believe we are in control, but the independent world always creeps into our minds, throwing off our predictions and subverting our expectations.

My use of machine learning and AI is an inquiry into this power struggle between subjects and objects. I build Machine Subjects that manifest foundational processes that carve imaginary boundaries into the underlying continuity of the world. My machines deconstruct, categorize, organize, and reduce the infinite complexity of the world as independent of thought. Machine Subjects participate in a process of abstraction that breaks sensed structures into atomic particles that serve as the raw material from which new structures are built. The resulting ‘mental’ images are of the world—their mechanisms uncover underlying statistical truths about the world, but they are also of us—they are projections of bounded subjective understanding.

In my appropriation works, machines learn from cultural artifacts rather than sensory information. In deconstructing, categorizing, predicting, and reconstructing cultural artifacts, I emphasize the tension between subjects and objects. The machine is both an alien subject attempting to understand our culture and a cultural object that manifests our understanding of ourselves.

Machines Watching Science Fiction

The Watchers is a series of media artworks contextualized within a larger body of work, titled Watching and Dreaming, that was initiated in 2014. Constituent works have been shown at the 2016 Digital Art Biennial in Montréal, Canada, Transmediale 2017 in Berlin, Germany, and at the Surrey Art Gallery in Surrey, Canada. The body of work extends my doctoral research at the intersection of cognitive neuroscience, computer science, and generative art. My research investigated the cognitive and neurological mechanisms that allow biological brains to build internal representations of the world. Dreams, mental images, mind wandering, and even sensory perception are enabled by systems of imagination that construct the real by building models from patterns learned from experience. Integral to subjectivity, imagination is a process that imposes structure (for example perceptual categories) on sensory information. A central argument manifested in these dreaming and watching machines is that both dreaming and watching are highly constructive and depend on shared mechanisms of imagination.

The Watchers generate perceptual images by deconstructing, learning, and reconstructing popular cinematic depictions of AI. Films in the series are chosen for their pop-culture influence, their release during a significant age of computer science research, and the diversity and richness with which they depict the artificially ‘intelligent’ and personified technologies. Such science fiction films serve as cultural depictions through which we can understand ourselves through the lens of the other. Recent pieces in the Watching series appropriate Stanley Kubrick’s 2001: A Space Odyssey (1968), Ridley Scott’s Blade Runner (1982), and Steven Lisberger’s TRON (1982).

‘Mental’ images are constructed by the machine using the same general process that operates on frames in the original source film such that the machine’s subjective process:

1. Breaks the frame into image fragments so that each fragment is of generally the same color value (mean shift segmentation). Depending on the film, this results in approximately 30–47 million image fragments per film.

2. Extracts average color, width, and height features from each fragment.

3. Groups fragments according to the similarity of their features (k-means clustering) to generate percepts.

4. Constructs visual percepts by centering, stacking, and averaging the constituent image fragments in each group. This results in the generation of tens to hundreds of thousands of percepts that serve as the visual (and ‘mental’) vocabulary for each work.

5. Reconstructs each frame in the original by replacing each extracted fragment with its most similar percept. This is the system’s recognition of the fragments contained in each frame. By considering each fragment (training sample) in relation to the visual vocabulary (model), the machine is recognizing each frame as a collection of instances of perceptual categories, i.e. objects of perception that persist over time.

The sounds in the original soundtrack also go through a similar process; the source sound is broken into fragments that are grouped (according to sonic features) and averaged to construct audio percepts. This sonic vocabulary is the raw material used in the reconstruction of the soundtrack.

The ephemeral nature and pacing of these moving-image works mean that the richness of their visual vocabulary cannot be sufficiently appreciated during normal viewing. I thus created a subseries of works, titled Percepts from Watching, that directly present a subset of the visual vocabulary. These works correspond to each of the three audio-visual works and, shown in forty-nine-inch square light boxes, they allow viewers the opportunity for close readings of this vocabulary. By showing these luminous collages in the same space as video projections, the installation facilitates dialogue between moving and still images. The location of percepts in these collages is determined by the similarity of their color, width, and height such that similar percepts tend to be located nearby in the composition.

Watching the Watchers

Upon entering a darkened room with black walls, the viewer is confronted by three forty-nine-inch tall, highly saturated and high-contrast projections. Each projection has a different character, pacing, and color palette but they are all composed of soft ephemeral shapes that blend together and are punctuated by hard edges. Hanging nearby is a set of three forty-nine-inch square luminous images that appear to hang in space. What begin as chaotically moving abstractions become surprisingly referential over time; the naturalistic movement of the camera and figures coalesce and becomes briefly readable before sliding back into chaos. (Videos: https://vimeo.com/187856950, https://vimeo.com/262430601).)

Viewers familiar with the sources may recognize scenes from 2001: A Space Odyssey, Blade Runner, and TRON. The viewer sits down on seating reminiscent of the 1980s and watches these abstract forms appear, oscillate, and disappear from the image plane.

Sound fills the room and is composed of short but somewhat understandable fragments. Some words are clear, while others are muffled, muted, or fragmented. Dialogue, music, and sound effects all blend together into a diverse shifting and at times rhythmically halting soundtrack. The images and sounds are just at the threshold of readability such that a person familiar with the source films may follow the plot. The viewer constantly attempts to read the work as cinema and searches for hints of narrative and intelligible dialogue. At some point this search becomes overwhelming as the images and sounds ebb and flow between perceptual readability and the chaotic abstraction of dreams; the viewer eventually tires of their cinematic reading; the sounds and images overtake them as their gaze softens and they become entranced. The viewer loses awareness of time as they settle deeply into the comfortable seating and their mind begins to wander, reflecting on what it could mean for machines to watch or dream.

Once they snap back into self-conscious awareness, the viewer stands up to examine the still and luminous images; the viewer realizes that the palette and quality of each of the three collages are shared with each of the three projections. Standing in front of the light boxes, the viewer is struck by the immensity of their structure, composed as it is of thousands of miniscule color regions forming a complex texture.

Up close, it’s clear these collages are constructed from irregularly shaped fragments filling a rough frame floating in empty black space. Some fragments are subtle gradients of color while others contain surprising detail and appear photographic. As the viewer scans over each collage, they may pick out forms, props, and parts of familiar characters and sets from the source films. Stepping away, the viewer’s eye is lost in organic structures where patches of color overlap each other and form macro structures that span large areas of the image.

These ‘mental’ images manifest the machine’s alien recognition of the source material. The subjectivity of the machine differs significantly from ours. We easily link shapes moving and transforming through time as unified objects; to these machines, changes in size and shape over time in a 2D plane are irreconcilable with their understanding. A change of lighting or an occlusion challenges the machines’ sense of what an object is; they naively search for color and shape regularity in the image plane. They do not have a concept of a person, a prop, an animate or inanimate object, or even 3D space. This fragility of their perceptual systems results in both the ephemeral quality of percepts and the temporal instability of the sequence of images. What we perceive as an object changing over time is understood by the machines as objects popping in and out of existence from frame to frame.

Despite this instability, the human viewer can often find stability in the chaos and read these abstractions as naturalistic movements of the camera and figures. While our perceptual systems are much more advanced, our visual (and auditory) experience is also a construction that depicts the world, not a copy of it. Models that encapsulate our understanding of objects and their behavior are brought to bear in our perceptual processes; our models are constructive and abstract, just as the machines are. The ‘mental’ image is a manifestation of the machines’ subjectivity; it is generated by a model whose structure results from the algorithm’s interaction with a world that is independent of thought and subjective imagination.

Zombie Formalist

Thanks to the support of the Canada Council for the Arts, I am (at the time of writing) embarking on a new body of work that diverges from what has been discussed above. While still concerned with subject and object relations and making use of machine learning methods, this body of work moves away from cognitive processes embedded in previous works. This proposed body of work is instead situated firmly in painting history and intended to bridge media and visual arts.

The market proliferation of new artworks rehashing the 1960s avant-garde, in particular post-painterly abstraction and color field painting, has been dubbed “zombie formalism.” Such works do well in the market partially because they are unchallenging and look good in contemporary living rooms. The Zombie Formalist, a key piece in the series Machines of the Present Consume the Imaginations of the Past, veils a critique of consumerist culture in what appears to be a formalist work. The piece is manifest in an interactive light box that generates abstract images following the style of painters such as Gene David, Barnett Newman, Kenneth Noland, and Karl Benjamin. The light box is envisioned to be elegant, wall-hung, and encloses a square twenty-six-inch flat-panel display, a small computer, and a small camera that faces back at the viewer. The hardware will be high quality and aesthetically minimal to emphasize an image that appears to be a still backlit print in a contemporary art frame.

I will produce two Zombie Formalists that exploit face recognition and construct new generative compositions when no one is looking at them, giving the illusion of static images. I envision the exhibition of a diptych that contrasts compositions learned from different audiences and/or value metrics. One light box could determine ‘value’ from the duration of viewing in person while the other could calculate ‘value’ from the number of social media “likes” given to uploaded compositions. The system will use machine learning to determine what aesthetics are valued by the audience(s) and which are not. Over time, the Zombie Formalist will be able to create new images in the style of previous images that were highly ‘valued’ by a particular audience. These light boxes are the ultimate Zombie Formalists in that they constantly generate formal compositions in response to the whims and fads of audiences.

While the light boxes embody themes of commodity and populism, I will also produce a series of prints and moving images generated by machine interpretations of the canon of Western painting. These still and moving images emerge from interactions between machine imagination and the underlying statistical properties of the training data. Using the same learning algorithm used in Percepts from Watching, source paintings will be deconstructed pixel by pixel where the similarity of color values determine the final emergent composition.

Paintings will be selected to form a historical arc from the emphasis on realism during the Northern European renaissance, to the surrealist and cubist problematizations of realism that manifest the tension between realism and abstraction. The moving images will show the subjective decompositional process from the original to an abstract form. Still images will show the abstract ‘mental’ image and will be printed and mounted for exhibition while the moving images will be shown on wall-hung displays.

Kitsch as Commodity

In addition to the still and moving images, and light boxes, Machines of the Present Consume the Imaginations of the Past will also include a selection of small print on-demand sculptural objects. These kitsch objects (mugs, phone cases, etc.) center the installation on themes of commodity and consumerism. Images from the body of work will be printed on these kitsch objects to reference the gallery gift shop and the role of artistic images as commodities to be consumed.

Beyond the promotion of artists and artworks through social media networks such as Pinterest and Instagram, there is also a growing trend for artworks to be delivered not as objects, but through purely digital means such as Sedition (seditionart.com) and Meural (meural.com). One of the major themes of this body of work is the tension between the traditional gallery context and these emerging digital dissemination platforms. While these platforms may indicate a degree of acceptance of contemporary art presented by digital means, mainstream capitalism is in parallel incorporating technology to better target ads. As machine learning improves in accuracy and becomes more easily embedded in systems we confront daily, we are likely to see more billboards using cameras to target individuals in public. They may be located in indoor spaces and estimate age and gender to present ‘appropriate’ ads, or may be large-scale billboards and recognize the make and model of cars driving by. Subjective Machines are appearing in our day-to-day lives and seamlessly hooking into big data collected on social networks to profile and predict our desires and future consumption. Why would Facebook not provide face metrics to advertisers as preferential looking may prove even more valuable than “likes”? How far away are billboards that recognize us as individuals and provide targeted ads based on our Facebook profiles or Google searches?

Machines of the Present Consume the Imaginations of the Past seeks to highlight this likely future. The project draws a link between the ‘smart’ billboard that sells what the marketer thinks you want, social media that models you in order to show you what you want to see, and the current art market. The Zombie Formalist models the aesthetic values of the audience and shows them more of what they like. Perhaps the most interesting art object for the commodity-oriented market is an artwork that automatically follows the trends and always stays ‘current’ and ‘fresh’ for its audience. The criticism of the “filter bubble” can also be set upon this scenario where products and cultural artifacts are never expected to be challenging. The Zombie Formalist produces art that always looks good in the living room and never offends.

This work points to a future where AI automatically generate products that suit the short term and fickle desires of the market. A sense of cultural value or tradition is replaced by the notion of salability. In this future (our present?), consumption becomes a compulsion that fulfills our need to be endlessly distracted and keeps us invested in the capitalist illusion of constant growth.

Ben Bogart is an interdisciplinary artist working with generative computational processes and has been inspired by knowledge in the natural sciences in the service of an epistemological inquiry. He has shown processes, artifacts, texts, images and performances internationally. Ben holds both master’s and doctorate degrees from the School of Interactive Arts and Technology at Simon Fraser University and has been funded by the Canada Council for the Arts, the British Columbia Arts Council, and the Social Science and Research Council of Canada.