Parsing the Languages of the New Media

The First Principle of New Media identified in The Language of New Media is that the resulting objects of new media practices “are composed of digital code; they are numerical representations.”

When we look at a picture of the Mona Lisa on a computer screen, we do not consciously perceive digital code or numerical representations; there is an implicit phenomenological distinction being drawn between two descriptive categories: the operation of the computer (how information behaves within a computer as a formal system), and the conscious perceptions of a person interacting with a new media object through a computer interface (we see “new media objects” rather than digital code or numerical representations).

While these two descriptive categories refer to the same physical behavior (what, in a given instance, a person might be doing with a computer), they also require different vocabularies to adequately describe precisely what physical processes are involved and how these processes affect us. In the case of a person doing something with a computer, these vocabularies differ in terms of “how” something might be happening, agree on “that” which might be happening, and converge on “why” certain events result.

The First Principle of New Media elects to analyze the resulting objects of new media practices in terms of how information behaves within a computer as a mechanistic formal system. The assumption here is that analyzing new media in such a way will lead to a more fundamental or profound account of new media, compared to a study of what people do with new media; the desire behind such an assumption is often a belief that a more fundamental account of a phenomenon might be understood as objectively grounding subsequent inferences about observations of relevant phenomena — in this case, that phenomenon being new media objects.

Contemporary computers are formal machines designed to provide a syntactical specification for a set of formal symbols, the rule-governed means to manipulate these symbols, and the physical means to express these symbols using arbitrarily distinct formal systems.

The information stored on a computer is stored in a symbolic form, and numbers are just one possible way to represent these symbols. A loose analogy can be drawn with natural languages: when we have something in mind that we would like to express, we may write it down on paper so that it is expressed as chemical pigments and light waves, or we may speak it aloud so that it is expressed as air molecules and sound waves. If we say something aloud, it would not be correct to assume that the spoken expression is more or less fundamental than its written equivalent; if we want to understand natural language communication, it would not be correct to privilege the spoken word over the written word, or to focus strictly on how words exist in minds before they are communicated to others.

When we talk about information stored in a computer, we frequently talk about binary code being composed of 1′s and 0′s; the fact that we use the Arabic numerals “1″ and “0″ to represent this code is simply a matter of convention. TRUE and FALSE or ON and OFF work just as well as 1 and 0; the convention we use has little effect on how the stored information is structured so long as the convention allows exactly two states in binary opposition.

It is certainly possible to treat all the 1′s and 0′s involved in storing a software application on a computer (along with all the plain-language text unique to that software application, such as online documentation) as a single integer, represented by a very large base-2 number. However, the same 1′s and 0′s can also be viewed equivalently as instructing the computer hardware to move bits of information from one place to another; or, those 1′s and 0′s can be viewed as truth-value assertions in a long expression of Boolean logic.

In his essay, “The Practical Logic of Computer Work,” Philip E. Agre asserts that:

“The design of a computer begins with formalization — an intensive and highly skilled type of work on language. A computer is, in an important sense, a machine that we build so that we can talk about it; it is successful if its operation can be narrated using a certain vocabulary. As the machine is set to running, we too easily overlook the complicated reflexive relationship between the designer and user on which the narratibility of the machine’s operation depends. The corpus of language that was transformed in producing the machine, like any discourse or text, is necessarily embedded in an institutional and historical context, and the machine itself must therefore be understood as being, in some sense, a text.”

To describe a computer program as a long expression in Boolean logic shouldn’t be counter-intuitive: computer programs are, after all, written in artifical “programming languages.”

This means that computers are just as much logic machines as they are arithmetic machines. That the information computers store can be represented numerically is very much incidental to how the stored information is structured, and is neither a fundamental nor an axiomatic observation about how computers behave.

Why we so often talk about computers as though they are arithmetic machines (or fancy calculators) is in large part a matter of what sorts of patterns we are culturally taught to be sensitive to. A modern city-dweller well-acquainted with the cycles of the business day and traffic laws might easily see patterns in a contemporary cityscape which a farmer from 1800 might have to struggle to perceive. We are taught a good deal of arithmetic in public schools, but very little formal logic; we are furthermore taught science in the tradition of a Platonic-Pythagorean conception of nature, which holds abstract constructs to be the most fundamental description of the world. Since we know science gave us computers, why wouldn’t arithmetic therefore appear to be essential to the most fundamental description of what computers do?

Given, however, that computers are just as much logic machines as arithmetic machines, it is not at all self-evident why the numerical features of computerized information storage should be privileged in an analysis of how practical computers behave.

Furthermore, it is not unique to new media objects that they originate in a numerical form: musical notation, too, can be viewed mathematically, as can the design of architecture (or the motion of planets for that matter). In fact, the information contents of practical computers are hardly unique in their ability to be described mathematically, and their ability to be described mathematically is hardly fundamental to their operation. It is therefore just as problematic to describe computers as arithmetic machines as it is to describe the arithmetic features of new media objects as characteristic of new media.

It is perhaps more relevant to discuss contemporary computers in terms of the relative convenience with which they allow us to perform sophisticated manipulations of symbolic information.

“The most likely reason modern media has discrete levels is because it emerged during the Industrial Revolution… Not surprisingly, modern media follows the logic of the factory.”

The argument here suggests that the way new media objects implement computer code is a product of the industrial mindset, with the implication that the values of industrial division of labor, specialization, and standardization led to the modern computer. This suggestion involves a complex set of interrelations between the thought processes introduced by industrialization, the structure of computers, and how these thought processes interact with the structure of computers when people create new media objects.

New media objects are conceived of as collections of discrete, indivisible units, such as pixels; and this conception presupposes a contradistinction to traditional media — such as sculpture or chemical photography — where surface properties vary with continuous and arbitrary degrees of detail.

The use of “discrete” here connotes precision, while “continuous” connotes imprecision: however accurately one attempts to measure the height of a bronze sculpture, for example, changes in temperature will cause the metal to expand or contract slightly on different days, contributing to an inherent imprecision in one’s measurement; a digital picture file, however, will always have the same number of pixels no matter on what day one decides to make a tally.

As it is a central feature of industrial mass production that one be able to manufacture large numbers of precisely identical objects, there are a number of superficial reasons why computers might seem to be the product of an industrial mindset: industrial fabrication techniques facilitated computers coming into widespread use, the individual components of computer hardware are in many respects both standardized in their construction and specialized in their function, and the binary code used by computers very much resembles an idealization of industrial order and production.

These congruences aside, however, the aforementioned argument as presented in The Language of New Media involves a number of substantial problems. Most obviously, the written alphabet is a system of discrete symbols: letters came into use long before industrialization, are just as indivisible as pixels in a digital image, and type set in a monospaced font falls into a grid not unlike the arrangement of pixels on a computer screen. Moreover, letters can be assigned numerical meanings: Hebrew is one example of an alphabet that does this.

There are also historical problems with attributing the discrete operations performed by computers to an industrial mindset. The history of computing machines reaches back to antiquity, and its early history can be found in such relics as the Antikythera mechanism. It could be argued that it was “the logic of the factory” that spawned the invention and design of digital computing machines, but it was, rather, a theological motivation that compelled Gottfried Leibniz in the late 1600′s to formalize the system of binary code used by today’s computers; Leibniz furthermore envisaged machines that would perform calculations using his binary system. Although it may be the case that industrialization substantially helped such computing machines in becoming a material reality, their conception lies very much apart from the industrial mindset.

While consumer use of computerized media might in many respects seem to follow “the logic of the factory” — especially as numerous commercial websites profit from user-generated content, which transforms the consumer into a type of specialized producer — the formal and material qualities of modern computerized media follow from a quite different logic.

The Second Principle of New Media describes what Lev Manovich identifies as “the fractal structure of new media.” This Principle holds that the resulting objects of new media practices have “the same structure on different scales.” Elaborating on this premise, Manovich observes:

“Media elements, be they images, sounds, shapes or behaviors, are represented as collections of discrete samples … These elements are assembled into larger-scale objects but continue to maintain their separate identities.”

Although there is an element of truth to this observation, the description is inaccurate in important ways: “fractal” typically refers to a type of self-similarity that manifests itself on different scales. While a digital image might be composed of discrete pixels just as a web page might be composed of several discrete JPEG images, an individual pixel resembles neither a web page nor an image file.

The main problem here isn’t with the description of new media objects as modular, but with what inferences might be drawn from Manovich’s particular description of modularity.

A more accurate description might substitute “interoperable” for “fractal.” Even so, while the parts of many industrial products — such as automobiles — are modular and designed for interoperability in a sense similar to that proposed by Manovich, so are parts of language in certain respects.

Moreover, to describe digital images as modular insofar as they are comprised of pixels is to omit an important distinction between how computers typically store visual information and how that visual information is displayed. The JPEG file format, which Manovich mentions in his discussion of new media’s modularity, does not explicitly store information about individual pixels. Rather, the JPEG format uses mathematical models to abstract visual information, then uses these models to generate pixels for display.

Pixels are representations of information, and computer programmers structure the information such that the representation can appear to users as an image. Any information that can be stored on a computer can be represented as pixels on a monitor. The way information is structured in a JPEG image is, however, quite distinct from the way information is structured in a block of text.

The information content of the above text can be interoperably represented as pixels with either of the following two images: