DCS861D: The User Interface fromFront to Back 

Art, Science, and Representation

Lecture 1

Description

Computer mediated experiences may be found on the Internet, in theme parks, museums, hospitals, scientific research laboratories, corporate conference rooms, military training facilities, and contemporary art installations. At these interfaces, humans and computers communicate through sensations of space, light, touch, and sound. It is the purpose of this course to present current approaches for merging real space with cyberspace. It will survey applications in the arts and sciences, and discuss the interface technology, application software and development tools, and computing infrastructure required to support such environments.

Resources
Readings from textbooks, conference proceedings, and journals in computer graphics, computer art, virtual reality, augmented reality, human-computer interaction, and collaborative computing.

Textbooks:
C. Dodsworth (Editor). Digital Illusion: Entertaining the Future with High Technology. ACM Press, Siggraph Series, 1998. ISBN 0-201-84780-9

S. Wilson. Information Arts: Intersections of Art, Science, and Technology. MIT Press, 2002.
ISBN 0-262-23209-X

P. Anders. Envisioning Cyberspace: Designing 3D Electronic Spaces. McGraw-Hill, 1999.
ISBN 0-07-001632-1

General Outline:

  1. Art, Science, and Representation
  2. Immersive Reality
  3. Augmented Reality
  4. Collaboration
  5. Human-Computer Convergence

 

Art, Science, and Representation

Modeling Form and Process

  • A model is something that mimics the relevant features of an entity or situation.
    • Toy airplanes, trains and baby dolls are all valid models of the real world.
    • Road or geological maps are models.
  • Expected - one-to-one correspondence between reality and model.
    1. Can embody an abstraction hierarchy or even metaphorical representations

History of Modeling

  • Ancient civilizations - Egyptians and Greeks.
  • Most likely predates ancient civilizations by at least 20 millennia.


(Paleolithic cave paintings from southern France and northern Spain.)


(Lascaux, France)

Edward Wachtel –

  • Why did Paleolithic artists paint and etch images of animals on the walls deep in the bowels of darkened, dangerous, inhospitable and least accessible caves, when they could have painted under natural light either at cave entrances or on other media such as bark, wood, animal skin and stone?
  • Why would they consistently superimpose one creature upon another, when there was an abundance of blank wall space elsewhere?

Wachtel’s Answer - Artists wished to include time and motion into their creations.

  • La Mouthe, France.
    • Three foot high image etched and painted in red, brown and black, superimposed with long uneven sometimes overlapping, vertical lines known as spaghetti engravings.
    • When a gas lantern was slowly swung back and forth, left to right, a few feet from the cave wall, the colors of the tectiform began to shift from black to brown to red.
    • The engraved lines became animated, allowing the head of one animal to stand out clearly, then fade, to be replaced by another.
    • Spaghetti lines became a forest or dense brush that concealed or revealed the animals within.
  • Font-de-Gaume, France.
    • Main hall - space that can be lit by the placement of lamps at three locations.
    • First light - hind appears in black and red.
    • Second light - hind is replaced with a black bison.
    • Third light - bison transformed into a mammoth.
  • Pair-non-pair, France - Caves animals are painted or etched with additional parts.

Ibex with two heads

Mammoth with multiple trunks.

Visual effects may explain how our hunter ancestors used the caves.

  • Wild animals well camouflaged - often invisible.
  • Discovering quarry based on surprise with quick response essential.
  • Caves used for ritual hunts or initiation rites for young hunters.
  • Youths required to respond in an appropriate fashion.
  • Ability to recognize and react quickly to the wild game could be tested and rewarded.
  • Cave walls upgraded when new hunting scenarios were encountered.

Were caves the first recorded instance of a simulator?

  • A simulator is an environment for conducting experiments with a model of an entity or system for the purpose of understanding the entity or evaluating various strategies for operation of the system.
    • e.g. flight simulator
      • Mathematical/physical model of an airplane and its environment built within a computer.
      • Pilot must acts appropriately to take-off, fly and land the plane.
      • Correct activity (landing not crashing) rewards and reinforces values required of pilot.

Geometry and Representation

  • Geometry used as foundation of visual representation.
  • First Use - ancient Egyptians.
  • Geometry means "measure of earth,"
  • Used to reestablish social order to an environment hurled into social chaos when the Nile river yearly overflowed its banks eradicating land boundaries.
  • Geometry is the study of spatial order through the measure and relationship of form.
  • Plato considered geometry to be a philosophical language – used for reasoning.
  • Geometry rediscovered by Western Europe during Crusades.
  • Euclid’s Elements translated from Greek and Arabic to Latin in late 12th Century.
  • Euclid’s Elements Printed in Venice in 1482.

Fra Luca Pacioli - by artist Jacopo dé Barbari.

  • Fra Luca Pacioli (1445-1517), Franciscan monk, teacher of mathematics and theory.
  • Student Duke Guidobaldo da Montefeltro stand before a table with the tools of Renaissance mathematician's profession: chalk and erasing sponge, goinometric contact gauge, a pair of dividers and a copy of Euclid's Elements open alongside a small drawing slate on which a problem from book XIII of the treatise has been drawn.
  • Small dodecahedron fashioned from wood as well as the rhombicuboctahedron constructed out of glass.
  • Pacioli was the driving spirit behind the construction of regular three dimensional polyhedra and their derivatives. These were used for teaching geometry and the theory of proportions.

Painting’s significance:

  • Represents the ongoing dialogue between art and mathematics
  • Pacioli's used the contemporary visual arts for popularization of mathematical science.
  • Close association with Leonardo da Vinci.
  • Leonardo's devised over sixty illustrations of geometrical solids for Pacioli’s book De divina proportione published in 1509.
  • This book distibuted throughout Europe disseminated Fra Luca's beliefs in Euclid's laws as a foundation for understanding the laws of the Universe as well as the artistic expression of Nature's forms.
  • His book and others written by artists such as Albrect Dürer effected a change in visual thinking eventually influencing the education of Galileo Galilei, William Harvey, René Descartes, Johannes Kepler and Issac Newton.

Reorientating visual thought before Pacioli.

  • Onset of the fifteenth century, artists had already become facile observers of nature.
  • Early quattrocento artists (e.g. Masaccio in Italy, van Eyke in the Netherlands) had accumulated a growing set of prescriptions for rendering spatial relationships and portraying light and shadow.
  • Artists of the fourteen hundreds - true art meant imitation of nature.
  • Imitation - the creation of figures that appeared to breathe in virtual spaces sensed as real.
  • Artists required to understand nature through its abstraction to the Platonic geometric ideals.
  • Required a rationalized theory of light, space, and form founded on mathematics.
  • Foundation: confluence of two streams of Renaissance thought about nature:
    • Theoretical, philosophical tradition following the ancient Greeks such as Aristotle, Plato, Euclid, and Ptolmey;
    • Pragmatic, experimental tradition carried forth by the artisans which included artists and engineers.

15th Century Florence:

Fragmented visual space of Medieval art transformed into the single plane viewer centric visual space that dominated Western art for the next five centuries until cubism in the first quarter of the twentieth century.

  • (Re)invention of visual perspective - Fillippo Brunelleschi, artist and architectural engineer (1377-1446).
  • Appled optical theory to painting - the artist's/viewer's eye position defined the image's center point and horizon line.
  • Brunelleschi's friend Masaccio exploited these procedures in his work on the Trinity fresco in 1426, the first perspective demonstration directly influenced by Brunelleschi.

  • Leon Battista Alberti, a lawyer, natural philosopher, artist and architect.
  • Educated in Euclid's, Ptolmey's, and Alhazen's Optics as well as Euclid's geometry.
  • Endowed Brunelleschi's empirical methods with a firm mathematical underpinning.
  • Seminal treatise On Painting (De Pictura, 1435) he proclaimed Brunelleschi's work to be a scientific principle of art and recast it in the language of Medieval optics.
  • Perspective became a system for recording the intersections of light rays on a plane as they proceeded in a pyramidal pattern from object to eye.

  • Woodcut by Albrect Dürer from Underweysung der messung (1525), his treatise for drawing polygons in perspective.
    • Used to teach artists to think of their pictures as maps, which nature represented as filling the fixed coordinates of the cartographic grid.
  • Grid became the frame-of-reference for rationalized observation and systematic decomposition of natural objects into abstract regular and irregular polygons.
  • Naturalness becomes endowed with a scientific foundation.
  • Any painting becomes a rational image based on the universal laws of geometry, painted by an individual who became, in effect, the first modern scientist.

Impact of geometry in Renaissance science and art:

  • Geometry became the requisite language for recording and interpreting nature.
  • Artificial perspective methods came under geometric control.

The Flagellation of Christ - Piero della Francesca (c. 1415-1492).

  • Artist and mathematician
  • Produced three treatises encompassing arithmetic, algebra, and geometry.
  • Built on the work of Alberti and covered the regular polyhedra and perspective.
  • Fra Luca Pacioli's teacher.

The Flagellation

  • Strong architectural perspective.
  • All polygons are rendered in correct geometric perspective.
  • Vanishing point found below and to the right of the tormentor at Christ's left hand.
  • Vertical columns and convergent lines set up two spaces.
  • Where the torture occurs
  • Occupied by three spectators.
  • Christ and the statue surmounting the Doric column behind Him are correctly illuminated from a point light source hidden by one of the palace colonnades.

Strong architectural perspective used to create large spaces.

  • The Feast in the House of Levi by Veronese - is 5.5 meters high by 12.8 meters wide

Soft architectural perspective subjugates space to content.

  • Madonna of Saint George by Correggio.

Fusing architecture with a painting’s virtual space.

  • Dome paintings of Mantenga, Correggio.
  • Ceilings of the Baroque artists Giantonio Fumiani (Church of San Pantalon in Venice (c. 1704)) and Fra Andrea Pozzo (Church of Sant 'Ignanazio in Rome).

Andrea Mantegna

·  Ceiling Oculus, 1471-74

·  Fresco, diameter: 270 cm

·  Camera degli Sposi, Ducal Palace, Mantua

  • Covers the ceiling of the Church of Sant 'Ignazio in Rome.
  • Designed to continue the real architecture of the hall with the hemicylindrical ceiling
  • Viewer stands on a disk of yellow marble at the center of the church.

Contemporary Art – Thinking out of Box

Rachael Whiteread – House (1993).

  • What it would be like to be inside the walls of an house looking into the space of each room.
  • Used house that was scheduled for demolition.
  • Cast entire house in concrete.
  • House’s walls used as forms upon which to pour or spray concrete.

Response to House

  • Immediate and passionate response to Whiteread’s sculpture.
  • Some thought House to be a monument to homelessness.
  • Since the original house was built in the nineteenth century, some saw House to be a sign of nostalgia.
  • Many feminists viewed House as a sign of conflict within the home.
  • Others treated House as if it were real.
  • Milk bottles were left at its door.
  • One man addressed letters to this house that was no longer a house.
  • Artistic, social, and political communities clashed over the merits of Whiteread’s creation.

In the end:

  • House was demolished.
  • Whiteread was awarded the K Foundation Prize for worst artist in Britain.
  • Awarded the prestigious Turner Prize for best artist in Britain.

Maya Lin – Vietnam Veterans Memorial (1982)

  • "I wanted the names in chronological order because to hone the living as well as the dead it had to be a sequence in time."
  • "I though about what death is, what a loss is. A sharp pain that lessens with time, but can never quite heal over. A scar. The idea occurred to me there on the site. Take a knife and cut open the earth, and with time the grass would heal it. As if you cut open the rock and polished it."

Kurt Vonnegut - Diagram of the story of Cinderella

  • Forward to Transformations by Anne Sexton


… I drew for her this diagram of the story of Cinderella:

"G" was good fortune. "I" was ill fortune. "B" was beginning. "E" was end. Cinderella was low at the start. She sank even lower when her rotten stepsisters went to the party and she stayed home.

Then her fairy godmother appeared, gave her a dress and glass slippers and a carriage and all that. The steps in my chart represented those donations of valuable merchandise. Cinderella went to the party, danced with the prince. She crashed at midnight, but she wasn't as low as she used to be - because she remembered the party.

Then the glass slippers fit her, and she married the prince. She became infinitely happy forever - which includes now.

Jim Mason – G7 Stock Puppets

http://www.stockpuppets.com

  • The G-7 Stock Puppets are an Internet-driven kinetic installation that tracks the movements of global stock markets with seven larger-than-life marionette puppets.
  • From the opening to the closing bell, the puppets continue to rise and fall in serendipitous synchronicity with the "arbitrary" movements of the G-7 market indices.
  • Puppets are made from 8 foot tall fiberglass mannequins, dressed in gray-pinstriped double-breasted suits, accessorized with standard-issue red power ties.
  • Large VGA monitors for heads stream stock ticker symbols and index numbers for the individual markets.
  • Each puppet movement cues a face animation of the appropriate finance minister on the monitor head - the face morphing and contorting in relation to the direction of market movement.
  • The puppets are elevated and articulated by cables extending down from pulleys supported 25' overheard on seven individual puppet towers.
  • Surrounding this mechanized market ballet is an ambient soundscape of shouted buy and sell orders, bells and gavels, racing heartbeats and terrified curses.

musicBottles - Tangible Media Group, MIT

  • Deploys bottles s containers and controls for digital information.
  • The system:
    • Specially designed table and three corked bottles that "contain" the sounds of the violin, the cello and the piano in Edouard Lalo's Piano Trio in C Minor, Op. 7.
    • Custom-designed electromagnetic tags embedded in the bottles enable each one to be wirelessly identified.
  • Bottle opening and closing is also detected.
  • When a bottle is placed onto the stage area of the table and the cork is removed, the corresponding instrument becomes audible.
  • A pattern of colored light is rear-projected onto the table's translucent surface to reflect changes in pitch and volume.
  • The interface allows users to structure the experience of the musical composition by physically manipulating the different sound tracks.

Visiphone – Sociable Media Group, MIT

  • Opens a graphical as well as an audio portal through space.
  • Designed to provide a continuous, ubiquitous connection between people in different places.
  • Displays two parallel visualizations:
    • one derived from the local sound reaching the device (input audio)
    • the other from the sound emanating from it (output audio).
  • VisiPhone displays both the real-time audio and a brief history of the conversation.
  • Graphics begin at the top of the dome slowly traveling to the bottom of the dome through a rotating spiral path. The colors of the history graphics fade out as they approach the bottom part of the dome, enhancing the notion of evolving time.


 
 

Text Rain – Camille Utterback, NYU

  • Participants in the Text Rain installation use the familiar instrument of their bodies - to lift and play with falling letters that do not really exist.
  • Like rain or snow, the text appears to land on participants' heads and arms.
  • The text responds to the participants' motions and can be caught, lifted, and then let fall again.
  • The falling text will 'land' on anything darker than a certain threshold, and 'fall' whenever that obstacle is removed.
  • Falling letters are not random, but lines of a poem about bodies and language.

Eunmi Yang – Poem Vacuum Cleaner, Seoul

  • Reading poems purifies feelings.
  • If a viewer vacuums the poetic word on the floor, the word will be replaced with a flower.
  • This work says that the role of a poem is like that of a flower in a citylife.


 

Kage – Motoshi Chikamori, Kyoko Kunoh

  • Since the earliest times, the shadow has proved existence.
  • In KAGE, computer-graphics shadows of cone-shaped objects explore this shadow-substance characteristic.
  • The computerized shadows projected toward the floor are motionless, like all shadows, but as time passes, some of them begin to tremble.
  • When the objects are touched, various kinds of patterns appear on the computerized shadow images.
  • The ceiling-mounted projector also illuminates the viewers, so their shadows join the shadows of the objects on the floor.
  • When the false shadows created by computer graphics and the viewers' true shadows are both projected on the same plane, viewers recognize the shadow-existence dilemma.

Wooden Mirror Danial Rozin, NYU

  • The wooden mirror has 830 pieces of wood each about 40 mm square arranged into an octagon of 35 x 29.
  • Each piece of wood is connected to a servo motor that can tilt it about 30º up and down.
  • The whole piece is lit from above with a few spot lights, in a way that when the pieces are tilted upwards they become brighter, and when they tilt downwards they become darker.
  • There are perhaps just 10 levels of gray distinguishable enough in this range. Each 8 servo motors are connected to a serial servo controller which is a small circuit that takes serial commands and controls the motors.
  • In the center of the Mirror a tiny video camera is concealed.
  • It sends the viewer's picture to the computer via the AV port.
  • The software on the computer (Macintosh 8600 AV ) is a combination of custom software written in C and Macromedia Director. The software digitizes the video and reduces its size to 35 X 29 pixels . It then converts the color information into gray levels and adjusts the brightness of each pixel to compensate for the various shades of the wood pieces.
  • The software then compares the state of the wooden mirror to the required image and sends commands to the mirror to change the position of only those pieces that need to be moved.
  • All phases of design fabrication and programming were done by Daniel Rozin. All the construction of this piece was done by hand including all the mechanical connections and wiring. It took 10 months to build the mirror.