DCS861D: The User Interface from Front to Back 

Integration

Ubiquitous Computing

Ubiquitous computing, or calm technology, is the name for a discipline where computing technology is designed to be virtually invisible. Instead of desk-top or lap-top machines, the technology we use will be embedded in our environment.

Wearable Computer

    • Characteristics
      • Person's computer worn much as eyeglasses or clothing .
      • PC should interact with the user based on the context of the situation.
      • Heads-up displays, unobtrusive input devices, personal wireless local area networks, and other context sensing and communication tools make the wearable computer act as an intelligent assistant.
      • Wearables are always operational, tend to have sensors into their environement, and tend to have the ability to get information to the wearer even when the wearer doesn't expect it.
    • Operational modes
      • Constancy:
        • Computer runs continuously, and is always ready to interact with the user.
        • Unlike a hand-held device, laptop computer, or PDA, it does not need to be opened up and turned on prior to use.
      • Augmentation:
        • Traditional computing: computing is the primary task.
        • Wearable computing: computing is NOT the primary task.
        • Wearable computing assumption:
          • user will be doing something else at the same time as doing the computing.
          • computer should serve to augment the intellect, or augment the senses.
      • Mediation:
        • Unlike hand held devices, laptop computers, and PDAs, the wearable computer can encapsulate us.
        • Two aspects to this encapsulation:
          • Solitude: functions as an information filter
            • allows us to block out material we might not wish to experience
            • allows us to alter our perception of reality in a very mild sort of way.
          • Privacy: blocks or modifies information leaving our encapsulated space.
            • may serve as an intermediary for interacting with untrusted systems, such as third party digital anonymous cash ``cyberwallets''.
            • clothe our otherwise transparent movements in cyberspace.
            • can be used to create a new level of personal privacy because it can be made much more personal, e.g. so that it is always worn, except perhaps during showering, and therefore less likely to fall prey to covert attacks upon the hardware itself.
            • close synergy between the human and computers makes it harder to attack directly, e.g. as one might peek over a person's shoulder while they are typing, or hide a video camera in the ceiling above their keyboard.
            • can take the form of undergarments that are encapsulated in an outer covering or outerwear of fine conductive fabric to protect from an attacker  looking at radio frequency emissions.
            • able to make measurements of various physiological quantities
    • Six attributes (six signal paths) of wearable computing:

1.      UNMONOPOLIZING of the user's attention:

        • does not cut you off from the outside world like a virtual reality game.
        • built with the assumption that computing will be a secondary activity, rather than a primary focus of attention.
        • ideally will provide enhanced sensory capabilities.
        • It may mediate (augment, alter, or deliberately diminish) the sensory capabilities.

2.       UNRESTRICTIVE to the user: ambulatory, mobile, roving

        • can do other things while using it

3.       OBSERVABLE by the user:

        • can get your attention continuously if you want it to
        • medium is constantly perceptible by the wearer.

4.       CONTROLLABLE by the user:

        • Responsive
          • can grab control of it at any time.
        • Infinitely -- often -- controllable
          • constancy of user-interface results from almost-always observability
          • infinitely--often controllability in the sense that there is always a potential for manual override which need not be always exercised

5.       ATTENTIVE to the environment:

        • Environmentally aware, multimodal, multisensory.

6.      COMMUNICATIVE to others:

        • Can be used as a communications medium when you want it to.
        • Expressive:
          • allows wearer to be expressive through the medium
            • as a direct communications medium to others
            • as means of assisting the production of expressive media (artistic or otherwise).
    • Implied by the six properties is that it must also be:
      • CONSTANT: Always ready.
      • PERSONAL: Human and computer are inextricably intertwined.
      • PROSTHETIC: Can adapt to it so that it acts as a true extension of mind and body
      • ASSERTIVE: can have barrier to prohibition or to requests by others for removal
      • PRIVATE: others can't observe or control it unless you let them.
    • Fundamental issue of wearable computing - Personal empowerment
      • Through its ability to equip the individual with a personalized, customizable information space, owned, operated, and controlled by the wearer.
    • Aspects and affordances of wearable computing
      • Photographic memory: Perfect recall of previously collected information.
      • Shared memory: Two or more individuals may share in their collective consciousness, so that one may have a recall of information that one need not have experienced personally.
      • Connected collective humanistic intelligence: Two or more individuals may collaborate while one or more of them is doing another primary task.
      • Personal safety: Personal safety system is built into the architecture (clothing) of the individual.
      • Tetherless operation: Freedom from the need to be connected by wire to an electrical outlet or communications line.
      • Synergy: Produce a synergistic combination of human and machine, in which the human performs tasks that it is better at, while the computer performs tasks that it is better at.
      • Quality of life: Capable of enhancing day--to--day experiences, not just in the workplace, but in all facets of daily life.
    • How to Build One
    • Two Experiences at MIT
    • Future Computer Clothing - Beauty and the Bits - MIT Fashion Show
    • Wearable Computer Systems at Carnegie Mellon University
    • Musical Jacket

    • ifmachines
    • Smart Shirt - Georgia Tech
      • Georgia Tech Wearable Motherboard™ (Smart Shirt) for Combat Casualty Care
        • Uses optical fibers to detect bullet wounds
        • Uses special sensors and interconnects to monitor the body vital signs during combat conditions.
        • Systematic way of monitoring the vital signs of humans in an unobtrusive manner.
        • Sensors "plugged" into this motherboard and attached to any part of the individual being monitored, thereby creating a flexible wearable monitoring device.
        • Flexible data bus integrated into the structure transmits the information to monitoring devices such as an EKG Machine, a temperature recorder, a voice recorder, etc.


Bus transmits information to the sensors from external sources


and acting upon that environment through effectors.

      • Autonomous agents are computational systems that inhabit some complex dynamic environment, sense and act autonomously in this environment, and by doing so realize a set of goals or tasks for which they are designed.
    • Characteristics:
      • Autonomy - agents can operate on their own without the need for human guidance, even though this would sometimes be invaluable. Hence agents have individual internal states and goals, and they act in such a manner as to meet these goals on behalf of its user. A key element of their autonomy is their proactiveness, i.e. their ability to "take the initiative" rather than acting simply in response to their environment.
      • Cooperation -  In order to cooperate, agents need to possess a social ability, i.e. the ability to interact with other agents and possibly humans via some communication language.  It is possible for agents to coordinate their actions without cooperation.
      • Learning - For agent systems to be truly "smart," they have to learn as they react and/or interact with their external environment. Agents are disembodied bits of "intelligence." The learning may also take the form of increased performance over time.
    • Hive: Distributed Agents for Networking Things
      • Hive is a distributed agents platform, a decentralized system for building applications by networking local system resources.
      • Hive provides ad-hoc agent interaction, ontologies of agent capabilities, mobile agents, and a graphical interface to the distributed system.
      • Hive is designed to put computation and communication in everyday places such as your shoes, your kitchen, or your own body.
    • BOND. The Distributed Object Multi-Agent System
      • Java based distributed object system and agent framework
    • JATLite
      • (Java Agent Template, Lite) is a package of programs written in the Java language that allow users to quickly create new software "agents" that communicate robustly over the Internet.
      • JATLite provides a basic infrastructure in which agents register with an Agent Message Router facilitator using a name and password, connect/disconnect from the Internet, send and receive messages, transfer files, and invoke other programs or actions on the various computers where they are running.
      • Demo (IE4)
    • Resources :
  • Smart House
    • Around since mid-1980s
    • Technology used has ranged from simple timers that control household appliances at designated times during the day, to computer control of office environments.
    • Goal - hide computers in the environment while providing support for humans in their everyday activities
    • What's A Smart House System All About?
      • Scene Lighting
      • Voice Command Your Entire Home
      • Remote Access To Your Home Via PC or Phone
      • Remote Lighting/Appliance Control
      • One Stop Touch Screen Central Controller
      • Intelligent Timed Programmed Operation
      • Smart Appliance/Plant Equipment Control
      •  Smart Security/Intrusion Deterrent System
      • Smart Remote Video Surveillance
      • Smart Home Theatre Control
      • Intelligent Occupancy Sensing
      • Whole House Audio/Video Signal Distribution
      • Smart IR Signal Distribution
      • Smart HVAC
      • Smart Motorised Drapes
      • Smart Garden Sprinkle
    • IBM Home Director X10 Starter Kit
      • Control and program timed schedules for lights and appliances from PC
      • Macro capability can trigger a sequence of X10 commands.
      • One press can turn on a series of lights and appliances.
      • Programmed events and macros can be downloaded to the memory on the Home Director for stand alone operation (with the PC turned off).
      • Kit includes a 6-in-1 IR Remote Control that can control most commonly available audio/video components, as well as X10 via RF (using the included X10 RF  Receiver Base).
      • Can turn on TV, dim the lights and even close your drapes.


 
 

Context-Aware Computing

Example : Situated Interaction in Art Settings ( P. Marti, F. Gabrielli, L. Petroni and F. Pucci),
Personal and Ubiquitous Computing , Springer-Verlag

Setting: Sala del Mappamondo, Siena
Painting: Simone Martini. Maestà. 1317. Fresco. Palazzo Pubblico, Sala del Mappamondo, Siena, Italy.

Painting: Simone Martini. Equestrian portrait of Guidoriccio da Fogliano, 1328-30, Fresco, 340 x 968 cm, Palazzo Pubblico, Siena.

Observations:

  • The Maestà is better lightened by the natural daylight;
  • In the afternoon, the Guidoriccio is preferred since the artificial light brightens this fresco

Problem to solve:

  • Structure the interaction design of the two paintings taking into account contextual and situated needs and visiting strategies

General Understandings:

  • Sensing technology provides a way for including information about a situation and the context of its use.
  • Design of applications for these systems needs metaphors and interaction paradigms to fit user needs in new contexts of use.
    • e.g.  everyday leisure activities where the use of artifacts mediates the emotional engagement with physical and information spaces.
  • Cognitive artifacts (e.g. hand­held location aware tour guide) embed the situated visiting strategies and the space affordances in order to support the user involvement in the activity and with the physical environment.
  • Use context - unstructured environments such as amuseum or city.
  • Use situations - various and idiosyncratic - leading visitors to frequently adjust their goals and objectives during the visiting experience.

Museum Understandings:

  • Individuals generally do not anticipate alternative courses of action, or their consequences, until some courses of action are already under way.
  • Individuals often do not know ahead of time, or with any specificity, what future state they desire to bring about.
  • The visiting experience is a case in which individuals frequently have to "adjust" the way they interact with the environment, depending either on the action carried out or on the produced results.
  • The shift in goal might be produced mainly by two modalities:
    • i) the goal cannot be accomplished (lack of competence or physical constraints)
    • ii) different states of the world are suggested on the basis of the performed activity (incoming information activate alternative patterns of knowledge)
  • Ways in which individuals try to control interaction are contingent and derived from the situated action that they represent

Design Activities:

Three Main Issues

  • Context: the context in which the activities occur is composed of natural, material, social and cultural components that affect the course of user' actions and interpretations during the interaction (e.g. room lighting)
    • In the morning Maestà is noticed first because it is better illuminated by the natural daylight
    • In the afternoon Guidoriccio is preferred since the artificial light brightens this fresco
  • Situation: every course of action depends upon its material and social circumstances.
    • e.g. two different situations like a free exploration of a museum or a search for a particular museum content, need to be supported by a flexible tool able to consider different objectives associated to the situations.
  • Personalization:
    • A cognitive artifact is an artificial device designed to maintain, display or operate upon information in order to serve a representational function.
    • An artifact that mediates the fulfilment of an objective, providing individuals with the appropriate information at the right time (without pretending to know completely their interests and preferences that could change during a situation of use), can simplify the nature of the activity and, enhance the overall performance.
      • e.g. a system able to personalise the length and duration of the presentation according to different visiting strategies decreases the need for direct requests of more or less information.

Interaction Design Structure

  • Two levels:


1) Situation and Context Aware Interaction Mechanisms

    • The context and its physical, material, social and cultural components orient the exploration of the environment and the  meeting with its content.
      • Individual Level: individuals move in a space driven by
        • intentional motivations (personal interests and preferences)
        • situated strategies
        • properties of the environment
          • ethnographic  studies of visiting behaviours, we conceived a "visiting style module"  that linking the physical movements to the browsing of information space, provides personalised presentations according to the situated visiting strategies.
          • Explored the concept that visitors pathways mostly depend on natural and contextual affordances of the space, those properties that are «intrinsically» connected to a particular setting or that depend on the context of use. Hence, we experimented a design solution to augment the physical affordances of artworks by means of an auditory information space
      • Social Level:
        • Social memory is a product of knowledge distribution between the individual and tools or  other individuals who are involved in a specific activity.
        • Social memory undergoes a process of internalisation by which external activities are reconstructed and the knowledge that is acquired through a social process is individualised and can be reused in different contexts.
    • System is designed to support the process of embodying and transferring knowledge within a social group.
      • A visitor can take a snapshot of a situation of interest (externalisation of knowledge) and then reuse it to suggest a friend to follow a tour, to elaborate on contents , etc. (embodiment and transfer of knowledge).

2) Language, Contents and Reading Styles

    • Contents are structured as small blocks of information dynamically combined in form of audio presentations.
      • includes different types of contents
      • integrates contextual features  in order to make for more coherent flow of narration with respect to the interaction of the user with the environment.
      • auditory contents designed to provide cues about the surrounding physical space.

Use Scenario

Actual scenario in the Museo Civico in Siena:

    • Berthe and Samuel are two Belgian tourists.
    • They move in the centre of the room holding a portable guide called HIPS.
    • Berthe is attracted by the Maestà fresco.

HIPS (with a male quiet and polite voice):

In front of you there is the Maestà, one of the absolute masterpieces of the Sienese art, depicted by Simone Martini in 1315.

HIPS (a small pause, then a new male voice, with a strong Italian accent):

The Virgin is depicted as Sienese people's protector, and as a symbol of municipal justice: this particular devotion to the Virgin derived from the famous Battle of Montaperti in 1260, when Siena defeated the army of Florence and preserved its freedom.

Berthe laughs because the last voice had a strange Italian accent. Then she moves toward Santa Caterina:

HIPS: (with a female first person voice):

I'm Santa Caterina. I was born in Siena (...).

As she enjoys the comment, she takes a snapshot of the situation by pressing the hotspot button on the portable guide.

The system continues to provide information about the S. Caterina's life, but Berthe skips this part.

HIPS: (with a male quite and polite voice) :

The portrait of Caterina is set inside a Renaissance­styled shell; it looks like a real sculpture (...).

Berthe and Samuel move to leave the room.

HIPS: (with a male voice and 3D sounds effects)

Behind you, there is another important fresco of Simone Martini: Guidoriccio da Fogliano.

Curious of this artwork, she decides to go back and stops to admire the Guidoriccio. Samuel follows  Berthe and after few minutes they sit on a comfortable seat. They would like to visit the Pinacoteca but they don't know where it is. So, Berthe presses the Menu Button: by Find/Museums functionality she queries the system to know where the Pinacoteca is located and how to reach it. They go out the Museo Civico on the way to the Pinacoteca with the HIPS guide in their pocket.

Scenario exemplifies some basic concepts of situated interaction:

4.      The user is immersed in a rich audio environment.

      • Different reading styles characterise the way in which artworks are described from different perspectives (historical, artistic, anecdotal descriptions).

5.      The rhetorical styles are tailored to the context and to the iconographic contents

      • Artworks representing people are described at the first person, as if the character presents himself/herself.

6.      The rhythm of narration (length , duration) is tailored to the visitor's movement

      • Long and detailed descriptions are provided to visitors who move slowly and stop in front of each artwork.

7.      Experiential cognition is mediated by a natural input: the physical movement.

      • Reflective cognition is allowed by intentional and context driven interaction (explicit queries to the system).

Interaction Metaphors That Include Situation And Context

Individual Level

  • Physical spaces are not neutral
    • Objective: fill the gap between visitor's navigational strategies and information needs.
    • Technical point of view: continuously monitor the visitor's movements with a wireless connection between the portable guide (a PDA) and infrared emitters infrastructure.
    • The museum space becomes the system interface
    • physical movement is the main interaction vehicle
  • Adaptation of input and output to the situation

1.      Gap between the physical and the information space is bridged by the visitors' behaviours.

      • Ethnographic field studies in artistic exhibitions identified four categories of visitors based on their pathways, movements, and time of visit:
        • ant, fish, grasshopper, butterfly
      • Classification suggests how to isolate significant variables linked to physical movements and how to relate the physical movements to the browsing of information spaces.
      • Developed a visiting style module that classifies users within the four categories and tailors the delivered information accordingly.
        • Use an incremental bayesian algorithm
        • Iinformation can vary by length, duration and details.

2.      Visiting strategies can vary beyond physical pathways.

      • To access information that is not directly related to a certain position in the space, HIPS provides an off­line browsing function that supports the access to external information.

3.      Deliberate control of system behaviour is possible through the handling of simple and contextual buttons located on the PDA.

      • These controls change labels and function according to the current task:
        • cancel/confirm choices
        • stop/play audio comments
        • stop/more of this kind of information
        • cancel/select...
      • Design of these control buttons were inspired by the last generation mobile phones and ``game­boy like'' video­games.

4.      Environment sensitive User Interface

      • Visiting strategies are not sufficient to exploit the idea of the environment as interface.
        • Affordances of cultural settings play a central role in shaping the interaction, including:
          • properties that are «intrinsically» connected to a particular setting like the width of the artworks, their position, their artistic importance
          • architectural elements like access points to a room, arches and steps
          • dynamic and contextual configurations of elements present in the space (crowd, lights).
        • The role of the affordances in attracting the visitor, can be hampered when combined in certain configurations (crowd and bad light conditions often oblige the visitor to skip important artworks).
        • Possible design : audio triggers to attract the visitor's attention.
          • If the user reacts positively moving to the mentioned artwork listening to the description, then the system continues to provide information, otherwise it will just mention the artwork without further elaboration.

Social Level

  • Social memory develops from the externalization of knowledge through its internalization and recombination for later use in different activities.
    • HIPS provides very basic support for the development of a social memory in the community of visitors.
    • Externalisation of knowledge is realised by bookmarking a moment of the visit (pressing the "hotspot" button on the PDA, the visitor stores into the system the current position, an image of the artwork, the related description, a personal comment).
    • This knowledge is available for later use to suggest a friend to follow a tour, to elaborate on contents, to plan another tour etc. (embodiment and transfer of knowledge).

Situation-Aware Content

  • Audio descriptions in HIPS are segmented into Macronodes
    • Small blocks of information that are dynamically combined to form an audio presentation.
    • Each contains different kinds of contents with explicit reference to a physical position.
    • Flow of narration is made more fluid and harmonised to the context of visit.
  • Use of different reading styles, integration of 3D sounds and music provide means for designing rich audio environments.
  • Aims is to create an ``empathic effect'' mediated by human voices and immersive information spaces to engage the user in an intense meeting with art.

Example: The Aware Home - Georgia Tech
 

Context-Aware Toolkits

The Context Toolkit - Georgia Tech

  • Facilitates development and deployment of context-aware applications
  • Context - environmental information that is part of an application's operating environment and that can be sensed by the application
  • Set of context widgets and a distributed infrastructure that hosts the widgets
  • Context widgets are software components that provide applications with access to context nformation while hiding the details of context sensing.
  • Services of the Context Toolkit are:
    • encapsulation of sensors
    • access to context data through a network API
    • abstraction of context data through interpreters
    • sharing of context data through a distributed infrastructure
    • storage of context data, including history
    • access control for privacy protection
  • Example - Context Widget for Sensing Presence and Identity
    • The PersonNamePresence context widget senses the presence of a user and is able to identify her.
    • Relies on a sensor that provides both presence and identity information.
    • Uses iButtons that users snap in a reader to notify their presence.
      • iButton® is a 16mm computer chip armored in a stainless steel can
      • Three types:
        • Memory iButton - 64K and beyond of computer memory stores typed text or         digitized photos
        • Java™-powered cryptographic iButton
          •  A microprocessor and high-speed arithmetic accelerator.
          • Java Virtual Machine (VM) that is Java Card™ 2.0-compliant
        • Thermochron iButton
          • iButton tracks time and temperature,
          • Integrates a thermometer, a clock/calendar, a thermal history log, and 512 bytes of additional memory to store a shipping manifest.
      • Each button has a unique ID from which we derive the user's identity.
    • To dock, the user snaps her iButton into the reader.

    • The reader is mounted on an augmented whiteboard for impromptu  meetings.

    • When two users dock the whiteboard assumes a meeting is beginning and starts capturing


 audio and whiteboard drawings.

Human-Computer Integration

Affective Computing

  • Affective communication is communicating with someone (or something) either with or about affect
    • A crying child, and a parent comforting that child, are both engaged in affective communication
    • An angry customer complaining to a customer service representative
  • Focus in this area is on communication that involves emotional expression
  • Affective Mediation -- using computers to help communicate emotions to other people through various media

Brain Computer Interface

Biofeedback

  • Wadsworth Center (Albany, NY) - 1994
    • Used the mu (8-12 Hz) and the beta (13-28 Hz) rhythms in the EEG for communication
    •  Rhythms generated on the sensorimotor cortex
    • Subjects learned to move a cursor on a computer screen
    • The subjects use spontaneous EEG activity not tied to a specific evoking stimulus
    • One-dimensional control was realized with  electrodes above the left and right hemisphere.
    • Vertical cursor movement was established by summing up the mu power over both hemispheres.
    • When the sum exceeded a given threshold, the cursor moved upwards or otherwise downwards.
    • The task was either to hit a moving target on the screen or to move the cursor into a highlighted target on the screen.
    • A fast Fourier transformation (FFT) algorithm was used to calculate the power of the mu rhythm every 200 ms of EEG derivations on the left and right sensorimotor cortex.
    • Power values were converted into horizontal or vertical cursor movements by linear equations.
    • The coefficients of the linear equations were updated after each trial.
    • Two dimensional control was established by controlling horizontal movement with the sum of left and right mu power and vertical movement by the difference between left and right mu.
  • Emory University (Atlanta, GA) -  1997
    • Brain implant that can monitor extremely small-scale activity in the brain's motor area
    • Neurotrophic electrode - a hollow glass cone the size of a ballpoint pen's tip
    • Inserted through a hole drilled in the skull, into the cerebral cortex just above the ear
    • The patient's brain is scanned using magnetic-resonance imagingto show patterns of blood flow.
    • When the patient is asked to think about moving a limb, the motor area of the brain becomes active, and from its increased blood flow the precise location of the active region  can be identified.
    • This is where the electrode is implanted.
    • Inside the glass cone is a microscopically-thin gold wire, surrounded by nerve tissue extracted from the patient's  leg, which stimulates neurons from the surrounding cortex to grow into the cell.
    • Over a period of months, the  neurons fuse with the wire
    • Receives power from an induction coil sown into in a baseball cap worn by the patient.
    • Any signal picked up from motor neuron activity is detected and amplified by a tiny receiver placed just under the skull
    • Patient undergoes a training program using biofeedback
    • Electrical activity recorded by the implant controls the sound of a buzzer, and the patient gradually learns which thoughts make the buzzer sound  louder and faster.
    • Later, the buzzer is replaced by a cursor on a computer screen, and the patient learns to "think" the cursor from side to side.
    • Two patients:
      • First - suffered from a fatal degenerative motor neuron disease
      • Second - 53-year old stroke victim paralysed from the neck down
        • two implants, enabling him to separately control the horizontal and vertical movements of a cursor and select icons which trigger synthesised speech
  • Brainwave Hardware/Software
  • Implants
    • Kevin Warwick - University of Reading (UK) - 1998
      • human to host a microchip
      • inserted into Warwick's arm a glass capsule that holds several microprocessors


Half of it is an electric coil and half is a number of silicon chips

      • Cybernetics department has a number of doorways rigged so that they pass a radio signal  between the door frame
      • When he went through the doorways, the radio signal energizes the coil
      • It produces an electric current, which the chips use to send out an identifying signal, which the computer recognizes as being him


 

  • Cell-Transistor-Hybrid Systems
    • Neuron from a brain stem slice cultured on a triple gate field-effect transistor

    • Neuronal cell pattern produced  from brain stem slice after 14 days in culture