Using Multiple Devices Simultaneously for Display and Control

Proposal to Attend the Inter-agency Workshop on Smart Environments

Brad A. Myers

Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
5000 Forbes Avenue
Pittsburgh, PA 15213-3891
(412) 268-5150
FAX: (412) 268-1266
bam+@cs.cmu.edu
http://www.cs.cmu.edu/~bam

The Pebbles research project (http://www.cs.cmu.edu/~pebbles) has been studying the use of hand-held personal digital assistants (PDAs) at the same time as other computing devices.  For example, there are many ways that a PDA can serve as a useful adjunct to a personal computer to enhance the interaction with existing applications.  New applications may distribute their user interfaces across multiple devices so the user can choose the appropriate device for each part of the interaction.  A key focus of our research is that the hand-held computers are used both as output devices and as input devices to control the activities on the other computers. The following scenarios illustrate some of the capabilities we are already investigating:

• The presenter of a briefing or talk has a laptop where the display is projected onto a large screen. The laptop's powerful processor is needed to control the animations and external applications that are part of the presentation.  In the presenter's hand is a PDA on which the current slide's notes are displayed.  Gestures on the PDA cause the presentation to go forward, backward or skip to a specific slide under discussion.  Also on the PDA are custom controls to switch among various other applications on the laptop which the presenter will be demonstrating and discussing. Each member of the audience of the presentation sees on their personal hand-held the current slide, which is kept synchronized with the talk, but each person can also make private notes and annotations on the PDA.  When enabled by the presenter, an audience member's marks on the PDA will display on the main screen for general viewing and discussion.
• In a command center, the main large displays show maps, schedules, and other visualizations of the current situation that will be useful to the group.  Individuals enter and leave, each carrying their own PDA. While in the command center, someone might want more details on an item displayed on a large display.  Rather than disrupting the main activities and the main display, the PDA can be pulled out, and a special unobtrusive cursor will appear on the main display, so the user can point to the item of interest.  Then the user can privately "drill-down" to get the additional specialized information displayed on their PDA. The display of the information is appropriately adjusted to the limited size of the PDA screen.
• When the user is sitting and working, various devices are laid on the desk: a laptop, a PDA, a cell-phone, etc. They immediately communicate with each other to establish each device's capabilities and specifications. As the user works, various controls appear on the screens of the other devices rather than on the laptop's screen.  For example, scroll bars might drawn on the PDA so the user can operate them with the left hand while using the mouse with the right hand, which has been shown to be fast and effective. The user's custom shortcuts for the laptop applications also appear on the PDA, and the user has memorized  their location and can operate them quickly without looking. Information can be easily moved among the devices, and other information is automatically distributed based on predefined user preferences.
• As the user points a PDA at a light switch, at a photocopier in an office, at a machine tool in a factory, at a VCR at home, at a piece of test equipment in the field, or at almost any other kind of device, the device sends to the PDA a description of its input / output capabilities. The PDA converts this into an appropriate control panel, taking into account the properties of the controls that are needed, the properties of the PDA (the display type and input techniques available), and the properties of the user (what language is preferred, whether left or right handed, how big the buttons should be based on whether the user prefers using a finger or a stylus). The user can then control the device using the PDA. The device will not need to dedicate much processing power, hardware, or cost to the user interface, since it will only need to contain a description of its capabilities and storage for the current settings, along with hardware for wireless communication.  The PDA programs will use intelligent "model-based" techniques to create useful and appropriate interfaces that are customized for each user.

There are many significant research issues involved in bringing these visions to fruition, which we are committed to investigating. We are particularly interested in the appropriate ways to distribute the user interfaces across multiple devices, how to support multiple people interacting with the same screen using their various devices as auxiliary input and output devices (which is sometimes called "single-display groupware"), the automatic creation of appropriate and usable control panels from high-level specifications, and usability issues with multi-device interaction techniques.

The Pebbles research project has made substantial progress by building example applications, releasing them for general use, and formally testing them in usability experiments. Several of our existing applications support meetings where the participants are co-located. All participants' PDAs are in continuous two-way communication with the main computer which is often projected on a screen to serve as the focal point of the discussion. Some of our initial applications use the PDAs as remote mice and keyboards so that everyone in the meeting can control the main computer. This might be used to control a PowerPoint presentation while displaying the slide notes and titles on the PDA, as a shared whiteboard that supports multiple inputs simultaneously, for private side messages via a "chat" program, and to display multiple cursors for pointing and scribbling on arbitrary applications. Another set of applications supports a single user using the PDA as an extra input and output device to enhance desktop applications. The PDA can be used as a scrolling device, as a general-purpose button panel (to create screens of "shortcuts"), as an index page or table of contents for web surfing, and to cut and paste information back and forth from the PDA to the PC. These applications have been downloaded over 12,000 times already, and are available from our web site.  More detailed papers about them are also available.

Members of the Pebbles research project would like to participate in this workshop because we agree that users will increasingly have a collection of devices, and they will often be using more than one at a time. Our experience with building toolkits (such as the Garnet and Amulet systems) will help us build appropriate infrastructure to support the creation of these applications. We feel that our experience and research direction has much to contribute in studying how multiple devices can be used simultaneously in smart environments.