Simultaneous Use of Mobile and Fixed Devices

Proposal to Attend the CHI'2000 Workshop on
Future Mobile Device Interfaces

Brad A. Myers
Rob Miller
Benjamin Bostwick

Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
5000 Forbes Avenue
Pittsburgh, PA 15213-3891
(412) 268-5150
FAX: (412) 268-1266
bam+@cs.cmu.edu
http://www.cs.cmu.edu/~bam

The Pebbles research project (http://www.cs.cmu.edu/~pebbles) has been studying the use of mobile hand-held computers at the same time as other computing devices.  As people move about the world, they will be entering and leaving spaces where there are embedded or desktop computers, such as offices, conference rooms, and even "smart homes." We are exploring the many issues surrounding how to have the user interface, functionality, and information spread across multiple devices that are used simultaneously. For example, there are many ways that a personal digital assistant (PDA), such as a Palm Pilot or Windows CE device, can serve as a useful adjunct to a personal computer to enhance the interaction with existing desktop applications. New applications may distribute their user interfaces across multiple devices so the user can choose the appropriate device for each part of the interaction.  A key focus of our research is that the hand-held computers are used both as output devices and as input devices to control the activities on the other computers. The following scenarios illustrate some of the capabilities we are already investigating:

• The presenter of a talk has a laptop where the display is projected onto a large screen. The laptop's powerful processor is needed to control the animations and external applications that are part of the presentation. (Or similarly, the meeting room has a built-in computer with a data projector to control the presentation.) The presenter walks in with a hand-held PDA. The laptop communicates with the PDA, so on the PDA is displayed the current slide's notes.  Gestures on the PDA cause the presentation to go forward, backward or skip to a specific slide under discussion.  Also on the PDA are custom controls to switch among various other applications on the laptop which the presenter will be demonstrating and discussing. Each member of the audience of the presentation has carried in their mobile hand-held computer, and each attendee sees on their personal hand-held the current slide, which can be kept synchronized with the talk. Each person can also make private notes and annotations on the hand-held.  When enabled by the presenter, an audience member's marks on the PDA will display on the main screen for general viewing and discussion. Various techniques are used to coordinate the actions of multiple people who are interacting with the displayed content [Myers 1998].
• When the user is sitting and working at home or in the office, various mobile devices are carried in and laid on the desk: a laptop, a PDA, a cell-phone, etc. Some of the newest hand-helds such as the Palm V and the HP Jornada 430 have rechargeable batteries that are recharged when the device is in its cradle, so the user is supposed to have the device connected to the PC whenever the user is next to the PC and it might as well be doing something useful. We envision having the cradle right next to the PC on the left side of the keyboard, while the mouse is on the right. For example, as the user works, various controls might appear on the screen of the PDA rather than on the desktop computer's screen.  Our research has shown that users can use their left hand on the PDA to scroll documents shown on the desktop's screen just as fast or faster than using the regular GUI scroll bars or using devices such as the scroll wheel built into some mice [Myers 2000]. The user's custom shortcuts for the laptop applications also appear on the PDA, and the user has memorized their location and can operate them quickly without looking. Information can be easily moved among the devices, and other information is automatically distributed based on predefined user preferences.
• In a "Smart Meeting Room," the main large displays show the shared displays under discussion by the group. Built into in the room are cameras, microphones, and displays, so users can speak, gesture in the air, as as using conventional interaction techniques with keyboards and mice. Individuals enter and leave the room, each carrying their own PDA. While in the meeting room, someone might want more details on an item displayed on a large display.  Rather than disrupting the main activities and the main display, the PDA can be pulled out, and pointed at the item on the main display. Then the user can privately have the additional specialized information displayed on their PDA. The display of the information is appropriately adjusted to the limited size of the PDA screen. We are also investigating ways to fluidly move interactions among the modalities, so users can start speaking and gesturing in the air (interpreted by cameras and microphones), and then dynamically switch to handwriting and tapping on the PDA.

There are many significant research issues involved in bringing these visions to fruition, which we are investigating. We are particularly interested in the appropriate ways to distribute the user interfaces across multiple devices, how to support multiple people interacting with the same screen using their various devices as auxiliary input and output devices (which is sometimes called "single-display groupware" [Stewart 1997]), the automatic creation of appropriate and usable control panels from high-level specifications, and usability issues with multi-device interaction techniques.

The Pebbles research project has made substantial progress by building example applications, releasing them for general use, and formally testing them in usability experiments. Several of our existing applications support meetings where the participants are co-located. All participants' PDAs are in continuous two-way communication with the main computer which is often projected on a screen to serve as the focal point of the discussion. Some of our initial applications use the PDAs as remote mice and keyboards so that everyone in the meeting can control the main computer. The PDA might be used to control a PowerPoint presentation while displaying the slide notes and titles on the PDA, as a shared whiteboard that supports multiple inputs simultaneously, for private side messages via a "chat" program, and to display multiple cursors for pointing and scribbling on arbitrary applications [Myers 1998]. We are currently investigating a number of groupware issues, including appropriate floor control mechanisms, and how to fluidly move information between the public and private displays. Another set of applications supports a single person using the PDA as an extra input and output device to enhance desktop applications. The PDA can be used as a scrolling device, as a general-purpose button panel (to create screens of "shortcuts"), as an index page or table of contents for web surfing, and to cut and paste information back and forth from the PDA to the PC. These applications have been downloaded over 15,000 times already, and are available from our web site: http://www.cs.cmu.edu/~pebbles.

Brad Myers of the Pebbles research project would like to participate in this workshop because we are interested in discussing various visions of the future mobile environment. We believe that the fact that mobile devices are carried in and out of spaces where there are other mobile and fixed devices is an under-studied aspect, and that this can have a big impact on the user interfaces, communication mechanisms, hardware and operating system of the devices.

Short Biography of Brad Myers

Brad A. Myers is a Senior Research Scientist in the Human-Computer Interaction Institute in the School of Computer Science at Carnegie Mellon University, where he is the principal investigator for various projects including the Pebbles PDA Project, the User Interface Software Project, Demonstrational Interfaces, and the Natural Programming Project. He is the author or editor of over 190 publications, including the books Creating User Interfaces by Demonstration and Languages for Developing User Interfaces, and he is on the editorial board of five journals. He has been a consultant on user interface design and implementation to 30 companies. Myers received a PhD in computer science at the University of Toronto where he developed the Peridot UIMS. He received the MS and BSc degrees from the Massachusetts Institute of Technology during which time he was a research intern at Xerox PARC. From 1980 until 1983, he worked at PERQ Systems Corporation. His research interests include Mobile Devices, User Interface Development Systems, user interfaces, Programming by Example, programming languages for kids, Visual Programming, interaction techniques, window management, and programming environments. He belongs to SIGCHI, ACM, IEEE Computer Society, IEEE, and Computer Professionals for Social Responsibility.

References

[Myers 1998] Myers, B.A., Stiel, H., and Gargiulo, R. “Collaboration Using Multiple PDAs Connected to a PC,” in Proceedings CSCW'98: ACM Conference on Computer-Supported Cooperative Work. 1998. Seattle, WA: pp. 285-294.

[Myers 2000] Myers, B.A., Lie, K.P.L., and Yang, B.-C.J. “Two-Handed Input Using a PDA And a Mouse,” in Proceedings CHI'2000: Human Factors in Computing Systems. April, 2000. The Hague, The Netherlands: To Appear.

[Stewart 1997] Stewart, J.E. “Single Display Groupware,” in SIGCHI'97 Adjunct Proceedings: Human Factors in Computer Systems, Extended Abstracts. 1997. Atlanta, GA: pp. 71-72.