RESEARCH DIRECTIONS IN SMART SPACES AND INFORMATION VISUALIZATION SOL RFI 98-04 DUE 02/25/98 POC Dr. Kevin L. Mills Section I. Cover Page ===================== Title: Hand-Held Devices to Control Wall Size Displays Topic Area: Smart Spaces Name of Submitter: Brad A. Myers Telephone Number: (412) 268-5150 E-mail address: bam+@cs.cmu.edu Institution: Carnegie Mellon University Mailing address: Brad A. Myers Human Computer Interaction Institute Carnegie Mellon University Pittsburgh, PA 15213-3891 Section II. Details of Submitted Idea ====================================== (1) Innovative Capability Envisioned ------------------------------------ As offices, meeting rooms and homes begin to have computer controlled screens and wall-sized displays, it will be increasingly inconvenient to type or use a mouse to input and manipulate information, especially in multi-user situations. Using speech recognition and gestures in the air interpreted by cameras, will be important capabilities, but there are many situations in which these will not be adequate or appropriate. People in these meetings will have hand-held devices on which they are taking notes during the meetings. Such devices today are the 3Com PalmPilot and the IBM "ThinkScribe" (Digital Clipboard), but future devices will be more convenient and powerful. It is clear that these devices will have substantially larger memories, somewhat larger displays, better readability, wireless connections, and capabilities for audio recording and speech recognition. They may incorporate a camera, be lightweight, and have full-size displays (8.5x11). Our vision is that these devices should, in addition to being private note taking devices, also be used to input to and control all the other devices in the room, including the wall displays. Through the wireless connections, each of the personal devices can be connected to each other and to the large display, and the user will be able to naturally and seamlessly switch from local, personal note taking to public, global input and control. (2) Possible Applications and Benefits -------------------------------------- The hand-held device could serve as a "smart input device" to any computer. This might be a wall-sized display for meetings, or just a friend's computer that two people are sharing. Each person in a meeting can have their own cursor on the wall-sized display, and movements on the personal hand-held device can be used to control the remote cursor. New text can be entering by freehand writing on each hand-held device (keyboards can be attached when appropriate, but in meetings, keyboards are often socially unacceptable). Information can be exchanged between people by transferring it directly from one hand-held device to another. This might be used for real-time side notes during a meeting, as well as for transferring documents and other information among the people. For meetings, many of the formal aspects could be handled through the connections of people's hand-held devices, such as taking attendance, voting, scheduling future meetings, and distributing work items to the appropriate people and the handing out the minutes of the meeting. A person's "identity" could be stored in the hand-held device, so that when in front of any computer, that computer would know the person's preferences, and applications anywhere would conform to his or her style. Audio and video transcripts of meetings can be annotated with notes taken on the hand-held device, and the transcripts can be searched and selectively replayed by selecting the notes. This would not necessarily require speech or vision understanding. Clicking on special icons on the hand-held device might be used to annotate the transcript to say when an important event just happened, to enhance the usefulness of the transcript. (3a) Major Technical Obstacles ------------------------------ The wireless technology in the hand-held devices needs to support continuous long-term communication from multiple devices in the same room while the user is writing on the device. Many wireless technologies being developed today for hand-held devices are only able to handle one-to-one communication where there is an unobstructed line of site with only one device operating in a room at a time. The display technology needs to be larger, preferably foldable (e.g., so a 8.5 x 11 display can be folded and put in a pocket), lighter in weight, easier to read, and require less power. Today's widgets are not appropriate when multiple people are sharing the same display (which is called "Single-Display Groupware"). Some problems include: Selection handles for graphics and text selection need to be marked to show to which user they belong. Palettes and mode menus need to be able to show the status of each person sharing the display, not just of a single user as today. Pop-up menus and pull-down menu-bars when popped up by one user may cover the work area of another user. People will not be comfortable putting all their information into a hand-held device unless there is some security so if the device is lost, the information would still be available to the user, and would not be available to anyone who found the device. (3b) Plausible Approaches ------------------------- The wireless problems are fairly easily solved if the requirements are taken into account. The display requirements are much harder, but are being pursued by many vendors and researchers. Research is needed into new interaction techniques and new standards for interaction that will handle very large and very small displays, and also for single displays shared by multiple people. For security, continual or at least frequent back-ups of the hand-held device would be required. Some secure password or biometric scheme should be available to encrypt the content against unauthorized access if the device is lost or stolen. Section III. Additional Information =================================== Our current Pebbles project is exploring the use of PalmPilots to control the PC's display, and how PalmPilots can be used more effectively in meetings. See: http://www.cs.cmu.edu/~pebbles