Personal Universal Controller

Command Post



Using a Hand-Held as a
Personal Universal Controller


The Pebbles project is investigating how a hand-held computer can be used as a "Personal Universal Controller".

Personal digital assistants (PDAs) like the PocketPC and Palm Pilot are becoming increasingly ubiquitous, and with wireless technologies such as BlueTooth and IEEE 802.11, they will be in close interactive communication with other devices. Furthermore, cell-phones and pagers, which primarily used for communication, are increasingly becoming programmable. We are  investigating how these kinds of hand-held devices devices can be used to control all kinds of home, office and factory equipment. The concept is that when users point their own hand-held at a light switch, at a photocopier in an office, at a machine tool in a factory, at a VCR at home, at a piece of test equipment in the field, or at almost any other kind of device, the device will send to the hand-held a description of its control parameters. The hand-held uses this information to create an appropriate control panel, taking into account the properties of the controls that are needed, the properties of the hand-held (the display type and input techniques available), and the properties of the user (what language is preferred, whether left or right handed, how big the buttons should be based on whether the user prefers using a finger or a stylus). The user can then control the device using the hand-held. The device will not need to dedicate much processing power, hardware, or cost to the user interface, since it will only need to contain a description of its capabilities and storage for the current settings, along with hardware for wireless communication. The hand-held programs will use intelligent "model-based" techniques to create useful and appropriate interfaces that are customized for each user. Furthermore, though a collaboration with the Universal Speech Interfaces project at CMU, we are demonstrating that the same specification can be used to generate a speech interface, and then the appliance can be controlled using speech and/or a graphical user interface.

In our preliminary study, using the Palm and PocketPC interfaces shown below were twice as fast and resulted in one-fifth the errors compared to the manufacturer's interface. See the publications below.

Note that you cannot use any of the Pebbles software to control real applications (other than a PC) at this time. See Links section below.

Specification Language

A key part of our vision is the description of the control parameters that the controller uses for automatically creating control panels. We have developed a language for specifying these control parameters, and a protocol for transmitting changes to the state of these parameters between the appliance and the controller. This language and the protocol are based on the XML standard. You may be interested in the documentation and DTDs for these languages:




Part of this research was performed in collaboration with Maya Design.

CMU people on the PUC project include:


  • Prof. Jodi Forlizzi
  • Pegeen Shen
  • Kevin Litwack
  • Rajesh Seenichamy
  • Suporn Pongnumkul
  • Jennifer Li
  • Mathilde Pignol
  • Marc Khadpe

Publications about Pebbles and the PUC

The complete specifications for the PUC protocols are available. See specification language section above.


  • Talk by CMU and Maya on progress to date presented to the Pittsburgh Digital Greenhouse, February 13, 2002. PowerPoint Slides.

Best Reference:

  • Jeffrey Nichols, Brad A. Myers, Michael Higgins, Joe Hughes, Thomas K. Harris, Roni Rosenfeld, Mathilde Pignol. "Generating Remote Control Interfaces for Complex Appliances." CHI Letters: ACM Symposium on User Interface Software and Technology, UIST'02, 27-30 Oct. 2002, Paris, France. To appear. pdf

Other Papers:

  • Jeffrey Nichols, Brad Myers, Thomas K. Harris, Roni Rosenfeld, Stefanie Shriver, Michael Higgins, Joseph Hughes. "Requirements for Automatically Generating Multi-Modal Interfaces for Complex Appliances," IEEE Fourth International Conference on Multimodal Interfaces, Pittsburgh, PA. October 14-16, 2002. pp. 377-382. pdf
  • Jeffrey W. Nichols. "Using Handhelds as Controls for Everyday Appliances: A Paper Prototype Study." ACM CHI'2001 Student Posters. Seattle, WA. March 31-April 5, 2001. pp. 443-444. Adobe Acrobat (pdf)
  • Brad A. Myers, Jeff Nichols, Rob Miller. "User Interfaces that Span Hand-Held and Fixed Devices" Workshop on Distributed and Disappearing User Interfaces in Ubiquitous Computing at CHI'2001, Seattle, WA. Albrecht Schmidt, Peter Ljundgstrand, and Anind Dey, editors. University of Karlsruhe Faculty of Information Technical Report 2001-6. ISSN 1432-7864. html
  • Jeffrey Nichols, Brad A. Myers, Rob Miller. "Personal Interfaces in Ubiquitous Environments". Workshop on Building the Ubiquitous Computing User Experience at CHI'2001, Seattle, WA. html
  • Brad A. Myers and Jeffrey Nichols, "Communication Ubiquity Enables Ubiquitous Control." 'Boaster' for Human-Computer Interaction Consortium (HCIC'2002). Winter Park, CO, Feb, 2002.  html
  • Jeffrey Nichols. "Informing Automatic Generation of Remote Control Interfaces with Human Designs" CHI'2002 Student Posters. Minneapolis, Minnesota, April 20-25, 2002. pp. 864-865. Adobe Acrobat (pdf)

Article about this research:

  • Michael Yeomans, "CMU Scientists Improving Computers' People Skills," Pittsburgh Tribune-Review, Tuesday, October 22, 2002. pages B7, B10. html
  • Mike Crissey, "Designers Work on All-in-One Remote," Associated Press, August 27, 2002. html


Funding for the use of Pebbles as a Personal Universal Controller is provided by:


We are investigating various ways to connect our PUC to appliances, including the following:

Note: the PUC system cannot be used to control real appliances today. Some systems for using handhelds as remote controls today include:

  • Philips Pronto
  • OmniRemote Pro by Pacific Neo-Tek, Inc. for Palm
  • NoviiRemote by NoviiMedia for Palm and PocketPC
  • VITO Remote by VITO Technology for PocketPC
  • UniversalRemoteCE by Pyramid Peak Design for PocketPC
  • etc.