Technologies for multimodal interfaces

A project in the DARPA Command Post of the Future Initiative

Principal Investigators: Ken Forbus, Brian Dennis

Multimodal interfaces that enable commanders and their staff to interact with information systems and each other via sketching, speech, and graphical visualization could revolutionize work processes in a command post. Progress in computing technology is rapidly providing much of the infrastructure to achieve such interfaces, including signal processing, digital ink, advanced displays, and other technologies. However, there are four areas where fundamental progress is needed to achieve such interfaces:

  1. Sketch understanding is needed so that the software understands what is being drawn in domain-specific terms. A commander’s concept sketch, for instance, needs to be interpreted as a partial specification of a course of action (COA).
  2. Analogical processing is needed to help turn concept sketches into proposed COAs, to understand what differences are significant, to detect discrepancies between plans and events, and to recognize patterns of events in terms of higher-level concepts (e.g., feint)
  3. Multi-dimensional presentation is needed to support the presence of a range of interactive devices, collaboration in achieving a task, and differing staff member tasks.

To create the multimodal interfaces needed for the CPOF, we are proposing software to bridge these gaps. Specifically, we will create a sketch understanding system that can be used in multimodal interfaces for constructing and evaluating courses of action via map-based sketching and speech. We will implement analogical processing systems for supporting sketch understanding, elaboration of courses of action based on libraries of operations plans, visualization of similarities and differences involving alternate COAs and the evolving situation versus plans in execution, and for suggesting higher-level explanations for enemy actions. We will develop software and techniques for automating the task of deriving multidimensional presentations based upon multiple axes of choice including, device output capabilities, physical location of users, multi-device output and task sensitivity.

More information can be found here.

Back to Sketching: Towards a computational model page | Back to Previous Projects page | Back to QRG Home Page