Sketching: Towards a computational model
Sketching is a powerful means of interpersonal communication. We view sketching as a combination of interactive drawing and linguistic interaction. The drawing carries the spatial aspects of what is to be communicated. The linguistic interaction provides a complementary conceptual channel that guides the interpretation of what is drawn and provides information that is not easily depicted spatially. Most people are not artists, and even artists cannot produce, in real time, drawings of complex objects and relationships that are recognizable solely visually without breaking the flow of conversation. The verbal description that occurs during drawing, punctuated by written labels, compensates for inaccuracies in drawing. Follow-up questions may be needed to disambiguate what aspects of a drawing are intended versus accidental.
Our goal is a computational model of sketching. Research on sketching provides an arena for investigating the intersection of conceptual knowledge, visual understanding, and language, making it a valuable area for investigation in order to understand human cognition. Creating software that can interact with human partners via sketching could have a revolutionary impact on human-computer interaction.
The framework we are developing characterizes sketching in terms of four dimensions of knowledge and skills: visual understanding, language understanding, conceptual understanding, and drawing skills. Variations along these dimensions determine how many different types of interactions something having those skills can participate in. Our work focuses heavily on improving visual understanding and conceptual understanding.
nuSketch: A Multimodal Architecture for Sketching
nuSketch is designed as a general-purpose multimodal architecture to support sketching. We are currently using it in three applications:
- COA Creator. A system for creating military courses of action via sketching.
- Tactical Decision Coach. A system for helping students learn tactics, by sketching solutions to tactical problems that are then critiqued by comparing and contrasting the student's solution to expert solutions, via analogy.
- sKEA. The sketching Knowledge Entry Associate is being developed to enable experts to enter knowledge via sketching.
This research is leading us to develop domain theories that express visual symbology (i.e., the representational contents of visual symbols, genres of diagrams, and relationships between diagrams) and everyday physical semantics (i.e., the assumptions about the physical world that are typically used in interpreting diagrams). We are also using techniques from qualitative spatial reasoning to do high-level visual perception and visual analogies.
Selected Relevant Papers
Ferguson, R. W. and Forbus, K.D. (1995). Understanding illustrations of physical laws by integrating differences in visual and textual representations. AAAI Fall Symposium on Computational Models for Integrating Language and Vision
Ferguson, R. W. & Forbus, K. D. (2000). GeoRep: A flexible tool for spatial representation of line drawings. Proceedings of the 17th National Conference on Artificial Intelligence. Austin, Texas: AAAI Press. [NB: on the paper the conference is incorrectly identified as the "18th National Conference"]
Forbus, K. D., Ferguson, R. W. & Usher, J. M. (2001). Towards a computational model of sketching. Proceedings of the International Conference on Intelligent User Interfaces. Sante Fe, New Mexico.
Forbus, K. D., Ferguson, R. W. & Usher, J. M. (2000). Boundary-based multimodal input for geographic planning sketches. In P. Healy (Ed.), Proceedings of the First International Workshop on Interactive Graphical Communication. London: Queen Mary College, University of London.
Ferguson, R. W., Rasch, R. A. J., Turmel, W. & Forbus, K. D. (2000). Qualitative spatial interpretation of Course-of-Action diagrams. Proceedings of the 14th International Workshop on Qualitative Reasoning. Morelia, Mexico.