Dialogue Theory for Virtual Environments.

We are developing a theory of parsing and generation which will bridge between language-multimedia surface structure and internal meaning structures. The idea is that a system could receive a variety of perceived data and this could include spoken andor typed natural language, visual information, tactile information and possibly other inputs. It will then use a new kind of multimedia grammar to parse those inputs and discover their associated structure. Finally, it will use the discovered structure to translate the input into meaning structures that can be used internally for understanding an input and deciding on appropriate answers. Our theory at this point is bidirectional. Specifically, the same algorithm used to parse inputs can be used to generate Outputs. If a meaning structure is to be expressed to the user, the system must find the set of language constructs that will express the target meaning. This involves a parsing-like process in which the structure of the meaning is analyzed and the correct rules needed to express the meaning are discovered. Here those rules then lead to a set of multimedia language constructs that can be assembled to express the originally given meaning. The theory includes the specification of a set of operators 01, 02, 03,... each with a syntactic component, a semantic component, and some applicability criteria. The parsing algorithmwhich can also be used for generation searches for the sequence of operators 01 that will account for the observed surface structure. Their semantic parts then are invoked to find the target meaning. A key to the research is to find a representation for the operators that will enable fast and efficient processing.