Extending the Rule Space Model to a Semantically-Rich Domain: Diagnostic Assessment in Architecture.
Final rept. Apr 89-Aug 93,
EDUCATIONAL TESTING SERVICE PRINCETON NJ
Pagination or Media Count:
This paper presents a technique for applying the Rule Space model of cognitive diagnosis to assessment in a semantically-rich domain. Responses to 22 architecture test items, developed to assess a range of architectural knowledge, were analyzed using Rule Space. Verbal protocol analyses guided the construction of a model of examinee performance, consisting of processes for constructing an initial representation of an item labeled understand, forming goals and performing actions based on those goals solve, and determining whether goals have been attempted and satisfied check. Item attributes, derived from these processes, formed the basis for diagnosis. Our technique extends Rule Spaces applicability by defining attributes in terms of item characteristics and the causal relations between characteristics and the problem-solving model. Data were collected from 122 architects of various ability levels students, architecture interns, and professional architects. Rule Space successfully classified approximately 65, 90, and 40 of examinees based, respectively, on attributes associated with the understand, solve, and check processes of the problem-solving model. The findings support the effectiveness of Rule Space in a complex domain and suggest directions for developing new architecture items by using attributes particularly effective at distinguishing among examinees of different ability levels.
- Computer Programming and Software