Evaluating SME-Elicited Knowledge

Authors: Julie Fitzgerald, Mike Pool, Bob Schrag.


Evaluation of mixed-initiative systems is an evolving field of study. Information Extraction and Transport (IET) has been conducting large-scale knowledge base evaluations for the past six years. We describe an approach to mixed-initiative systems evaluation that was devised for DARPA’s Rapid Knowledge Formation (RKF) program and how this was implemented in the course of two very distinct domain challenge problems. We assess the methods and results used and make recommendations for future mixed-initiative evaluations.

For more information, contact IET