Case Study - Technology Solution - Knowledge Base (KB) Evaluation
Whether a knowledge base (KB) has been populated with traditional knowledge engineering methods (using subject matter experts), through an automated learning algorithm, or a combination of both, mechanisms are needed to measure its quality - how well does it do the job for which it was built? IET has had extensive experience over the last decade in the design and implementation of knowledge base evaluations. The result is a two-step process that has been proven effective and efficient for performing these evaluations; 1) generation of challenge problems that will be used during the evaluation and 2) evaluation execution - measuring the results of the application of these challenge problems.
Although each challenge problem exercise is unique, depending on the domain and customer requirements, IET has developed a repeatable methodology that has been successfully employed during a variety of knowledge base evaluation situations. The steps in this process include:
- Generate a scenario - a scenario is a combination of individual events and/or activities, with related scenarios composing a larger story. The first step is to research the chosen domain(s) for rich stories that can be broken down into a number of scenarios ranging from the simple to the more complex.
- Collect source materials and data - once the stories and scenarios have been identified and worked, data collection can begin. Sources of data might be intelligence reports, sensor reports, newspaper articles, etc.
- Analyze data - analysts read the collected source materials to assess and extract data and use this knowledge of the data to develop a functional classification scheme and identify and define domain models. This process also defines the systematic, documented ground truth (e.g., model instances with their constituent nodes, links, and patterns).
- Prepare and deliver the "Domain Package" - that gathers all the materials associated with a challenge problem release in a wrapper of explanation and information about that problem. These packages include source materials, domain models, and background knowledge.
IET begins the task of evaluation execution by defining the measures or KB attributes that are desired (e.g., performance, scalability, completeness, maintainability, stability as opposed to brittleness, etc.). These measures fall into two main categories:
- Objective measures - Such as performance and scalability. A sample metric for measuring complexity might be checking the KB for undesirable syntactically discernible style conditions.
- Subjective measures - Such as stability as opposed to brittleness and completeness. One example would be IET's metric of knowledge compositionality or the degree to which a KB's atomic representations match those of a human (the supposition being that representations that are highly compositional are inherently more extensible and maintainable).
Using these measures, IET employs a process modeling approach to facilitate the discovery of KB building "sweet spots" - regions of KB process indicators associated with local maxima in the desired KB attributes.