Heuristic Evaluation Quality Score (HEQS): A Measure of Heuristic Evaluation Skills
http://www.usabilityprofessionals.org/upa_publications/jus/2007_february/kirmani_rajasekaran_heqs.html
  peer-reviewed

Kirmani, Shazeeye and Shamugam Rajasekaran
Journal of Usability Studies
2007

Abstract:

Heuristic Evaluation is a discount usability engineering method involving three or more evaluators who evaluate the compliance of an interface based on a set of heuristics. Because the quality of the evaluation is highly dependent on their skills, it is critical to measure these skills to ensure evaluations are of a certain standard. This study provides a framework to quantify heuristic evaluation skills. Quantification is based on the number of unique issues identified by the evaluators as well as the severity of each issue. Unique issues are categorized into eight user interface parameters and severity is categorized into three. A benchmark computed from the collated evaluations is used to compare skills across applications as well as within applications. The result of this skill measurement divides the evaluators into levels of expertise. Two case studies illustrate the process, as well as its applications. Further studies will help define an expert's profile.