Title
Reliability and Validity of Query Intent Assessments
Author
Verberne, S.
van der Heijden, M.
Hinne, M.
Sappelli, M.
Koldijk, S.
Hoenkamp, E.
Kraaij, W.
Publication year
2013
Abstract
In most intent recognition studies, annotations of query intent are created post hoc by external assessors who are not the searchers themselves. It is important for the field to get a better understanding of the quality of this process as an approximation for determining the searcher's actual intent. Some studies have investigated the reliability of the query intent annotation process by measuring the interassessor agreement. However, these studies did not measure the validity of the judgments, that is, to what extent the annotations match the searcher's actual intent. In this study, we asked both the searchers themselves and external assessors to classify queries using the same intent classification scheme. We show that of the seven dimensions in our intent classification scheme, four can reliably be used for query annotation. Of these four, only the annotations on the topic and spatial sensitivity dimension are valid when compared with the searcher's annotations. The difference between the interassessor agreement and the assessor-searcher agreement was significant on all dimensions, showing that the agreement between external assessors is not a good estimator of the validity of the intent classifications. Therefore, we encourage the research community to consider using query intent classifications by the searchers themselves as test data. © 2013 ASIS&T.
Subject
Communication & Information
MNS - Media & Network Services
TS - Technical Sciences
Infostructures
Informatics
Information Society
Information Retrieval
Information Seeking
Classification Scheme
Intent Recognition
Query Formulation
Reliability and Validity
Research Communities
Spatial Sensitivity
Artificial Intelligence
Software Engineering
To reference this document use:
http://resolver.tudelft.nl/uuid:29937db0-3c66-4555-b373-4d7a6c780931
DOI
https://doi.org/10.1002/asi.22948
TNO identifier
482957
Source
Journal of the American Society for Information Science and Technology, 64 (11), 2224-2237
Document type
article