Print Email Facebook Twitter Reliability and Validity of Query Intent Assessments Title Reliability and Validity of Query Intent Assessments Author Verberne, S. van der Heijden, M. Hinne, M. Sappelli, M. Koldijk, S. Hoenkamp, E. Kraaij, W. Publication year 2013 Abstract In most intent recognition studies, annotations of query intent are created post hoc by external assessors who are not the searchers themselves. It is important for the field to get a better understanding of the quality of this process as an approximation for determining the searcher's actual intent. Some studies have investigated the reliability of the query intent annotation process by measuring the interassessor agreement. However, these studies did not measure the validity of the judgments, that is, to what extent the annotations match the searcher's actual intent. In this study, we asked both the searchers themselves and external assessors to classify queries using the same intent classification scheme. We show that of the seven dimensions in our intent classification scheme, four can reliably be used for query annotation. Of these four, only the annotations on the topic and spatial sensitivity dimension are valid when compared with the searcher's annotations. The difference between the interassessor agreement and the assessor-searcher agreement was significant on all dimensions, showing that the agreement between external assessors is not a good estimator of the validity of the intent classifications. Therefore, we encourage the research community to consider using query intent classifications by the searchers themselves as test data. © 2013 ASIS&T. Subject Communication & InformationMNS - Media & Network ServicesTS - Technical SciencesInfostructuresInformaticsInformation SocietyInformation RetrievalInformation SeekingClassification SchemeIntent RecognitionQuery FormulationReliability and ValidityResearch CommunitiesSpatial SensitivityArtificial IntelligenceSoftware Engineering To reference this document use: http://resolver.tudelft.nl/uuid:29937db0-3c66-4555-b373-4d7a6c780931 DOI https://doi.org/10.1002/asi.22948 TNO identifier 482957 Source Journal of the American Society for Information Science and Technology, 64 (11), 2224-2237 Document type article Files To receive the publication files, please send an e-mail request to TNO Library.