Evaluation of a speech-driven telephone information serviceusing the PARADISE framework: A closer look at subjectivemeasures

Evaluation of a speech-driven telephone information serviceusing the PARADISE framework: A closer look at subjectivemeasures

Smeele, P.M.T.
Waals, J.A.J.S.
TNO Technische Menskunde

2003

For the evaluation of a speech-driven telephone flight information service we applied the PARADISE model developed by Walker and colleagues [1] in order to gain insight into the factors affecting the user satisfaction of this service. We conducted an experiment in which participants were asked to call the service and book a flight. During the telephone conversations quantitative measures (e.g. total elapsed time, the number of system errors) were logged. After completion of the telephone calls, the participants judged some quality related aspects such as dialogue presentation and accessability of the system. These subjective measures together represent a value for user satisfaction. Using multivariate linear regression, it was possible to derive a performance function with user satisfaction as the dependent variable and a combination of objective measures as independent variables. The results of the regression analysis also indicated that an extended definition of user satisfaction including a subjective measure ‘Grade’ provides a better prediction than the analysis based on the narrow definition used by Walker et al. Further, we investigated the correlation between the subjective measures by conducting a principal components analysis. The results showed that these measures fell into two groups. Implications are discussed.

Informatics

http://resolver.tudelft.nl/uuid:0e351ced-bf7f-41d6-b32b-bc4a39f0ef4b

12228

Eurospeech, 1949-1952

conference paper