Quantitative and qualitative evaluation of the COMPASS mobile app: a citizen science project

Abstract

To evaluate the usability of the COMPASS application with mixed-methodology, using a citizen science approach. Parents/tutors of 10-11 years old children attending a primary school in Barcelona, Spain, were invited to take part in the study. We conducted semi-structured interviews on a subset (n = 7) of participants, two weeks after using the app for the first time. A list of suggestions of improvement was extracted from the interviews. The System Usability Scale (SUS, range 0-100) was administered to all participants before and after the improvements were implemented. We provide both a quantitative analysis (t-test of change in SUS scores) and a qualitative thematic analysis of the interviews. A total of 22 participants were included in the study. The mean score before implementation of changes was 68.5 (Standard deviation, SD = 11.1), and improved to 73.1 (10.5) (p-value = 0.025). Regarding the qualitative assessment, we obtained 24 codes and grouped them into 3 categories. It uncovered problems in the installation phase and the main barriers to use: lack of time and the need for the app to evolve. The new version of COMPASS, improved by taking into account the participants' comments and suggestions, was more usable than the initial version.