Quality indicators: developing “MOOCs” in the European Higher Education Area

Abstract The objective of this research is to design an integrated system of evaluation of the quality of the Massive Open Online Courses (MOOC) taking into account the features of this type of courses. The criteria included in the evaluation of the quality of e-learning training in a broad sense are considered, and specifically those developed for the MOOCs. Using a quantitative, descriptive and sectional design, a team of experts have valued those indicators proposed by the scientific literature. To do this, eight courses, focusing on the field of entrepreneurship and innovation, were selected. Most of the analysed courses meet designated quality criteria by having a clear and organized academic structure. In addition, each course states clearly the timing and hours of dedication by the learner. However, only a few courses do that on the open Internet and on their own academic institutions’ websites. None of the MOOCs establishes the admission requirements. The main contribution of the current research is that it gives MOOC developers some quality indicators, related to course design and platform, to better plan their applied design and implementation, which has a decisive positive impact on the educational quality of such e-learning platforms.


Introduction
The implementation of the European Higher Education Area (EHEA) has entailed that training programmes taught by the Spanish universities must be evaluated on the basis of a series of standards and quality criteria. Such External Quality Assessment (EQA) System can be described as a methodical, structured and continuous focus on the support and improvement of the quality of education (Vroeijenstijn, 1995).
Among the assessment standards and criteria applied by those agencies belonging to the Red Española de Agencias de Calidad Universitaria (Spanish Network of University Quality Assurance Agencies) (Red Española de Agencias de Calidad Universitaria [REACU], 2011), which derive from the standards and guidelines for quality assurance in the European Education Area (European Standards and Guidelines, ESG) proposed by the European Association for Quality Assurance in Higher Education (European Association for Quality Assurance [ENQA], 2009), we must underline those that evaluate the quality of the training programme, the relevance of public information and transparency of information regarding the programme, as well as the effectiveness of the aid to learning systems, which involves services of guidance and adequate and effective resources for the learning of students.
The exposed criteria are transversal to any training methodology, both on-campus and e-learning. But, within online methodology, different teaching-learning processes mediated by new technologies can be named, being the so-called Massive Open Online Courses (MOOC) a new approach in the field of education which "has burst rapidly in education" (Roig Vila, Mengual Andrés, & Suárez Guerrero, 2014, p. 27).
It is our belief that generic criteria of e-learning environment evaluation are being used to assess the quality of the MOOCs, although, as stated by Raposo-Rivas, Martínez Figueira, and Sarmiento (2015), there is a lack of unification of criteria. This fact involves, on the one hand, a difficulty in evaluating systematically the quality of these systems of teaching and learning, and on the other hand, the absence of specific educational assessment criteria adapted to the features of a MOOC.
In general terms, among the criteria included in the evaluation of the quality of e-learning training, researchers agree to emphasize that it is essential that students be informed on methodological procedures to be followed so that they may acquire the established academic competencies. This same criterion is also taken into account in relation to the quality of the MOOCs (Castaño, Maíz, & Garay, 2015;CRUE, 2015). Other research works also support, together with a clear statement of the process of learning and gaining skills, a necessary and a clear presentation of the course objectives, as well as an internal coherence between activities, contents and objectives (Arias, 2007;CRUE, 2015;Pavón, Pérez, & Varela, 2000) to prevent students from feeling confused or disoriented in the absence of a teacher in a physical classroom. Majó and Marqués (2002) point out that all teaching-learning activities must be explained to the students, indicating the form of clustering and the methodology to be used. (…) the description of the instruments to be used to determine to what extent the students have achieved expected learning outcomes. To indicate what is going to be evaluated, in what way and when (Majó & Marqués, 2002, p. 281).
A second-quality criterion observed by various authors is the proper selection and organization of its contents (Castaño et al., 2015;Pavón et al., 2000;Santoveña, 2005), as well as the distinction between basic and supplementary information (Arias, 2007).
One of the available strategies to organize the content consists of its hierarchical organization based on descriptive titles and subtitles (Marqués, 2000), structuring information using short paragraphs to facilitate reading and, finally, linking related concepts (Majó & Marqués, 2002, p. 170). On the other hand, and in reference to the contents, some aspects are relevant, such as the quantity and quality of the bibliographical selection (Alemán, Sancho-Vinuesa, & Gómez, 2015;Arias, 2007;CRUE, 2015;Santoveña, 2005).
A third element to ensure quality is related to the existence of an educational guide that may help students to know in which course stage they are, their progress and tasks still to be completed (CRUE, 2015;Sánchez, león, & davis, 2015;Norm UNE 66181:2012). An example is the existence of tools such as navigation maps or frames with indexes (Marqués, 2000).
Additionally, it is interesting for students to have the feasibility to adjust contents according to a scale of difficulty (CRUE, 2015;Marqués, 2000), with different itineraries and even revision activities (Arias, 2007;CRUE, 2015) so that they can respond to different initial knowledge settings, taking into account personal characteristics of each learner (Majó & Marqués, 2002). In this regard, different itineraries allow students to be placed in a certain level of difficulty, whether using initial testing or their own personal considerations (Norm UNE 66181:2012;Alemán et al., 2015).
A fifth element, according to Alemán et al. (2015), is to use different pedagogical resources. It is also important to be careful with the delivery of the content through teaching resources (presentation, overview, diagrams, synthesis of the syllabus, etc.) (Arias, 2007) both to help to introduce themes of the course and to ensure a proper follow-up (Marqués, 2000).
With respect to the diversity of resources, Castaño et al. (2015Castaño et al. ( ) (2015 bring up the fact that the use of varied resources helps to focus attention on the course. Such variety refers to materials in video and audio formats, but also computer applications and animated graphics (CRUE, 2015;Sánchez et al., 2015). In fact, as noted by Santoveña (2005) and CRUE (2015), it is important to show various multimedia resources in an integrated fashion and to combine different types of information. This variability is considered fundamental in what refers to additional resources, namely that the MOOC platform can grant access to social networks (twitter, Facebook and linkedin, etc.) (Sánchez et al., 2015), even Skype or blogs, because in this way it promotes interaction between participants (Castaño et al., 2015), while providing supplementary material (CRUE, 2015;Majó & Marqués, 2002) such as videos produced outside the MOOC platform, external academic articles or other external links pointing to additional enriching information.
Finally, in relation to teaching resources, the implementation of a support and guidance system is necessary (CRUE, 2015;Majó & Marqués, 2002) in order to solve doubts arising over the navigation, as well as activities for educative reinforcement. The latter, given the large number of participants and high rate of abandonment in the MOOCs, must be able to meet a wide and diverse audience of students (Castaño et al., 2015) as well as enable to overcome any potential deficiencies that could have been detected in the evaluation (Arias, 2007).
The sixth criterion of quality focuses on the course as a facilitator of the student's motivation (Alemán et al., 2015). Motivation is the force that starts and maintains a behaviour and ultimately gives results (Álvarez-Álvarez, 2005). The high rate of abandonment of the MOOCs (Jordan, 2014) makes it necessary to reflect on those strategies and conditions that facilitate the maintenance of motivation throughout the formative process in students. One of the ways to sustain this motivation is to provide reinforcement to students, such as delivery of awards, which enhances the positive attitudes towards the study and keeps interest in the ongoing course (CRUE, 2015;Santoveña, 2005). In addition, motivation can be given through attractive design of the contents of the course (Arias, 2007;CRUE, 2015), since, as Castaño et al. (2015) advocate, there is a direct link between a course attractively designed and the motivation of its participants.
The seventh criterion to be considered is the technical and aesthetic quality of the course (liu et al., 2015;Majó & Marqués, 2002), a good quality in images and audio (Arias, 2007;CRUE, 2015), and that the images presented should always be linked to a textual alternative and vice versa (Majó & Marqués, 2002). In short, there must be a clear coherence between the images, graphics, audio and text (Arias, 2007;Santoveña, 2005).
The eighth criterion refers to temporary aspects. The MOOCs, as well as other nonpresential types of formation, introduce a temporary configuration that facilitates learning with customizable schedules and autonomous rhythms (liu et al., 2015). Therefore, a clear time specification is necessary (Alemán et al., 2015;CRUE, 2015), as it allows the user to organize their time flexibly, with organizational formulas consistent with the amount of work required (CRUE, 2015;Pavón et al., 2000).
linguistic correction is valuable as the ninth criterion of quality, which is accuracy in spelling, grammar and syntax, as well as the adaptation of the register to the level of the potential user (Arias, 2007;Majó & Marqués, 2002). This means that the contents have to be structured by "a clear language to avoid idiomatic expressions" (Sánchez et al., 2015, p. 41).
Finally, the last two dimensions identified by various authors refer to the adaptation of the syllabus to the users' profile (Alemán et al., 2015;Arias, 2007;CRUE, 2015;Majó & Marqués, 2002), as well as the educational values that emerge throughout the training process (Arias, 2007;Majó & Marqués, 2002;Marqués, 2000). It is of utmost importance that the contents of the interaction messages with the user are positive (Arias, 2007), as well as the reinforcement of tutorised activities been varied, not threatening, appropriate for the students and the specific situations (Marqués, 2000). In addition, as stated by Majó and Marqués (2002) and Marqués (2000), the contents and messages must not be negative or biased and will not make discrimination by reason of gender, social class, race, religion or beliefs. Ultimately, texts and contents should respect the principle of non-discrimination on grounds of birth, race, sex, religion, opinion or any other personal or social circumstances (Arias, 2007).
The above-mentioned criteria of evaluation of the quality of MOOCs should be completed according to the quality characteristics of the technological platform. In this regard, its visual and structural design should be taken into account in terms of aesthetic quality of the sections (icons, menus, and forms, etc.) (liu et al., 2015;Marqués, 2000;Santoveña, 2005), the position of the use in navigation (Alemán et al., 2015;Majó & Marqués, 2002) and the communicative resources included (chats, news, forums, and email, etc.) (Majó & Marqués, 2002;Marqués, 2000).
Following the analysis, the research group considered it relevant to add four more dimensions to the 12 already presented. The first two are linked to the quality of the course and the other two deals with the platform.
In terms of quality, the authors consider of importance to add a dimension concerning the promotion of the course and its price. The promotion includes indicators about the dissemination of the course through various platforms, offline resources (flyers, posters, etc.), the participation of opinion leaders to act as guarantors of the course and the specification of the entity responsible for the course and its certification.
The price refers to initial specification of the linked economic cost. The importance of these two dimensions is that they assume an operationalization of the criterion established by the European Association for Quality Assurance (European Association for Quality Assurance [ENQA], 2009) concerning the relevance of public information and transparency of information regarding the programme.
As for the dimensions for evaluating the quality of the platform, it is proposed to include the level that allows modification of the language and their compatibility with operating systems (Android, linux, Apple iOS, and Microsoft Windows, etc.) and different media (PC, Tablet, and Mobile, etc.). Alemán et al. (2015) raised the importance of the technological platform's versatility, although they do not establish the indicators that should be included.
Recently, some authors have developed tools to analyse the quality of the MOOCs, but they have focused on quality regulations based on the UNE 66181 (Baldomero & Salmerón, 2015). However, from the analysis done in the previous section, the general objective of this research is the design of an integrated evaluation system of the quality of the MOOCs, which considers the characteristics of this type of courses.
Several studies from the fields of Humanities and Social Sciences noted the existence of gaps between the training curriculum provided for the students and what the market demands Romeo & Yepes, 2008;Romeo, Yepes, & Carro, 2010;Romeo et al., 2012Romeo et al., , 2013. In this sense, one of the main needs of new graduates in these fields is to find niches of employment in a highly competitive market, characterized by continuous changes in different levels, resulting in the need to develop entrepreneurial skills. That is the reason for the current analysis, from the integrated evaluation system of quality, of various MOOCs whose themes belong to the field of Humanities and Social Sciences, and focus on the field of entrepreneurship and innovation.

Method
This research is based on a quantitative, descriptive and sectional design. The assessment instrument is configured considering 48 items, generated theoretically from the analysis of the main works in relation to the quality of the MOOCs and other proposals for e-learning training, and the most commonly included dimensions in those researches presented in the previous section (Tables 1 and 2).
The indicators are structured into two categories, depending on whether they refer to the course itself (Table 1) or to the platform (Table 2). In terms of the indicators related to the course, they have been structured in 14 dimensions concerning Methodology, Content organization, Teaching guide, Content quality, Teaching resources, Motivation, Technical quality, Chronological aspects, language, Interaction, Users' individualization and uniqueness, Values, dissemination and promotion and Price (Table 3). Furthermore, the assessment of the platform includes visual and structural design, base language, compatibility and communication resources (Table 4). In the majority of cases, a five-point likert scale (1-nothing, 5-very) was used except for the items of the dimensions dissemination-Promotion and Price, in this case a dichotomous scale was chosen.
The coders were enrolled on the platforms to rate the quality of the courses by the indicators and dimensions above mentioned. They were intentionally selected (Bisquerra, 2004), ensuring that all of them were experts in Communication and had previous experience in MOOCs.
The MOOC courses included in the analysis were those belonging to the field of the social sciences and focus, especially, on entrepreneurship and innovation. The courses were selected on the basis of the access availability during the time of the study (december 2014-January 2015) and according to higher quality educational platforms defined by Roig Vila et al. (2014) (except Udacity and OpenHPI, including courses exclusively related to information technology and computer science). In addition, it was considered relevant to add UnX (Colmenia, 2014a), the first Ibero-American community of digital entrepreneurship. different platforms were used by accessing the search engine to select the most optimal categories: social sciences and information, technology and design, in the case of Coursera (Coursera, 2015a), and pedagogy and science of arts and letters, in MiriadaX (MiríadaX, 2014a). The catalogue of courses UniMOOC (UniMOOC, 2014a) and UnX is not so broad as to differentiate categories. Besides, those two platforms are very focused on entrepreneurship. Keywords as "entrepr*" and "innova*", both in Spanish and in English, were also introduced to access courses in both languages. On the other hand, the web MOOC list (2014) (<https://www.mooc-list.com>)     (2000); Majó and Marqués (2002) was used in which, by selecting the category, the period required and entering keywords, retrieved a list of all the courses available at the time. In accordance with the schedule of the analysis, it was necessary for courses to be available from december 2014 to January 2015.
In total, eight courses met the aforementioned criteria: four, from the platform UniMOOC; two, from Coursera; one, from MiriadaX, and one, from UnX. UniMOOC is a platform aimed at training entrepreneurs for digital economy. The entity that guarantees these courses from Existence of systems of reinforcement/reward ("badges") Presentation of content in an engaging way technical quality Quality images and charts included in its different formats Quality audio messages consistency between images, graphics, audio and text chronological aspects Specification of the estimated time of dedication to the course Specification of the timing of activities to be carried out coherence and adequacy of the timing language correct spelling and syntax adapted language to the level of the user or the target interaction Existence of systems that facilitate access to and contact with teachers collaborative learning tools users' individualization and uniqueness the system attends to the functional peculiarities of the users the system allows the users to organize their time in flexible hours Existence of asynchronous information repositories Values Emission of positive messages to users texts and the contents are in accordance with the principle of nondiscrimination on grounds of birth, race, sex, religion, opinion or any other personal or social circumstances dissemination and promotion dissemination of the course through different platforms other than the one in which it was obtained Yes-no dissemination of the course through resources offline (means of communication, flyers, posters, brochures, etc.) Specification of the entity responsible for the course and its certification use of opinion leaders such as former students or professors of prestige as endorsers of the course Price Specification of the course cost services from the beginning UniMOOC is the Institute of International Economics of the University of Alicante which, in turn, is the driving force behind the project. Coursera is associated with most renowned organizations and universities in the world. It holds more than 100 bodies from 25 different countries. The courses discussed in this platform are supported by the University of Virginia, and by the National Autonomous University of Mexico.
The MiriadaX platform is focused on the Ibero-American context of higher education. In this case, the studied course is certified by the Centro de Estudios Universitarios (CEU) Cardenal Herrera of Valencia.
Finally, UnX also focused on the Ibero-American context, aims at the digital entrepreneurship field: in this case, Fundación Centro Superior para la Enseñanza Virtual (CESV), Telefónica, Universidad Nacional de Educación a distancia (UNEd) and the Massachusetts Institute of Technology (MIT) are collaborating in the course included in this research.
Seventy-five per cent out of the analysed courses is related to entrepreneurship and the sectors of the future; 12.5% is about the field of communication, while 12.5% regards courses linked indirectly to entrepreneurship. The average duration of studies courses is 12 h (Sd = 8.77), with a range between 4 and 24 h.
For the quantitative analysis, a sheet of each of the MOOCs with five transversal elements has been prepared. These include: the title of the course, the platform, the entity responsible, the number of modules for each MOOC, as well as the access link. Subsequently, the research team evaluated the different MOOCs depending on the indicators listed in Tables 3 and 4.

Results
Quantitative analysis of the quality of platforms shows that the Coursera platform achieves significantly higher scores both in regard to the possibility of incorporating different languages, and to the compatibility and communication resources. It is noteworthy that, while all platforms are compatible with different operating systems, only Coursera and UniMOOC use Responsive Web design to adapt themselves to different devices (Figure 1).
The quality of the course itself, and in general evaluations of the MOOC, describe to a greater or lesser extent the fact that the score is higher than average quality with a difference of 0.81 among the best (3.89) and the worst score (3.08) ( Table 5).
The majority of the courses have a clear and organized structure of their content. They are always divided into modules which, in turn, are divided into units of content. In all cases, the objectives of the course are defined and clearly specified: in some cases, they are at the beginning of the course; in others, the objectives of each module are presented separately.
In general terms, all courses stand out positively in regard to chronological aspects. Most clearly specify their timing and hours of dedication required. In addition, flexibility of timetables is suitable for all students. Of eight courses analysed, only one (M04) showed deadline for evaluation activities. However, once deadline is reached, teaching resources are still available to students.
Though the courses meet quality criteria, there are some items that stand out negatively. The most notorious case deals with the lack of access to teaching staff. Almost all MOOCs have a forum for discussion and questions in which the students participate. However, it is not specified if the teacher has access to the forum or will participate in it. Thus, in most courses, students are the ones to help each other -notably in the case of M04 on the MiriadaX platform and M07 on Coursera.
On the other hand, speaking on the promotion, only a few courses make visible propaganda, and not only on other Internet channels, such as websites or blogs: advertising on courses in the websites of the responsible institutions is also low. However, most of the courses appear on other blogs, as recommended choices or course evaluation. In addition, Figure 1. average quality of the tested platforms. it also favours promotion the fact that in some MOOCs interaction among students is endorsed through microblogging tools using hashtags. It is worth mentioning that it has been unable to evaluate dissemination concerning indicator offline due to the complexity of the scope and coverage of this medium. The differentiation of levels of deepening is other general lack in all courses. Although some specify the requirements for the course, no event sets the level of specialization of the MOOC; that is, if the training content requires a base upon theoretical-practical or not. In addition, groups of students are not generated according to their previous knowledge.
Each of the courses specific analysis shows that M03 and M04 clearly differ from the remaining in terms of their bibliography. Both MOOCs provide a list of books and resources to support and complement the content given in the module, while in the other courses this support is virtually non-existent. On the contrary, the use of a repository of information is an example of good practice in all cases except for M07, which does not have it.
It should be added that, although most of them try to foster the creativity of the user, M06 is the one which uses innovative resources. Through mysteries and attractive challenges included in the videos, along with discussion forums, it is able to attract the attention of the student to thinking, conceiving and researching in order to find the answer.
The videos used in the different modules of the majority of courses have no common clear stylistic guideline. In some cases there is a noticeable difference, already structural and qualitative, between parts of a same course (Table 6) Related to dissemination and promotion dimension (Table 7), in the majority of the MOOC's dissemination is made through different platforms, the entity responsible for the course and its certification is included, and opinion leaders such as former students or professors of prestige as endorsers of the course are used. We could not found information about dissemination of the course through resources offline (means of communication, flyers, and posters, etc.).  Finally, the analysis shows that all MOOCs are free, so obtaining the recognition and/or badges does not imply an extra cost: the idea of open course is, then, fulfilled.

Discussion and conclusions
We focused this research on the design of an integrated evaluation system of the quality of the MOOCs. Recently, some authors have developed tools to analyse this quality but they have focused on quality regulations based on the UNE 66181 (Baldomero & Salmerón, 2015). Our work goes further, since it not only includes regulatory issues but gives MOOC developers quality indicators to plan their design and development, influencing in a very direct way in their pedagogical quality (Glance, Forsey, & Riley, 2013;Roig Vila et al. 2014).
Agreeing with the results of Baldomero and Salmerón (2015), a platform with high quality is Coursera, being MiriadaX and UNX the ones with the lowest scores. UniMOOC gets the highest level of quality, although this includes only courses with a clear focus on training for entrepreneurs in the digital economy.
All the analysed courses have scores reaching levels from medium to high quality. Nevertheless, there are some items that stand out negatively. From a pedagogical point of view, the most notorious case deals with the lack of access to teaching staff, which in most courses is replaced by the collaborative work between peers. The differentiation of levels of deepening is other general lack. This question raises the difficulty of adaptation to individual characteristics of students in a context of mass access to the courses (Valverde, 2014).
From a technical point of view, the videos used in the majority of courses have no clear stylistic guideline. Additionally, only a few courses make visible external advertising in the websites of the responsible institutions, which could indicate that the sense of identity and corporate image of the entire course is lost.
It is also important to note that both pedagogical and technical aspects as well as the specifics of the platform must be addressed systemically when designing and planning the implementation of a MOOC. This article is based on the evaluation of MOOCs focused on entrepreneurship and innovation in the field of social sciences. It is necessary to test the proposed evaluation system in courses that deal with other topics within the scope of the social sciences, in order to establish the degree of consistency of the indicators. In further work, it will be important to analyse the degree of transferability to other areas of knowledge. Finally, it should be necessary to add the evaluation of students and managers of the courses in order to analyse the effect of the quality indicators in their results, evaluated both from their perceptions as from objective criteria (dropout rates, hours of dedication, etc.).
Summarizing, we agree with the SCOPEO Report (2013) to point out lines that are required to make the MOOC successful: the MOOCs will have future if they meet the objectives given in these three areas: teaching, if the student believes that with this methodology learns and is formed; business, if companies value that employees have been formed with this type of learning; and institutional, if universities, companies and platforms that manage the MOOCs get results in terms of positioning against the competition, of attraction to new customers-students and if the relationship investment vs result is satisfactory (SCOPEO, 2013, p. 47). Therefore, the MOOCs need indicators of quality as they "go away a fad and they become a consolidated model, sustainable and with appearance of permanence" (SCOPEO, 2013, p. 85).