The use of mixed methods research in interdisciplinary educational journals

Abstract Mixed methods research is becoming increasingly important in several scientific areas. The analysis of prevalence rates is a new line of research that has emerged in mixed methods research, and this methodological approach has only been applied carefully in a handful of journals. The purpose of this article was to analyse the prevalence of mixed methods research in interdisciplinary educational journals. Moreover, the main characteristics of the mixed methods articles identified were examined. This study used a mixed methods approach to analyse these aspects. Specifically, a partially mixed sequential equal status multiple-case study design was applied with a development mixed methods purpose. Three educational journals in different disciplines were reviewed from 2005 to 2010 (Academy of Management Learning and Education, Educational Psychology Review, Journal of the Learning Sciences). The findings show differences among the journals in the prevalence rates and characteristics of the mixed methods studies.


ABSTRACT:
Mixed methods research is becoming increasingly important in several scientific areas.
The analysis of prevalence rates is a new line of research that has emerged in mixed methods research, and this methodological approach has only been applied carefully in a handful of journals. The purpose of this article was to analyse the prevalence of mixed methods research in interdisciplinary educational journals. Moreover, the main characteristics of the mixed methods articles identified were examined. This study used a mixed methods approach to analyse these aspects. Specifically, a partially mixed sequential equal status multiple-case study design was applied with a development

INTRODUCTION
Mixed methods research finds its roots in the 1960s as a concept of combining quantitative and qualitative research methods within the same study (Leech & Onwuegbuzie, 2009), and it has progressed in design and significance over the years (Creswell, 2003;Tashakkori & Teddlie, 1998. This methodological approach is built on the premise that it can be more fruitful to consider how the strengths of quantitative and qualitative approaches can be merged within a mixed methods research approach. The results obtained through the different methods combined can enrich and improve our understanding of the phenomena under study and foster fresh ideas about them, in order to give answers to questions that are difficult to answer by using a single method (Greene, Caracelli, & Graham, 1989;Tashakkori & Teddlie, 2003, 2010 In the first edition of Tashakkori and Teddlie's (2003) Handbook of Mixed Methods in Social and Behavioral Research, the applications and examples of mixed methods research across disciplines are related to diverse fields, such as education (Rocco et al., 2003). Since 2003, several reviews have been conducted in educational research to distinguish how mixed method research is conducted in different educational fields. In particular, Niglas (2004) examined the level of integration between qualitative and quantitative aspects and concluded that integration remains relatively modest in educational studies, especially at the stage of analysis. Leech, Collins, Jiao, and Onwuegbuzie (in press) found a low rate of mixed methods research studies (4%) in the field of gifted education. However, recent reviews in specific educational disciplines have presented evidence that reveal a high prevalence of mixed methods research studies. Collins, Onwuegbuzie, and Jiao (2007) pointed out that journals in the field of education published the highest proportion of mixed methods research (32%) among nine fields representing the social sciences. Hart, Smith, Swars, and Smith (2009)  In this context, the present study sought to analyse the prevalence of mixed methods research in three specific educational journals, and to document the utilization of mixed methods research by examining the purposes and specific designs of mixed methods research articles published. Three educational journals in three different disciplines (i.e., management education, educational psychology, science education) were examined: the Academy of Management Learning and Education (AMLE), the Educational Psychology Review (EPR), and the Journal of the Learning Sciences (JLS).
Therefore, this study sought to answer the following research questions: As Alise and Teddlie (2010) stated, the analysis of prevalence rates of methodological approaches is a new line of research that has emerged in mixed methods research over the past 5 years. In addition, mixed methods research has only been reviewed carefully in a handful of journals; consequently, an interesting topic in mixed methods research is the continued examination of how this methodological approach is being applied and the purpose and design characteristics that follow. The present research represented a mixed methods study: a quantitative approach was used to address the first research question, and a subsequent qualitative approach was used to address the second research question. Thus, a sequential mixed methods analysis (SMMA) was utilized (Onwuegbuzie & Teddlie, 2003;Tashakkori & Teddlie, 1998).
This article is structured as follows. The first section (Mixed Methods Research) describes several general aspects of the mixed methods approach, highlighting the main designs of mixed methods research and purposes for mixing. The following section (Method) describes the methods used in this study to identify the published studies that used mixed methods research techniques. The results section reports the main characteristics of these studies in each of the journals analysed. Finally, in the 6 Discussion Section, the results obtained are compared to other studies, and a number of recommendations are made for future research.
The way in which these two factors are combined helps to determine the resulting design. The notation proposed by Morse (1991) is useful and easy for representing the different possible designs. In her system, the abbreviations quan and qual are used to represent the quantitative and qualitative parts, respectively. When one method has greater weight than does the other, the former is shown in capitals letters (i.e., QUAN, QUAL), whereas the latter is written in lower case (i.e., quan, qual). Furthermore, the 7 symbol + is used to indicate a simultaneous design, whereas the arrow → refers to a sequential design. Therefore, the various combinations of data collection strategy and priority produce four blocks that give rise to nine mixed methods designs ( With regard to the purpose of conducting mixed methods research designs by integrating different types of data in the same study, several potential reasons have been noted by various authors (Creswell, 2003;Greene et al., 1989;Morgan, 1998;). Two of the most widely stated reasons are triangulation (i.e., to seek convergence in data); and complementarity (i.e., to measure facets overlapped from a phenomenon). The main aim of triangulation is to achieve a convergence of the results obtained via quantitative and qualitative approaches, such that these results are more reliable (Jick, 1979). What is sought, therefore, is a corroboration or correspondence of results obtained through different methods. According to Greene et al. (1989), complementarity seeks to clarify or to illustrate the results obtained with one method by also applying the other. In this case, the designs used are usually sequential (Johnson & Onwuegbuzie, 2004). For example, a QUAN→qual design could be used, whereby the qualitative part can help to evaluate and to interpret the results obtained from the main quantitative study. Another potential purpose of mixed methods research is development (i.e., the intent to help develop or to inform the other method). In this case, it is again usual to use sequential designs, in which one of the methods (normally the one with least weight) helps in some way to improve upon the subsequent implementation of the other method (normally the main or dominant one). A further purpose of mixed methods designs is to enable expansion (i.e., seeking to analyse and to explore different facets of a phenomenon so as to obtain a richer and more detailed understanding of it).
With respect to these main purposes, other authors have indicated a wider range of reasons. For example, Collins, Onwuegbuzie, and Sutton (2006)

METHOD
This study used a mixed methods research approach to analyse the prevalence and application of mixed methods research. Specifically, the mixed methods research purpose of our study was development, as previously described and also expanded below. Adapting Leech and Onwuegbuzie's (2009) three-dimensional typology of mixed methods research designs, our study was a partially mixed sequential equal status multiple-case study design: we gave equal importance to the quantitative and qualitative parts of our study, and the implementation was sequential. Therefore, our mixed methods study used a QUAN→QUAL design (Morse, 1991).
Articles representing the same time period (2005)(2006)(2007)(2008)(2009)(2010) were selected from three journals. andthe selection process was implemented using a sequential design, wherein the quantitative first phase informed the second qualitative phase to identify and to analyse the mixed methods research design used and the purpose(s) of each design. A content analyses (Krippendorf, 2003). was implemented to determine the prevalence of mixed methods research articles and the characteristics of designs that appeared during this period.
Our study was a partially mixed sequential equal status multiple-case study design (Leech & Onwuegbuzie, 2009) because a mixed methods research approach was not implemented across all components of the research process. In fact, quantitative and qualitative analyses were conducted separately, and integration of the analyses occurred at the conclusion of the study. In addition, the study was considered a multiple-case study following Yin's (1993Yin's ( , 2003 case study approach for involving the choice of three journals (each one representing a single-case study) and treated equally in the same multiple subunits of analysis (i.e., quantitative and qualitative) representing an embedded case study. In the final step, the three study-cases were merged into a larger unit of analysis for a cross-case comparison.
Specifically, in each single-case study (i.e., one journal), a descriptive research design was used for the quantitative phase to analyse the following subunits: (a) number of articles published per year; (b) number of articles that were non-empirical; and (c) number of empirical ones, specifically, quantitative, qualitative, and mixed methods.
A sequential mixed methods analysis (SMMA; Onwuegbuzie & Teddlie, 2003;Tashakkori & Teddlie, 1998) was undertaken to analyse the articles sequentially. The purpose of this QUAN→QUAL design was development (Greene et al., 1989), whereby the results from the first quantitative method informed the use of the other method.
Creswell and Plano Clark (2007) distinguished three purposes of mixing or integrating quantitative and qualitative methods. Using Creswell and Plano Clark's (2007) purpose's for mixing with respect to our case, the purpose for integrating was to have one dataset build on the other. Specifically, the results from the first quantitative method (identification of mixed methods studies) were used in the second qualitative method (analysis of the characteristics of the mixed methods research identified). Different search strategies might provide different results. Bryman (2006) pointed out that using an electronic search strategy can provide a biased sample of mixed methods studies because not all authors of articles reporting mixed methods research foreground the fact that the findings result from a combination of quantitative and qualitative research, or do not report key words that drove the online search strategy. As a result, the search strategy used to find mixed methods studies might influence the number of articles identified. For the qualitative phase of this study, all articles published in the three journals were read and reviewed, representing a search strategy that also has been used in some previous reviews (Hart et al., 2009;Niglas, 2004;Powell, Mihalas, Onwuegbuzie, Suldo, & Daley, 2008). Taking into account this important limitation of electronic search, we used the search strategy based on reviewing and reading all the articles published in each journal..

Quantitative findings
The AMLE is the highest ranking journal in the specific field of education in management with an impact factor for 2010 of 2.533 (Thompson Reuters, 2011). It should also be noted that, to date, no review has been conducted of the mixed methods studies published in this journal. Regarding the research methods described in this journal over a period of 6 years (2005 -2010), there was a predominance of empirical studies using a quantitative methodology. Qualitative research constituted a minority approach. Table 1 presents the distribution of articles by year and classified according to type of study. Because this journal is published four times a year, the present analysis covered 24 numbers (i.e., sections of a volume), in which a total of 98 articles were published. Table 1 It can be seen in Table 1, there is a clear predominance of empirical studies (75.5%). The majority of articles used a quantitative methodology (55.1% of all articles and 73% of empirical studies). Six mixed methods articles were published (6.1% of all articles and 8.1% of empirical articles).

Qualitative findings
Table 2 displays the design characteristics of the six mixed methods studies that were identified. As seen in Table 2, two of the six mixed methods studies gave equal weight to the quantitative and qualitative approaches, whereby the priority of the remaining four articles was given to the quantitative approach. With respect to data collection, one study included a simultaneous strategy and five studies represented a sequential approach. Of the six articles, analysis of priority and implementation in 11 combination revealed empirical studies in three of the four main designs. Specifically, two articles used equal weight and a sequential strategy (one QUAN→QUAL and one QUAL→QUAN), one article reported a study with different weight and simultaneous implementation (QUAN+qual), and three articles had a design based on different weight and sequential implementation (two qual→QUAN and one QUAN→qual). There were no designs involving equal weight and simultaneous data collection. Finally, the purposes for mixing in the six articles were: development (three studies), expansion (two studies), and complementarity (one study). Table 2 Educational Psychology Review

Quantitative findings
The EPR was selected due to it being one of the most widely recognised journals in the field of Educational Psychology at the international level. According to the Journal Citation Reports (JCR_ (SSCI), its impact factor for 2010 was 2.474 (Thompson Reuters, 2011). The journal focuses on psychology, child and school psychology, and educational research in the interdisciplinary area of educational psychology and includes both psychological and educational integrated studies.
Editorials, book reviews, interviews, comments or notes, in memoriam or biographies were omitted from the sample of articles because these items did not reflect original theoretical or empirical research.
Regarding the predominant research methods that appeared in this journal over the period of 6 years (2005)(2006)(2007)(2008)(2009)(2010), empirical studies were again the most common, with the majority of articles being qualitative, followed by quantitative and thereafter, mixed methods research being published. Table 3 presents the distribution of articles according to the same categories as were applied to the previous journal. In recent years, EPR has appeared with a 3-month period and, thus, for the period of years studied, a total of 24 numbers were analysed, in which 118 articles appeared. As can be seen in Table 3, theoretical articles accounted for the majority of articles, with an 80.5% of those published. This result might be contributed to the fact that the journal is a review journal. Two mixed methods articles were published (1.7% of the total articles and 8.7% of empirical articles). Table 3 Qualitative findings Table 4 presents the characteristics of the two mixed methods studies that were identified. With regard to priority, both mixed methods studies had different priority and equal implementation. In terms of design, the two studies used a sequential approach. Specifically, the two articles both utilized sequential implementation (QUAL→quan) with qualitative emphasis. Finally, regarding the purpose of the mixed methods studies, one article sought triangulation and the other article sought complementarity. Table 4 Journal of Learning Sciences

Quantitative findings
The JLS is one the most widely acknowledged journals in the fields of the Social Sciences, Psychology, and Educational Research. It features in the most important national and international scientific databases and the (JCR) for 2010 gave it an impact factor of 1.700 (Thompson Reuters, 2011). The JLS publishes multidisciplinary research on topics related to learning and education, with articles reporting on new methodologies that enable rigorous investigation of these topics. Findings revealed a predominance of empirical studies, which mainly used a qualitative methodology. Table   5 displays a clear trend in favour of empirical studies, specifically using a qualitative approach. A total of 88 articles was analysed, of which 84.1% articles were empirical studies. Articles using a qualitative methodology accounted for 43.2% of the total, and of those articles, 51.4% reported empirical research. Qualitative studies were followed by mixed methods research articles (22.7% of the total and 27% of the empirical research articles) and quantitative articles (18.2% of the total and 21.6% of the empirical ones), respectively. Table 5 Qualitative findings 13 The 20 articles that referred to a mixed methods tradition had the following methodological characteristics (Table 6). Regarding priority, 7 of the 20 articles gave equal weight to the quantitative and qualitative parts of the study, whereas the remaining 13 of the 20 articles prioritised one over the other (the predominant method being a quantitative approach). With respect to the implementation of data collection, two studies revealed a simultaneous collection strategy, whereby the remaining 18 used a sequential collection strategy. The analysis of how priority and implementation were combined revealed two studies whose design utilized different weight and simultaneous implementation (QUAN+qual and QUAL+quan), seven designs were based on equal weight and sequential data collection (five QUAL→QUAN, one QUAN→QUAL and the one quan→qual), and 11 were based on different weight (once again in favour of the quantitative approach) and sequential implementation (seven QUAN→qual, one quan→QUAL and three QUAL→quan). Finally, two main purposes were identified: complementarity was identified in 19 articles, and triangulation was identified in the remaining article. Table 6 DISCUSSION AND CONCLUSIONS

Comparison of the journals
The comparison of results is presented in Table 7 and reflects a cross-case conclusion of this mixed methods research study. This table reveals a predominance of empirical studies (56.3%) over theoretical articles (43.6%) due to the prevalence of empirical research published in EPR. The JLS published the majority of empirical articles, followed by the AMLE and the EPR, respectively. It also can be seen that the majority of articles over the 6-year period were quantitative in nature (25.6%), followed very closely by qualitative studies (21.4%) and mixed methods studies (9.2%). This comparison confirms that in recent years, these three multidisciplinary education journals published empirical studies equally using a quantitative or qualitative methodology, followed by mixed methods research articles, which appear most frequently from 2007-2008 henceforth. In any case, the prevalence rates of mixed 14 methods studies in these three journals are very different (0.2% in EPR, 6.1% in AMLE, and 22.7% in JLS). Table 7 With respect to the studies that used both quantitative and qualitative methods, Table 8 presents the characteristics of each mixed methods design, as well as each stated methodological purpose.
In summary, it appears that mixed methods research is still in the development stages with respect to these educational-relevant journals. Furthermore, when a mixed method approach was used, the research design was predominantly a sequential different-weight mixed methods research design. Table 8 This mixed methods multiple-case study research sought to describe each singlecase study first quantitatively and then qualitatively, thereby allowing us to compare among cases in relation to their quantitative subunits of analysis (i.e., number of non empirical and empirical studies: quantitative, qualitative and mixed), and within cases of each journal in relation to its qualitative subunits of analysis (i.e., interpreting the characteristics of mixed methods design in research articles published). The emergent findings enable the overall study to be more robust than a single-case study design.
Specifically, by following a replication logic similar to multiple experiments (Yin, 2003), the case-to-case analyses operate as a case-to-case generalization, thereby contributing to an accumulation of evidence representing the population of articles published in reputable behavioural sciences journals (following Onwuegbuzie, 2003).
Three literal replications were made because the journals selected had the following similar characteristics: interdisciplinary educational field of knowledge, international recognition, peer-reviewed, and with a reputation established by their impact factor index.
It should be noted that only one of the mixed methods studies identified in the journals, concretely in the JLS (Enyedy & Mukhopadhyay, 2007), cited the use of mixed methods in the abstract without including any bibliographic reference. This 15 finding suggests that the mixed methods approach still is not familiar to researchers in these interdisciplinary educational disciplines.

Limitations
The present study was subject to limitations that might have included: (a) descriptive credibility (i.e., the factual accuracy of the accounts; Maxwell, 1992), (b) interpretive validity (i.e., the extent to which the interpretation of the analysis represents an understanding of the phenomenon; Maxwell, 1992), and (c) voluptuous legitimation (i.e., the extent of the researcher's level of interpretation with respect to knowledge based on data; Lather, 1993). According to Onwuegbuzie and Leech (2007), these threats to internal and external validity can occur at one or more of the following three stages of the research process: the research design/data collection stage, the data analysis stage, and the data interpretation stage. Therefore, validity could have been compromised.

Implications regarding the application and publication of mixed methods research
Mixed methods studies require more time, work, effort, and resources than do studies that use only a single method due to increased time demands arising from the time it takes to implement both aspects of the study (Niglas, 2004). Therefore, this might explain the publication of fewer mixed methods articles than monomethod articles. As a result, considerable attention should be paid to promoting the understanding of mixed methods design characteristics by academic institutions when making evaluation, promotion, and tenure decisions.
Another important barrier to carrying out mixed methods research is related to the challenges of publishing mixed methods studies and practical constraints such as page limits in journals. By limiting space, journals can discourage publication of mixed methods research. One of the biggest challenges related to publishing mixed methods research is describing the complexity of mixed methods studies within the page limits.
Although such limits pose a challenge to all researchers, they are particularly problematic for mixed methods researchers due to the quantity of information that must be conveyed for a study combining two different methods in detail so that reviewers and readers can understand and replicate the methods used. Moreover, there is a risk of diluting or diffusing one of the methods (the less important one or the one less accepted by academia) by attempting to do too much within the page limit. In summary, by limiting space, journal editors discourage the publication of mixed methods research. Therefore, to encourage mixed methods research, journal editors should be willing to publish long articles.

The quality of this mixed methods study
As noted previously, our study was mixed methods in nature, following the guidelines for Good Reporting of A Mixed Methods Study (GRAMMS) provided by O'Cathain et al. (2008). First, we have described the justification for using a mixed methods approach toward investigating our research questions by noting the quantitative phase to answer our first research question, and subsequent qualitative phase to answer our second research question. As noted in the introductory section, these two questions are relevant in the field of mixed methods research. Second, we have described the design of our study in terms of the purpose (development), priority (equal importance of the quantitative and qualitative parts), and sequence of methods (sequential; QUAN→QUAL). Third, we have described each method in terms of sampling, data collection, and analysis.
Fourth, we have examined the issue of integration, indicating that the mode of integration is connecting the two datasets by having one build on the other. The results from the first quantitative method (identification of mixed methods studies) were used in the second qualitative method (analysis of the characteristics of the mixed methods identified). Fifth, we have not identified any limitation of one method associated with the presence of the other method. Finally, we have described insights gained from mixing or integrating methods. We suggest the use of GRAMMS for authors of mixed method research to further the understanding of design characteristics.

Recommendations and future research
Conducting mixed methods research in the interdisciplinary educational sciences could increase understanding of certain aspects of studies that have already been carried out in this field. The prevalence rate of articles in our study is similar to the results of Truscott et al. (2010), who also reviewed articles across different educational disciplines. The remainder of the educational reviews achieved relatively high prevalence rates possibly because they covered greater periods of time focused in disciplinary educational journals (Alise & Teddlie, 2007;Collins, et al., 2007;Hart et al., 2009;Ross & Onwuegbuzie, 2010), although one study documented a low prevalence rate possibly because it involved the analysis of a specific educational discipline (i.e., gifted education; Leech, et al., in press). Reviews of articles published with respect to mixed methods design in this and other fields should be conducted. In this regard, Creswell, Plano Clark, Gutmann, and Hanson (2003) point out that a central challenge for mixed methods research is the explicit clarification of several key aspects, such as: (a) to identify the main purposes of using a mixed design, (b) to clarify the factors analysed when determining the type of mixed design, and (c) to describe the decisions made when assigning the respective weight (equal or different) to each methodological part of the research. In relation to the implementation of data collection, researchers should specify whether the mixed design is sequential or simultaneous.
In our opinion, it would be interesting to conduct further reviews covering both a broader time period in other educational journals, so as to obtain a more detailed picture of how mixed methods design is being applied in the education field. In sum, researchers need to be aware of the methodological designs available to apply in their respective disciplines in order to have the opportunity to publish findings using the most suitable methodological approach, including mixed methods research designs.    The numerical values inside the boxes are absolute frequencies and the values in parentheses are the percentage derived from the total in each case.