Volume index - Journal index - Article index - Map ---- Back
MOOC, indexations, bibliometrics, scientific journals, network analysis, higher education, JCR, Scopus
Massive open online courses (MOOCs) have been considered as a revolution in the divulgative and scientific literature, with a great incidence on educational and formative context (Martin, 2012; Cooper & Sahami, 2013; Aguaded, Vázquez-Cano & Sevillano, 2013; Vázquez-Cano, López-Meneses & Sarasola, 2013; Yuan & Powell, 2013; Downes, 2013). The latest Horizon Report (Johnson et al., 2013) provides a prospective study of the use of educational technologies and future trends in various countries and especially highlights the impact of MOOCs in today’s educational context. Moreover, the Ibero-American Edition oriented to higher education believes that the «massive open courses» will be implemented in institutions of Higher Education within the next four to five years (Durall & al., 2012).
MOOCs have attracted a worldwide interest because of their potential to offer free training accessible to anyone regardless of their country of origin, a previous training without the need to pay for tuition (Vázquez-Cano & al., 2013). Since early 2010, the emergence of these courses has begun to be viewed from a more academic perspective when different prestigious universities began their mass activities; among others: Stanford, Harvard, MIT, and the University of Toronto. There is consensus in the scientific community about the importance and popularity of the movement, mainly by its international scope and the opportunity to offer a diversified Higher Education through prestigious institutions, which even recently was only possible for a small group of people. At the same time, there are discrepancies and doubts about the pedagogical value and future of the MOOC movement in Higher education. The scientific community focus on its impact on the educational and social context from different positions; some of them consider it a destructive development (Touve, 2012), while others see it as a deeply renewing and creative movement (Downes, 2013).
The last two years have seen a peak of over sizing with a high impact and widespread dissemination on media and networks. A Google search on the term MOOC produces more than three million results, whereas a search for more established terms from the scientific literature such as «e-learning» or «mobile learning», generates less than half the results. This gives us an idea of what might be called a «disruptive» event.
In this paper, we analyze the MOOC‘ scientific impact in two of the most prestigious scientific databases WOS (Journal Citation Reports) and Scimago (Scopus) to focus on the main implications for future research and the most significant global bibliometric data, with special emphasis on articles, authors, institutions, and the more representative semantic fields according to citations and database impact. Thus, we can also determine the impact on the scientific world and if the results in this area may also be considered «disruptive».
Arguably, David Wiley (professor at the State University of Utah, United States), with his open education course offered in 2007, created the first MOOC in history. Subsequently, in 2008, George Siemens and Stephen Downes designed the course that is considered the genesis of the MOOC movement «Connectivism and Connective Knowledge (CCK08)». This event, along with the landmark in the autumn of 2011, when 160,000 people were enrolled in a course on artificial intelligence offered by Sebastian Thrun and Peter Norvig at Stanford University through a startup company called «Know Labs» (now Udacity), converted to an MOOC movement, meant a turning point for the academic and scientific community.
From these events, many teachers, institutions, and universities have started to develop plenty of open courses, multiplying exponentially their impact on the learning processes of Higher Education. The academic and scientific world have analyzed the benefits of this training model in numerous publications, mainly in divulgative journals, scattered in blogs, wikis, journals, social networks, and so on. A sign of this is the prolific activity of researchers such as Stephen Downes with a continuous process of open publication (www.downes.ca), Sir John Daniel with his thoughts and research on quality assessment (Daniel, 2012) and George Siemens with his approaches to the movement from the connectivism principles (Siemens, 2013), among many others. The publishing phenomenon in this movement has followed a similar pattern to other «disruptive innovations». For example, the Twitter microblogging phenomenon that first appeared in 2006, only produced three articles until 2007, but in 2011, there were hundreds (Williams, Terras & Warwick, 2013). The MOOC phenomenon presents an opportunity for emerging research in the coming years in three priority research areas: technology architecture (models and tools in the service of masses), pedagogical model’s reviews and the principles on which it is based (monetization, assessment, accreditation, etc.), implications for rethinking course offerings, and the educational model of Higher Education.
Today, publishers are beginning to seriously invest in MOOCs and offer publications for the development of courses. The Elsevier publishing group has entered the edX group, and Coursera is negotiating with several publishing groups. This commercial interest may have a negative impact on teaching and monetization aspects that could be seen in the near future (Howard, 2012). On the other hand, research on the development of courses and their principles is limited, due to restrictions for researchers to access to interview students from different platforms or develop surveys on teaching functionality or technological development of the different courses. This is generating many studies by teachers who have designed their own MOOC course or by platforms that assess the impact of their own courses with the corresponding bias in investigations in both cases.
Now that the peak of «excitement» has passed, it is time to analyze the current state regarding the impact of the movement in the scientific community taken as reference two of the most representative data bases in academia and the scientific world: Journal Citation Report and Scopus to verify impact among researchers and the lines of research undertaken. Thus, this analysis could serve as a reference for future researchers to highlight both the benefitsand challenges that MOOC movement has to face for its improvement and consolidation in the educational context (Aguaded, Vázquez-Cano & Sevillano, 2013; Touve, 2012).
To date, there are no studies presenting a rigorous analysis on the MOOC movement from the conceptual and bibliometric perspectives. Regarding the analysis of the impact of publications from a bibliometric perspective, there have been some studies examining the impact from a comparative approach with other concepts (Martínez-Abad, Rodríguez-Conde & García-Peñalvo, 2014) or the impact of movement in different databases (Liyanagunawardena, Adams & Williams, 2013). No research has been developed to analyze and assess the implications of the MOOC movement in two of the most prestigious databases with greater global impact in accordance with criteria and variables that allow us to analyze the state of the art, the areas with higher impact, and the main implications for the MOOC movement. For this reason, it seems appropriate to conduct a study that analyzes the different variables, both bibliometric and semantic, that allow researchers and others interested in MOOCs to have an updated overview of the scientific impact of the movement from different variables and perspectives of study to detect the difficulties and weaknesses, including new challenges.
The research aims were twofold:
• To quantify from a bibliometric approach the MOOC scientific production in the form of articles in JCR and Scopus databases during the period 2010-2013, according to the following variables: total number of published papers; number of received citations; major citable journals; average citations per year; name, country, and institutional affiliation of the most cited authors; and articles’ methodological approach.
• Analyze the key words used in articles to establish the thematic and conceptual implications to better understand the MOOC movement.
This investigation stems from the principles embodied in bibliometric studies in the field of education (Fernández & Bueno, 1998), with the use of descriptive, quantitative, and correlational techniques with the application to the study of semantic keywords with the technique of social network analysis (Knoke & Yang, 2008) via networks generated in UCINET and visual representation with VOSviewer. The use of databases from a comparative perspective is a research method used in measuring the impact of a term or trend and is usually referenced to three international databases: JCR, Scopus, and Google Scholar (Jacso, 2005; Levine-Clark & Gil, 2009), Recently, the results of Google Scholar have been seriously questioned (Delgado, Robinson & Torres, 2014), and the recovered entries frequently found to incorporate unreliable references. For this reason, this research has been limited to the two databases with greater impact and international recognition, JCR and Scopus (Delgado & Repiso, 2013).
For the analysis, we used a technique based on the bibliographic data quantification of articles; with this approach, we obtained several indicators that have been used in other studies in relation to the following: authors, countries, institutions, and subject areas (Davis & Gonzalez, 2003; Chiu & Ho, 2005). Subsequently, we turned to the analysis of keywords frequency (Bhattacharya & al., 2003; Ding, Chowdhury & Foo, 2001) with special attention to the analysis of co-occurrence within the specific research domain of MOOCs. With the same conceptual goal, different areas of study have demonstrated successful implementations (Cahlik, 2000; Neff & Corley, 2009; Viedma & al., 2011).
Initially, the search equation «mooc» or «MOOC» or «massive open online course» was used in both JCR and Scopus databases. With the initial information from both databases, a total of 63 publications in JCR and 180 in Scopus were retrieved, and they were finally reduced to 48 and 111, respectively, by removing books, books chapters, repeated records, irrelevant publications, conference proceedings, and documents that did not fit the purpose of the study or were out of the 2010-2013 interval. We used the automated mechanisms for analysis included in both databases, with representation in figures and tables. Data extraction was performed by direct consultation of the databases according to the following variables:
• Total number of articles and quartile position.
• Number of citations received by each article.
• MOOC article citations in journals.
• Year/month of publication and average citations per year.
• Authors.
• Authors’ institutional affiliation.
• Productivity by country.
• Article’s methodological approach (theoretical, quantitative, qualitative, and mixed).
• Keywords in Scopus and JCR (networks using UCINET and word clouds by generating .txt file (WoS) and csv (Scopus) and key words visual representation in VOSviewer program).
We opted to present the quantitative data of both databases to respond to the first objective of this research. In a second phase, we present graphs of keywords in both databases and their analysis to define the major implications in the study of MOOC movement according to key topics developed until today. Table 1 shows the number of articles published in the 2010-2013 interval in both databases. The articles in Scopus are double in quantity those published in JCR; but the number of articles in relation to other concepts such as «e-learning» in the same period (1243 items) is significantly lower (Martínez-Abad, Rodríguez-Conde & García-Peñalvo, 2014).
The 159 articles were distributed heterogeneously among the different quartiles of databases. Most published articles in both databases are concentrated in 2013 (134%-84.27%) as shown in table 2 (http://goo.gl/yjS2XK). The increase in the number of citation of articles in both databases since 2010 is significant, but it continues to have a low incidence, as is shown in table 3 (http://goo.gl/uny7Eo); no article reached 5 citations in JCR and Scopus. Table 4 (http://goo.gl/6cFFJt) shows the evolution of citations distributed by month. It shows a significant increase in the number of articles published since the second half of 2013 as the MOOC movement generated more interest and data. The year 2013 concentrates almost all citations in the interval studied (84.27%). Table 5 (http://goo.gl/YSgkzD) shows that the average number of citations in the past three years increased substantially. In 2013, the average citation was 3.33 in JCR and 7.83 in Scopus, which multiplies 3- and 6-fold, respectively, the citation rates from 2010 to 2012. Despite this increase, it still represents a low rate with respect to the dissemination of informative articles on the MOOC network literature (Google Scholar shows 2125 MOOC citations in the same period).
Table 6 presents the impact of the most cited authors in the two databases, and as can be seen, it is low. For example, Professor Rita Kop (Yorkville University, Canada) with her article «The Challenges to Connectivist Learning on Open Online Networks: Learning Experiences During a Massive Open Online Course» only receives two citations in each of the two databases, whereas in Google Scholar, this article receives 98 citations. This implies a low effect on high-impact databases.
Table 7 (http://goo.gl/4Tm3vs) shows the most active countries during this early period of the movement. It is remarkable that the United States accounts for half of all citations received in both databases. The second country is the United Kingdom but quite a distance behind; Australia, Canada, and Spain occupy the following positions. Other countries are far ahead of those mentioned with an almost symbolic authors’ representation in JCR, which does not exceed 2%. In table 8 (http://goo.gl/y1GHhS), we can see that American universities are the most representative in the MOOC movement, followed by European, Canadian, and Oceania universities. The role of Spanish universities representing 50% of the European scientific production in JCR and 81.83% of production in Scopus is remarkable.
The methodological approach of the articles is a relevant aspect providing an overview of how the research and reflection on MOOC movement is being addressed at this early stage and expansion. The results show that, even today, the main body of research has focused on the theoretical reflection and essays, with a percentage of 80% in both databases. Table 9 (http://goo.gl/CTRWFh) shows the classification of articles according to their methodological approach. Thus, we can see that the ten articles with more citations in both databases have an eminently theoretical approach.
The theoretical approach of the articles with the highest citation index in both databases shows that MOOC research is still at an early stage, and the efforts made to date focus more on the informatics field than on the scientific and academic context (table 10). Some of the biggest names in MOOC research, such as George Siemens, Stephen Downes, and Sir John Daniel who have more than 200,000 search results on Google about MOOCs, have not yet published high-impact articles in these two databases.
The ten journals with the highest citation are published mostly in North American institutions (80%); Canada is represented by the journal «International Review of Research in Open and Distance Learning», one of the most productive in the MOOC movement, Australia by «Distance Education» and the only European journal is the Spanish one, «Comunicar», as you can see in figure 1 (http://goo.gl/KvMqGE).
Once the descriptive and quantitative analysis of the impact of MOOC movement in both databases was implemented, we conducted an analysis of the relations established between keywords through a graph representation. We then chose the words «Abstract» and «Keywords» as the basis for obtaining the word network once the article has been uploaded to the archive, and for the development of binary count, we considered a minimum of two items:
• Of a total of 530 terms extracted from Web of Science, the program determined that only 67 terms meet this criterion.
• Of a total of 1,715 terms extracted from Scopus, the program determined that only 323 terms meet this criterion.
After fixing these criteria, the map of keywords was generated. The matrix was previously built in the UCINET program to calculate the nodal degrees of intermediation and closeness of the five most representative concepts and descriptive keywords in both databases; results are displayed in table 11 (http://goo.gl/4oKw54). If we take into account all the criteria together, that is, nodal degree, rank, closeness, and betweenness, we find that the most relevant values are concepts related to materials or instruments used: video and educational resources as well as educational learning experience, environment, design, and evaluation. The networks presented in Figures 2 and 3 are the graphical representation of the matrix of relations among keywords in JCR and Scopus, respectively.
Figure 2: Graphical representation of keywords in JCR.
In JCR, Figure 2 shows the central position of concepts in the network (Spencer, 2003) and shows a high score of 67% with a total number of 23 nodes. The maximum degree (maximum number of relations of a node in the network) is 3,199 (video), indicating that each keyword is intertwined with an average of 3. The graph density is 0.07, a low value, well away from the value of 1 (high density). The results show that the aspects with a high standardized range (Nrmdegree: percentage of connections of a node on the total network) and a higher node degree focus on the following items: «educational resource», «openness», «assessment» and «impact».
Figure 3. Graphical representation of keywords in Scopus.
In Scopus network (Figure 3), the results of the betweenness degree are significant 27,248, providing relevant information regarding the frequency with which a node appears on the shortest (or geodesic) stretch connecting two others; that is displayed when a concept or keyword can be intermediary between others. We have summarized in Table 11 those nodes that have a higher degree of intermediation (=11) and are recurrent in the published articles: «education», «learning», «experience», «environment» and «design». The results of degree of closeness indicate that, in these five major nodes, those aspects that serve to interrelate the dominant categories in the publication of articles indexed in Scopus.
The scientific production of high impact on the MOOC movement in 2010-2013 is still in its early stages and undeveloped. The number of articles published in journals indexed in Scopus and JCR is low compared with other emerging concepts and research areas. The impact in Scopus with 111 articles is significantly greater than that in JCR with 48. Additionally, the published works present a medium-low impact index (JCR 3.33 and Scopus 7.83), which implies that these publications are not a referent of reflection for the analysis of the MOOC movement. This poses a problem for research in MOOCs, mainly because the vision of the movement from the academic world is focusing on the particular interest from certain platforms that use data for advertising or selling the benefits of this type of training without contrast or analyzing critically the data obtained. Moreover, the analysis in blogs and magazines raises the profile of the the MOOC movement, but this is not usually supported by rigorous research methods to better understand the strengths and weaknesses on which the movement is based.
The methodological approach of the ten articles with the highest citations in both databases presents mainly a theoretical approach; by contrast quantitative and qualitative approaches do not exceed 9% of the published articles, making it difficult to conduct a deep analysis from a scientific approach. Until now, universities and countries that are having greater scientific impact are the United States, Australia, Canada, the UK, and Spain. Also, the journals with the highest citation index are concentrated in the United States (80%) and, to a lesser extent, in Canada, Australia, and Spain.
Analysis by networks of abstract and keywords shows that MOOC’s relations are being linked thematically with the educational experience of learning, environment, design, and evaluation. These relations are not in direct line with the current main criticism that focuses on the pedagogical principles of connectivism, monetization, accreditation, and technological architecture of platforms and resources embedded in them (Hill, 2012; Daniel 2012, Vázquez-Cano, López-Meneses & Sarasola, 2013).
Finally, some of the biggest names in MOOC research, such as George Siemens, Stephen Downes, and Sir John Daniel, have not published in both databases and develop their reflections in lower impact journals and in their own Web pages, blogs, specific newspapers, social networks, and so on. These non-academic publications serve as a vehicle for reflection and analysis to the academic and scientific community, which should make scientific journals think about their role in the identification of emerging research fields and more call for papers on this topic to foster a deeper and scientific reflection.
Juan-Antonio Barrera, Librarian and Head of Library of Science Education at the University of Seville, and Adrián Macías-Alegre and Raquel Casanova Tristancho of Dokumentalistas.com who assisted in data extraction.
Aguaded, I., Vázquez-Cano, E. & Sevillano, M.L. (2013). MOOC, ¿turbocapitalismo de redes o altruismo educativo? Hacia un modelo más sostenible. Informe Scopeo, 2. MOOC: Estado de la situación actual, posibilidades, retos y futuro. Universidad de Salamanca.
Bhattacharya, S., Kretschmer, H. & Meyer, M. (2003). Characterizing Intellectual Spaces between Science and Technology. Scientometrics, 58, 369-390.
Cahlik, T. (2000). Search for Fundamental Articles in Economics. Scientometrics 49(3), 389-402.
Chiu, W.T. & Ho, Y.S. (2005). Bibliometric Analysis of Homeopathy Research during the Period of 1991 to 2003. Scientometrics, 63(1), 3-23.
Cooper, S. & Sahami, M. (2013). Education Reflections on Stanford’s MOOC: New Possibilities in Online Education Create New Challenges. Communications of the ACM, 56(2), 28-30. (DOI:10.1145/2408776.2408787).
Daniel, J.S. (2012). Making Sense of MOOC: Musing in a Maze of Myth, Paradox and Possibility. Journal of Interactive Media in Education, Special issue, 1, 1-20.
Davis, J.C. & Gonzalez, J.G. (2003). Scholarly Journal Articles about the Asian Tiger Economies: Authors, Journals and Research Felds, 1986-2001. Asian-Paci?c Economic Literature, 17(2), 51-61.
Delgado, E. & Repiso, R. (2013). El impacto de las revistas de comunicación: comparando Google Scholar Metrics, Web of Science y Scopus. Comunicar, 41, 45-52. (DOI: 10.3916/C41-2013-04).
Delgado, E., Robinson, N. & Torres, D. (2014). The Google Scholar Experiment: How to Index False Papers and Manipulate Bibliometric Indicators. Journal of the Association for Information Science and Technology, 65, 446-454. (DOI: 10.1002/asi.23056).
Ding, Y., Chowdhury, G. & Foo, S. (2001). Bibliometric Cartography of Information Retrieval Research by Using co-word Analysis. Information Processing & Management, 37(6), 817-842.
Downes, S. (2013). MOOC-The Resurgence of Community in Online Learning. (http://goo.gl/1GPk3y) (01-03-2014).
Durall, E., Gros, B., Maina, M., Johnson, L. & Adams, S. (2012). Perspectivas tecnológicas: educación superior en Iberoamérica 2012-17. Austin, Texas: The New Media Consortium.
Fernández Cano, A. & Bueno Sánchez, A. (1998). Síntesis de estudios bibliométricos españoles en educación. Una dimensión evaluativa. Revista Española de Documentación Científica, 3(21), 269-285.
Hill, P. (2012). Four Barriers that MOOC must Overcome to Build a Sustainable Model. E-Literate. (http://goo.gl/7F9Rs) (01-03-2014).
Howard, J. (2012). Publishers see online mega-courses as an opportunity to sell textbooks. Chronicle of Higher Education, 17 September. (http://goo.gl/tgqh9O) (01-03-2014).
Jacso, P. (2005). As we may search-comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Current Science, 89(9), 1537-1547.
Johnson, L., Adams Becker, S.,
Knoke, D. & Yang, S. (2008). Social Network Analysis. United States of America: SAGE.
Levine-Clark, M. & Gil, E. (2009). A Comparative Analysis of Social Sciences Citation Tools. Online Information Review, 33(5), 986-996.
Liyanagunawardena, T.R., Adams, A.A. & Williams, S.A. (2013). MOOC: A Systematic Study of the Published Literature 2008-12. The International Review of Research in Open and Distance Learning, 14(3), 202-227.
Martin, F. G. (2012). Will Massive Open Online Courses Change how we Teach? Communications of the ACM, 55(8), 26-28. (DOI: 10.1145/2240236.2240246).
Martínez-Abad, F., Rodríguez-Conde, M.J. & García-Peñalvo, F.J. (2014). Evaluación del impacto del término «MOOC» vs «eLearning» en la literatura científica y de divulgación. Revista de Formación del Profesorado, 18(1), 185-201.
Neff, M. & Corley, E.A. (2009). 35 Years and 160,000 Articles: A Bibliometric Exploration of the Evolution of Ecology. Scientometrics, 81(1), 657-682.
Siemens, G. (2013). What is the Theory that Underpins our MOOC? (http://goo.gl/itce4) (01-03-2014).
Spencer, J. W. (2003). Global Gatekeeping, Representation and Network Structure: A Longitudinal Analysis of Regional and Global Knowledge-diffusion Networks. Journal of International Business Studies, 34, 428-442.
Touve, D. (2012). MOOC’s Contradictions. Inside Higher Ed. 11 September. (http://goo.gl/Pu8OZJ) (01-03-2014).
Vázquez-Cano, E., López-Meneses, E. & Sarasola, J.L. (2013). La expansión del conocimiento en abierto: Los MOOC. Barcelona: Octaedro.
Viedma, M.I., Perakakis, P., Muñoz M.A., López A.G. & Vila J. (2011). Sketching the first 45 years of the Journal Psychophysiology (1964-2008): A Co-word based Analysis. Psychophysiology, 48, 1029-1036.
Williams, S., Terras, M. & Warwick, C. (2013). What People Study when they Study Twitter: Classifying Twitter related academic papers. Journal of Documentation, 69(3), 384-410.
Yuan, L. & Powell, S. (2013). MOOC and Open Education: Implications for Higher Education. Cetis. (http://publications.cetis.ac.uk/2013/667) (01-03-2014).