Volume index - Journal index - Article index - Map ---- Back
Google Scholar Metrics' launch in April 2012, a new bibliometric tool for the evaluation of scientific journals by means of citation counting, has ended with the duopoly exerted by the Web of Science and Scopus databases. This paper aims at comparing the coverage of these three databases and the similarity their journal rankings may have. We selected a sample of journals from the field of Communication Studies indexed in the three databases. Data was recollected on 1720 November, 2012. 277 journals were identified to which we calculated their hindex and ranked them according to such indicator. Then, we analyzed the correlation between the rankings generated. Google Scholar Metrics dobles the coverage of the other databases, reducing the bias toward English language both; web of Science and Scopus have. Google Scholar Metrics shows higher hindex values (an average 47% higher than Scopus and 40% higher than Web of Science), allowing to better rank journals. We conclude that Google Scholar Metrics is a tool capable of identifying the main journals in Communication Studies offering results as reliable and valid as the ones Web of Science and Scopus show.
Communication, Google Scholar, Scopus, Web of Science, citations analysis, bibliometrics, h index, ranking
The launch of Google Scholar Metrics (GSM) in April 2012 was at first greeted with jubilation, given the novelty of its appearance as an original and singular tool for the evaluation of the impact of scientific journals. In addition, it brought healthy competition into the scientific information market, dominated until then by Thomson Reuters’ Web of Science (WOS) and Elsevier’s Scopus. This led to various analyses that have subjected the new product to critical assessment trials.(Delgado López-Cózar & Cabezas, 2012a; Delgado López-Cózar & Robinson, 2012; Jacsó, 2012). Amongst the numerous criticisms received, most notable were those directed at the unusual presentation of journal rankings by language, and not by scientific discipline, and the limitation of search results using key words from journal titles to only 20 documents. Aware of these criticisms, Google launched a new version in November 2012. Here Google opted for offering general rankings by language, but also by thematic area and discipline, although limiting this option only to journals in English, excluding the other nine languages in which Google offers lists of journals (Chinese, Portuguese, German, Spanish, French, Korean, Japanese, Dutch and Italian). Given the continued limit of only 20 documents in the results display, one of the main functions of journal rankings is made impossible: the comparison of the impact of journals belonging to a particular discipline or scientific speciality (Hodge & Lacasse, 2011).
Communication journals have been placed within the thematic area of Humanities, Literature and Arts, in two disciplines: Communication and Film. In each of these, only the 20 journals with the highest h index are shown. In order to overcome these two limitations, and using various search procedures, the first objective of this paper is to provide a ranking for all those communication journals indexed in GSM. There have been a number of papers on the impact of communication journals by citation analysis, though they have fundamentally centered on aspects such as establishing the scant coverage of the Thomson-Reuters databases (Stephen, 2008), the validity of the impact indices as an evaluation indicator (Houser, 2006), the national or international orientation of the journals (Lauf, 2005), patterns of citation (So, 1988) and the similarity of journals based on network analysis (Rice & al., 1988; Hakanen & Wolfram, 1995; Park & Leydesdorf, 2009). Only Levine’s study (2010) dealt with measuring the impact of communication journals through Google Scholar, indicating the differences with the ISI database, although it covered only 30 journals. Therefore, the subject of this present paper is unpublished. On the other hand, in order to trust a new bibliometric product in which the impact of journals is measured by citations, it is advisable to test this by comparison with the rankings offered by WOS or Scopus, these being the standard reference systems in the world of scientific evaluation. These products have traditionally been criticised for their English language bias. Archambault & Gagné (2004) demonstrated how journals from the United States and Great Britain were significantly over-represented in the WOS, a problem which is more acute in social sciences and humanities. For this reason, it is appropriate to investigate to what extent the new Google product is capable of eliminating this bias and offering considered, dependable and valid results (Cabezas & Delgado López-Cózar, 2012b; Delgado López-Cózar & al., 2012). Therefore, the second objective of this paper consists of demonstrating to what extent the journal rankings generated by GSM coincide or differ from those of WOS and Scopus in the field of communication.
This study refers to scientific journals that deal with the phenomenon of communication (theory, history and research), media (press, radio and television), journalism, audiovisual media, cinema, rhetoric and journalistic message, advertising and public relations. In order to identify communication journals, the following sources of information have been consulted:
• ULRICH’S International Directory, which is considered the largest and most up-to-date directory of periodic publications in the world. It retrieved all existing scientific journals (academic/scholarly) that had been indexed by topic in the categories («subjects»): «Communication», «Journalism», «Communication Television and Cable», «Communication Video», «Advertising» and «Public Relations».
• GSM: Two strategies were employed here: Firstly, any indexed journals in the Communication category were downloaded. It should be noted that they were curiously listed under «Humanities», «Literature & Arts» and not under «Social Sciences». Secondly, a series of searches in journal titles was undertaken using the following keywords: «Communication», «Mass Communications», «Communication Research», «Journalism», «Media», «Film», «Advertising», «Cinema, Audiovisual», «Audio», «Radio», «Television», «Public Relations», «Public Opinion», «Movie». These searches were carried out in the following languages: English, French, Spanish, German, Italian, Portuguese, Chinese, Japanese, Korean, Arabic, Russian, Turkish and Polish.
• Communication & Mass Media Complete: Communication journals considered as «core», that is, entered in the database in their entirety (cover to cover). (www.ebscohost.com/academic/communication-mass-media-complete).
• WOS: Journals indexed in the topical categories of «Communication» and «Film, Radio & Television» (http://ip-science.thomsonreuters.com/mjl).
• Scopus: Journals indexed in the topical categories of «Communication» and «Visual Arts and Performing Arts» (www.info.sciverse.com/scopus/scopus-in-detail/facts).
After a manual filter of the entries for each search, to identify the relevant journals for the subject area covered by this paper, all the information was downloaded into a Microsoft Access® database, where titles were unified and any duplicates eliminated.
A total of 664 communication journals were identified. These journals were then searched for in GSM in the last week of November 2012. 277 journals appearing in appendix 1 (http://bit.ly/X7HFBO) were found. These are ordered according to both the h index and the average number of citations obtained by the articles which contributed to the h index. The h index, which is the bibliometric indicator adopted by GSM to measure the impact of journals, was proposed by Jorge Hirsch in 2005 for use in measuring researcher performance but was immediately applied to journals (Braun & al., 2005). A journal has an index equal to h when h of its articles have received at least h citations each; that is, a journal with an h index of 22 is one which has published 22 articles with at least 22 citations for each of these articles. The h index has received substantial attention, leading to numerous studies, proposals of new indicators, and already has at least two bibliographical reviews (Alonso & al., 2009; Egghe, 2010).
In order to carry out the comparative study with WOS and Scopus, the 277 journals located in GSM were searched for in these databases. Searches were limited to the same period as that used by GSM (2007-2011). Therefore, this comparative analysis only applies to the journals that are well represented in GSM and WOS (N=63) and in GSM and Scopus (N=102). Spearman’s correlation coefficient (rho) was used, habitually employed in bibliometric studies in order to measure the grade of association between two variables according to their position in different rankings (Leydesdorf 2009; Bollen 2009; Torres-Salinas & al., 2010). The h indices of the journals offered by the three platforms were correlated according to the geographical origin of the publication, the language of the edition and the publishing house.
664 communication journals have been identified by this study (Figure 1). Ulrich’s, the directory specialised in the control of periodical publications, was the system that located the highest number of journals. This was closely followed by GSM, a surprising fact when taking into account the general and open nature of the tool. Communication & Mass Media Complete also came close, a logical occurrence for an international database specialised in communication journals. The two large multidisciplinary databases (Scopus and WOS) had significantly lower coverage. However, it is worth noting that Web of Science contains 15 journals not registered in the other databases.
What is truly unprecedented is the limited overlapping that exists between the databases. 65% of the journals (433) appear indexed in one database only (Figure 1); Ulrich’s being once again the product with the widest coverage. This means, that in order to carry out a thorough inventory of communication journals, it is necessary to turn to several information systems, which come to be complementary.
Focusing on the primary objective of this paper, appendix 1 relates the communication journals ordered according to the h index provided by GSM for the period 2007-2011. As previously stated, there are 277 journals. As can be seen, GSM does not manage to cover half (41.7%) the sphere of communication journals circulating in the world. The strict criteria of inclusion adopted by GSM (journals with more than 100 published articles with at least one citation in the last 5 years), excludes a large number of publications unable to reach this threshold.
The values of the h indices obtained are not particularly low: 70% of the journals have an index equal to or greater than 5, with a maximum value of 43 and a minimum of 1. This allows a relatively reliable identification of the leading journals within the speciality, indicating pronounced differences between journals. The first quartile of the h index is dominated by journals of English language origin (United States and United Kingdom) and written in English. Only five journals are not edited in these countries and only one is published in a language other than English. It is necessary to move down to the last positions in the second quartile in order to find Chinese, Spanish, Brazilian, French or Portuguese journals; even then, they do not represent more than a third of the journals in this quartile.
The second objective of this paper is to compare GSM with WOS and Scopus, the traditional systems for the evaluation of the impact of journals by means of citation analysis. The main results obtained are set out below. From the point of view of size, GSM almost doubles (65.9%) the number of titles covered by Scopus and almost triples those indexed by WOS (figure 2).
When considering the geographical origin of the journals covered (table 1), GSM reduces the English language bias that has historically stigmatized both WOS and Scopus, representing in a more balanced manner the actual importance of the different nations publishing journals (table 1). A simple comparison of the distribution of the 664 journals by countries that constitute the sphere of communication journals identified in this study reveals how GSM is almost precisely adjusted to the volume of journal production in the world, as opposed to the completely biased distribution of WOS and Scopus. GSM not only indexes journals from more countries (30 GSM compared to 23 from Scopus and 13 from WOS), but also reduces the percentage of journals from the United States and the United Kingdom to 53.79%, compared to 74.85% from Scopus and 80.19% from WOS. Furthermore, it gives adequate room to countries with an undeniable weight in the production of journals, such as China and Brazil, whose journals do not even appear in WOS and only very nominally in Scopus.
If the publication language of the journals is analysed, a similar situation is found. GSM is better adjusted to the real use of different languages in existing communication journals throughout the world, eliminating the heavy English language bias of Scopus and WOS (table 2). So, whilst Scopus and WOS only register journals published in 7 different languages, GSM includes journals in 13 languages. English, the principal publication language for communication journals, has a very different prominence depending on the database. It represents 59.67% of GSM indexed journals (182) whilst in Scopus and WOS it reaches 88.66% and 86.92% respectively (172 and 93 journals). Spanish is placed as the second language in all of the databases but its percentage of representation varies between 5.15% in Scopus, 5.61% in WOS and 13.77% in GSM. It should be noted that, in GSM, Chinese and Portuguese are the third and fourth most used languages respectively (8.85% y 7.78%), whilst Scopus only registers one journal in Chinese (0.52%) and three in Portuguese (1.55%). WOS does not include either of these two languages (Table 2).
However, when the distribution of journals is studied according to their publishing houses, the same pattern is found for all three products (Table 3). The same six publishing houses in the three databases (Routledge y Sage, Taylor & Francis, Wiley-Blackwell, John Benjamins, Oxford University Press), in equal measure, publish the majority of the journals. Nevertheless, as occurred with language and country of publication, it is true that GSM is much more open to all types of publishing houses, having a lower concentration. Therefore, if the 10 most productive publishing houses represent 54.49% of the total in Scopus and 65% in WOS, this only reaches 40% for GSM.
Finally, from a bibliometric viewpoint, when a new product appears for evaluating the impact of scientific journals by means of citation analysis, it is most relevant to compare the extent to which the journal rankings given by GSM are similar or different to those given by WOS and Scopus. The comparative table of h indices of the journals in Google Scholar Metrics, WOS and Scopus has been uploaded to the following address http://bit.ly/YQZkZP.
Firstly, it can be confirmed that the average h index of the journals in the sample is 40% higher than that of WOS and 47% higher than that of Scopus. In some highly significant journals in the sphere of Communication («Public Opinion Quarterly», «Journal of Communication», «Telecommunications Policy», «Communication Research», «Public Relations Review») the GSM h indices are two or three times higher than the two other aforementioned databases. Secondly, a high similarity between the three rankings is observed, there being only slight differences as to which are the leading journals. The high level of correlation detected (0.895 for WOS and 0.879 for Scopus) shows this close similarity between rankings (Table 4). Naturally, this does not mean they are identical; for example, «Journalism Studies», which is the 17th journal according to GSM, falls to 27th place in WOS and 61st in Scopus.
The first conclusion that can be reached by this study is the difficulty of adequately identifying and locating the journals produced worldwide in the scientific field of communication. None of the databases used here are capable of exhaustively monitoring all of the existing journals, for which reason it is necessary to use all three databases together. Despite the problematic technique implied by this lack of bibliographic control, it is certain that a contributory factor is the multidisciplinary nature of Communication itself, having boundaries so vague as to prevent a clear delimitation of the field covered. Communication receives a substantial theoretical inheritance from many other fields, such as rhetoric, sociology, psychology and semiotics (Craig, 1999) and has undergone important fluctuations in its epistemological values. It has passed from rhetoric, discourse and the media of the masses, to centre on the new means of communication: public relations, advertising and human communication (Craig 2003; Chung & al, 2009).
Apart from this discovery, collateral to the objective of this paper, the main finding obtained is the more than ample coverage of GSM, not only for identifying communications journals but also its manifest utility as a tool for the evaluation of the scientific impact of scientific journals. It not only covers more journals than its competitors (WOS and Scopus) but also lacks their English language bias; registering journals originating from more countries and written in more languages. All this is achieved in spite of its restrictive indexing policy. These results confirm what has been suggested by previous empirical studies on Google Scholar (Bakkalbasi & al., 2006; Meho & Yank, 2007; Falagas & al., 2008; Bar-Ilan, 2008-2010; Kulkarni & al., 2009), which is the information source for generating the bibliometric data offered by GSM. Being built on Google Scholar, GSM is based on the most thorough and least biased academic and scientific data source currently in existence.
When the ranking of journals is compared with that offered by Scopus and WOS, it is confirmed that GSM offers greater indicators (almost double) and, most relevantly, a high correlation, something already demonstrated by other studies (Vanclay, 2008; Harzing & Wal, 2009; Delgado López-Cózar & al., 2012; Cabezas & Delgado López-Cózar, pending publication). Consequently, it can be affirmed that GSM measures journals in a very similar way to the classic journal evaluation systems (WOS and Scopus) for which, broadly speaking and for ranking only purposes, it is an equally reliable and valid alternative for measuring the impact of journals.
In short, this paper supplies the h index impact of 277 communication journals. Although this figure represents approximately less than half (41.7%) the sphere of communication journals circulating in the world, it includes those journals considered world leaders, which in addition occupy the top positions.
Alonso, S., Cabrerizo, F.J., Herrera-Viedma, E. & Herrera, F. (2009). H-Index: A Review focused in its Variants, Computation and Standardization for Different Scientific Fields. Journal of Informetrics, 3(4), 273-289. (http://dx.doi.org/10.1016/j.joi.2009.04.001).
Archambault, E. & Gagné, E.V. (2004). The Use of Bibliometrics in Social Sciences and Humanities. Montreal: Social Sciences and Humanities Research Council of Canada (SSHRCC).
Bakkalbasi, N., Bauer, K., Glover, J. & Wang, L. (2006). Three Options for Citation Tracking: Google Scholar, Scopus and Web of Science. Biomedical digital libraries, 3(7). (http://dx.doi.org/10.1186/1742-5581-3-7).
Bar-Ilan, J. (2008). Which h-index? A Comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257–271. (http://dx.doi.org/10.1007/s11192-008-0216-y).
Bar-Ilan, J. (2010). Citations to the ‘Introduction to Informetrics’ Indexed by WOS, Scopus and Google Scholar. Scientometrics 82(3), 495–506. (http://dx.doi.org/-10.1007/s11192-010-0185-9).
Bollen, J., Van de Sompel, H., Hagberg, A. & Chute, R. (2009). A Principal Component Analysis of 39 Scientific Impact Measures. PloS one, 4(6), e6022. (http://dx.doi.org/-10.1371/journal.pone.0006022).
Braun, T., Glänzel, W. & Schubert, A. (2005). A Hirsch-type Index for Journals. The Sciencist, 19(22), 8-10.
Cabezas-Clavijo, Á. & Delgado López-Cózar, E. (2012a). ¿Es posible usar Google Scholar para evaluar a las revistas científicas nacionales en los ámbitos de Ciencias Sociales y Jurídicas? El caso de las revistas españolas. EC3 Working Paper, (3). (http://eprints.rclis.org/handle/10760/16888).
Cabezas-Clavijo, Á. & Delgado López-Cózar, E. (2012b). Las revistas españolas de Ciencias Sociales y Jurídicas en Google Scholar Metrics, ¿están todas las que son? EC3 Working Papers, (2). (http://eprints.rclis.org/handle/10760/16892).
Cabezas-Clavijo, Á. & Delgado López-Cózar, E. (in press). Google Scholar y el índice h en Biomedicina: la popularización de la evaluación bibliométrica. Medicina Intensiva.
Chung, C., Lee, S., Barnett, G. A. & Kim, J. (2009). A Comparative Network Analysis of KSJCS and ICA in the Era of Hybridization. Asian Journal of Communication, 19(2), 170-191. (http://dx.doi.org/10.1080/01292980902827003).
Craig, R. T. (1999). Communication Theory as a Field. Communication Theory, 9(2), 119–161. (http://dx.doi.org/ 10.1111/j.1468-2885.1999.tb00166.x).
Craig, R.T. (2003). Discursive Origins of a Communication Discipline. Annual Convention of the National Comunication Associations. Miami Beach.
Delgado López-Cózar, E. & Cabezas-Clavijo, Á (2012b). Google Scholar Metrics revisado: Ahora empieza a ir en serio. EC3 Working Papers, (8). (http://digibug.ugr.es/handle/10481/22439).
Delgado López-Cózar, E. & Robinson-García, N. (2012). Repositories in Google Scholar Metrics or what is this Document Type Doing in a Place as Such? Cybermetrics, 16(1), paper 4.
Delgado López-Cózar, E.; Orduña-Malea, E.; Marcos-Cartagena, D.; Jiménez-Contreras, E. & Ruiz-Pérez, R. (2012). Journal Scholar: Una alternativa interna-cional, gratuita y de libre acceso para medir el impacto de las revistas de Arte, Humanidades y Ciencias Sociales. EC3 Working Papers, 2012, (5). (http://digibug.ugr.es/handle/10481/20375).
Delgado-López-Cózar, E. & Cabezas-Clavijo, Á. (2012a). Google Scholar Metrics: an unreliable tool for assessing scientific journals. El Profesional de la Información, 21(4), 419–427. (http://dx.doi.org/ 10.3145/epi.2012.jul.15).
Egghe, L. (2010). The Hirsch Index and Related Impact Measures. Annual review of information science and technology, 44(1), 65-114. (http://dx.doi.org/ 10.1002/aris.-2010.1440440109).
Falagas, M.E., Pitsouni, E.I., Malietzis, G.A. & Pappas, G. (2008). Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and Weaknesses. The FASEB Journal, 22(2), 338-342. (http://dx.doi.org/10.1096/fj.07-9492LSF).
Hakanen, E.A. & Wolfram, D. (1995). Citation Relationships among International Mass Communication Journals. Journal of information science, 21(3), 209-215. (http://dx.doi.org/10.1177/016555159502100306).
Harzing, A. & Wal, R. Van der. (2009). A Google Scholar H-index for Journals: An Alternative Metric to Measure Journal Impact in Economics and Business. Journal of the American Society for Information Science and Technology, 60(1): 41–46. (http://dx.doi.org/10.1002/asi.20953).
Hodge, D.R. & Lacasse, J.R. (2011). Ranking Disciplinary Journals with the Google Scholar h-index: A New Tool for Constructing Cases for Tenure, Promotion, and Other Professional Decisions. Journal of Social Work Education, 47(3), 579-596. (http://dx.doi.org/10.5175/JSWE.2011.201000024).
Houser, E.T. (2006). The Evaluative Use of Citation Analysis for Communication Jour-nals. Human Communication Research, 22(4), 563-574. (http://dx.doi.org/10.1111/-j.1468-2958.1996.tb00379.x).
Jacsó, P. (2012). Google Scholar Metrics for Publications: The Software and Content Features of a New Open Access Bibliometric Service. Online Information Re-view, 36(4), 604-619. (http://dx.doi.org/10.1108/14684521211254121).
Kulkarni, A.V., Aziz, B., Shams, I. & Busse, J.W. (2009). Comparisons of Citations in Web of Science, Scopus, and Google Scholar for Articles Published in General Medical Journals. JAMA: The Journal of the American Medical Association, 302(10), 1092-1096. (http://dx.doi.org/ 10.1001/jama.2009.1307).
Lauf, E. (2005). National Diversity of Major International Journals in the Field of Communication. Journal of communication, 55(1), 139-151. (http://dx.doi.org/ 10.1111/j.1460-2466.2005.tb02663.x).
Levine, T.R. (2010). Rankings and Trends in Citation Patterns of Communication Journals. Communication Education, 59(1), 41-51. (http://dx.doi.org/ 10.1080/03634520903296825).
Leydesdorff, L. (2009). How Are New Citation-based Journal Indicators Adding to the Bibliometric Toolbox? Journal of the American Society for Information Science and Technology, 60(7), 1327-1336. (http://dx.doi.org/10.1002/asi.21024).
Meho, L.I. & Yang, K. (2007). Impact of Data Sources on Citation Counts and Rankings of LIS Faculty: Web of Science versus Scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105-2125. (http://dx.doi.org/10.1002/asi.20677).
Park, H.W. & Leydesdorff, L. (2009). Knowledge Linkage Structures in Communication Studies Using Citation Analysis among Communication Journals. Scientometrics, 81(1), 157-175. (http://dx.doi.org/10.1007/s11192-009-2119-y).
Rice, R.E., Borgman, C.L. & Reeves, B. (1988). Citation Networks of Communication Journals, 1977–1985 Cliques and Positions, Citations Made and Citations Received. Human Communication Research, 15(2), 256-283. (http://dx.doi.org/10.1111/j.1468-2958.1988.tb00184.x).
So, C.Y. (1988) Citation Patterns of Core Communication Journals: An Assessment of the Developmental Status of Communication. Human Communication Research, 15(2), 236-255. (http://dx.doi.org/10.1111/j.1468-2958.1988.tb00183.x).
Stephen, T.D. (2008). Measuring the Reputation and Productivity of Communication Programs. Communication Education, 57(3), 297-311. (http://dx.doi.org/10.1080/-03634520801905600).
Torres-Salinas, D. & Jiménez-Contreras, E. (2010). Introducción y estudio com-parativo de los nuevos indicadores de citación sobre revistas científicas en Journal Citation Reports y Scopus. El Profesional de la Información, 19(2), 201-208. (http://dx.doi.org/10.3145/epi.2010.mar.12).
Vanclay, J.K. (2008). Ranking Forestry Journals Using the H-index. Journal of infor-metrics, 2(4), 326-334. (http://dx.doi.org/10.1016/j.joi.2008.07.002).