Journal Impact Factor

All citation-based metrics can probably be traced back to a paper by Gross and Gross (1927)Gross and Gross (1927); Archambault and Larivière (2009)Archambault and Larivière (2009) tap that article as the starting point for the Journal Impact Factor. Since the Journal Impact Factor is the best known and possibly most widely used citation-based metric for research, its origins can serve as an example for all journal-level metrics.

SNIP – Source Normalized Indicator of journal impact per Paper

As mentioned earlier, the lack of cross-field comparisons in the Journal Impact Factor means that its use is problematic for institutions that want to equitably evaluate the quality of research done by faculty and staff in an array of fields. The Journal Impact Factor of journals cannot be compared between fields “because citation practices can vary significantly from one field to another” (Moed 2011)(Moed 2011).

The source normalized indicator of journal impact per paper, ‘SNIP’, includes refinements to address this problem through the normalization of citations. SNIP normalizes the citation rates of articles against the average number of cited references per paper in that subject field (Moed 2010)(Moed 2010). In a simplistic example, if the articles in a field typically cite 10 articles, then a set of articles with a non-normalized citations rate of 12 would have a normalized citation rate of 1.2, which can be characterized as above average for that field.

To normalize away the differences in citation practices, however, requires properly delineating where those differences exist; in an article responding to criticisms of SNIP, Moed (2010)Moed (2010) cites Garfield (1979)Garfield (1979) who reported that citation practices differ not only between fields, but between specialties and sub-specialties.

To ensure that a journal's citations are normalized against an appropriate field of research, within SNIP a particular journal's field is not determined by categorization but instead it is defined by what articles cite the journal. In other words, the “field” is the collection of all articles outside the journal that contain citations to articles inside the journal. (It should be noted that the term “article” is being used loosely here. A more accurate description might be “citable document”, since within SNIP “articles, conference proceedings papers and reviews are considered as fully fledged, peer-reviewed research articles” (Moed 2010)(Moed 2010).) In short, SNIP normalizes a journal's citations based on the field's average citations, where what constitutes the field is defined by the citations to the journal.

A version of SNIP that runs against the Scopus® database is available on the Journal Metrics website, www.journalmetrics.com (Elsevier 2015)(Elsevier 2015).

SJR

Another problem with the Journal Impact Factor is that all citations count the same, regardless of their source. However, some would argue that citations from certain sources should be weighted more than others, much like the opinion of an expert carrying more weight than that of a novice. The SCImago Journal Rank indicator (SJR), does just that by using an implementation of social network analysis to rank the relative prestige of the journals. This more gradated ranking is useful because more “poorly cited journals are entering the indices, [therefore] it is essential to have metrics that will allow one to distinguish with greater precision the level of prestige attained by each publication” (González-Pereira et al. 2010)(González-Pereira et al. 2010).

The SJR calculates for each journal its eigenvector centrality, which is “a measure of centrality… in which a unit's centrality is its summed connections to others, weighted by their centralities”, where ‘centrality’ is “network-derived importance” (Bonacich 1987)(Bonacich 1987) . For the SJR, therefore, ‘centrality’ is an indicator of a journal's prestige. The eigenvector centrality calculated for a journal is then normalized using the total number of citations to the journal, resulting in a size-independent rank.

The resulting SJR metric is “aimed at measuring the current ‘average prestige per paper’ of journals for use in research evaluation processes” (González-Pereira et al. 2010)(González-Pereira et al. 2010).

PageRank / Eigenfactor.org

Another way to rank journals based on prestige instead of simple popularity, is the approach used by Eigenfactor.org®, which “ranks the influence of journals much as Google’s PageRank algorithm ranks the influence of web pages” (West 2015)(West 2015).

Bollen et al. (2006)Bollen et al. (2006) examined using the PageRank methodology to rank journals by prestige, using “the dataset of the 2003 ISI Journal Citation Reports to compare the ISI [Impact Factor] and Weighted PageRank rankings of journals.” To achieve the ranking, citations are used as a basis to “iteratively” pass prestige from one journal to the other, until “a stable solution is reached which reflects the relative prestige of journals.” This is an adaptation of the original PageRank method, the difference being that connections between journals are weighted, based on citation frequencies.

Prestige and citation rankings were found to be largely similar, with two notable exceptions. The first were “journals that are cited frequently by journals with little prestige” that rank much lower on a prestige index than they do on a citation index. The second were the converse of the first, “journals that are not frequently cited, but their citations come from highly prestigious journals” so that they rank much higher on a prestige index than they do on a citation index.

Based on specific examples of these two types of journals that have distinctly different rankings when sorted by citations versus prestige, in general it is theory-heavy journals that have low citation counts, yet high prestige. The journals with high counts but low prestige are “methodological and applied journals” or ones “that frequently publish data tables” (Bollen et al. 2006)(Bollen et al. 2006).

It is, however, unclear if this result is an indication of the importance of including work on theory to produce high-quality research, or simply evidence that a weighted PageRank methodology is effective at maintaining previous perceptions of the relative values of theoretical and applied research.

h-Index

Here, I would like to propose a single number, the ‘h index,’ as a particularly simple and useful way to characterize the scientific output of a researcher. (Hirsch 2005)(Hirsch 2005)

The “h-index” is indeed a simple metric. To find the h-index of a researcher, take all the articles that the researcher has published, and sort them in ascending order of how many citations each has received. Then start counting the articles, starting with the one with the greatest number of citations; when you come to an article which has a number of citations equal to the count of articles so far, that article/citation count is the h-index for the researcher. For example, a researcher who has an h-index of 6 has published six articles with at least six citations each.

The h-index is popular enough to have prompted the creation of various refinements to the basic method, e.g., the g-index (Egghe 2013)(Egghe 2013) and the h^m-index (Schreiber 2008)(Schreiber 2008), and to define two of the three citation metrics displayed on Google Scholar's user-citations page. Those three metrics are: the user's h-index, the i10-index, a variant of the h-index (Connor 2011)(Connor 2011), and finally a count of total citations.

Yet, the usefulness of the h-index is unclear. In the original proposal, the index was offered as a quantitative metric to be used “for evaluation and comparison purposes” (Hirsch 2005)(Hirsch 2005). To demonstrate this use the author reported the h-index value for a collection of example researchers, including Nobel-prize winners. But, as Adler et al. explain:

One can conclude that it is likely a scientist has a high h-index given the scientist is a Nobel Laureate. But without further information, we know very little about the likelihood someone will become a Nobel Laureate or a member of the National Academy, given that they have a high h-index. (Adler et al. 2009)(Adler et al. 2009)

Indeed, the h-index has be shown to be inferior to equally simple metrics: “Compared with the h-index, the mean number of citations per paper is a superior indicator of scientific quality, in terms of both accuracy and precision.” (Lehmann 2006)(Lehmann 2006).

Though, there is a more important question to ask than which of the simple, article-level metrics is a better indicator of research quality—are any of them actually useful? Evaluating the quality of the work being done by a researcher is not a simple task, so any simple tool will inevitably prove to be inadequate. In reference to the h-index and a number variants of it, Adler et al. write:

These are often breathtakingly naïve attempts to capture a complex citation record with a single number. Indeed, the primary advantage of these new indices… is that the indices discard almost all the detail of citation records, and this makes it possible to rank any two scientists. Even simple examples, however, show that the discarded information is needed to understand a research record. (Adler et al. 2009)(Adler et al. 2009)

Despite the complexity of citation records, simple, single-number metrics comprise the majority of commonly referenced article-level metrics. This mirrors the wide array of single-number journal-level metrics available. Perhaps this focus on lone-value quantification is a consequence of having electronic indices of research publications that make such quantification simple.

Presumably, using data sources not so readily reduced to a single number would produce a more complex measure of a researcher's work. Such measures are classed as “alternative metrics,” a.k.a. “altmetrics”, and are described next.

Altmetrics

Though technically not a bibliometric method, altmetrics have been used to perform analyses similar to the bibliometric methods already discussed, so they are included here.

Altmetrics use data available on the web such as “usage data analysis (download and view counts); web citation, and link analyses” (Zahedi et al. 2014)(Zahedi et al. 2014) are used to supplement and improve upon citation-based metrics for measuring the impact of science and research. Using information made available via the web, altmetrics can go beyond the journal articles and books included in citation-based metrics and include “other outputs such as datasets, software, slides, blog posts, etc.” (Zahedi et al. 2014)(Zahedi et al. 2014).

So, instead of compiling citations to research in journals, altmetrics involves compiling “mentions” of research in “main-stream media sources, and social media shares and discussions”, along with statistics on downloads, and reference manager counts (Adie and Roe 2013)(Adie and Roe 2013).

Altmetrics also typically retain the related meta-data of mentions and usage statistics, which allow for more complex analyses of the information. In other words, altmetrics not only track what research is being “mentioned”, but also where it is mentioned, and who is mentioning it, which can potentially provide a richer understanding of the citations.

Perhaps most importantly, by drawing on sources outside of academia and research journals, altmetrics might provide some indications of the “pratical impact” of research. This is discussed in the “Future research” section.

Research Quality Framework (RQF)

The history of the Research Quality Framework (RQF) is complicated. It started in 2003 when the Australian government established “an Expert Advisory Group, whose remit was to consult widely and develop a model for assessing the quality and impact of research in Australia” (Butler 2008)(Butler 2008). The framework proposed by that group was published in 2006, and prompted the establishment of the Development Advisory Group, to “refine the RQF model” (Donovan 2008)(Donovan 2008). The first RFQ was scheduled to be run in 2008, but a change in government in 2007 instead meant that the program was scrapped (Donovan 2008)(Donovan 2008).

However, the Research Quality Framework has influenced the development of similar programs, such as the Research Excellence Framework in the UK (Grant et al. 2010)(Grant et al. 2010), and discussion of its development are available in the literature (see Butler 2008Butler 2008, Donovan 2008Donovan 2008).

Research Excellence Framework (REF)

The Higher Education Council for England “funds and regulates universities and colleges in England” (HEFCE 2015)(HEFCE 2015).

The HEFCE will be using the Research Excellence Framework (REF) “for the assessment and funding of research in UK higher education institutions”. The REF focuses on three elements (HEFCE 2009a)(HEFCE 2009a):

1. Research quality – research “will be assessed against criteria of ‘rigour, originality and significance’. By ‘significance’, we mean the extent to which research outputs display the capacity to make a difference either through intellectual influence within the academic sphere, or through actual or potential use beyond the academic sphere, or both.”
2. Research impact – “…demonstrable economic and social impacts that have been achieved through activity within the [research institution] that builds on excellent research.”
3. Research environment – “…the extent to which the [research institution] has developed a research infrastructure, and a range of supporting activity, conducive to a continuing flow of excellent research and to its effective dissemination and application.”

However, the three elements are not equally weighted: “The assessment of research impact will be one of three distinct elements of the REF, being judged alongside research excellence and research environment, contributing 25% towards the overall outcome (as compared with 60% and 15% for quality and environment).” (Technopolis 2010)(Technopolis 2010)

A pilot exercise of the REF was conducted in 2009, reports on the outcomes and evaluations from that pilot are referenced in the following sections.

REF

Much like with ‘practical impact’ (discussed below), the HEFCE ran a pilot exercise of the evaluation of research quality at higher-education institutions. Specifically aiming to validate the use of bibliometric methods, the key finding of the exercise was: “Bibliometrics are not sufficiently robust at this stage to be used formulaically or to replace expert review in the REF” (HEFCE 2009b)(HEFCE 2009b). But it was not a complete rejection of bibliometrics, as “there is considerable scope for citation information to be used to inform expert review” (HEFCE 2009b)(HEFCE 2009b).

Thus, the REF uses expert panels, comprising both experts in the discipline as well as end-users of research, to evaluate the research quality of an institution, with bibliometric methods being included in the evaluation process. (Grant et al. 2010)(Grant et al. 2010)

This approach also aligns with the view given in a report commissioned by HECFE before the running of the pilot exercise from the Centre for Science and Technology Studies, Leiden University:

Citation counts can be seen as manifestations of intellectual influence, but the concepts of citation impact and intellectual influence do not necessarily coincide. Citation impact is a quantitative concept that can be operationalised in an elementary fashion or in more sophisticated ways, such as crude citation counts versus field-normalised measures. Concepts such as ‘intellectual influence’ are essentially theoretical concepts of a qualitative nature, and have to be assessed by taking into account the cognitive contents of the work under evaluation. Thus, the outcomes of bibliometric analysis must be valued in a qualitative, evaluative framework that takes into account the contents of the work. (van Raan et al. 2007)(van Raan et al. 2007)

Using bibliometrics within a qualitative framework—such as review by expert panels—was also the approach in the final version of the RFQ.

RFQ

Under the Australian RFQ, research quality would have been assessed by a panel of experts, with one quarter of the panel members being end-users of research. The panel would have used a spectrum of bibliometric measures to compliment the review process.

The government ministry created a ‘Quality Metrics Working Group’, which developed recommendations for what bibliometrics would—and would not—be appropriate. The working group specifically rejected the Journal Impact Factor (né ISI Impact Factor):

It was believed that actual citation counts are a far better citation measure for judging the performance of groups than surrogates based on the average citation rates of the journals which carry that work. There were also concerns about the way in which the indicator is calculated and anecdotal evidence of increasing manipulation of the indicator by a few journal editors. Even when ranking journals, some disciplines had already made it clear that they wished to look beyond the Impact Factor and undertake a more detailed assessment of the quality of journals. (Butler 2008)(Butler 2008)

The working group on metrics recommended that panels choose from a “suite” of metrics that included citations reported as simple counts, averages, or centile distributions. Also, it was recommended that some fields might want to include citations from “non-standard venues”, meaning from outside the standard indicies of research publications from which bibliometrics typically draw data—though this practice was discouraged, since drawing citations from venues outside those indices would be a labor-intensive process. And the working group specifically recommended “that no attempt be made to aggregate the indicators to produce a single score” (Butler 2008)(Butler 2008).

Ostensibly the working group recommended against aggregating quantitative measures to alleviate concerns that a single quantitative measure might have undue influence, but the advice also falls in line with concerns that “analysis of research performance on the basis of journals unavoidably introduces a ‘bibliometrically limited view of a complex reality’” (van Raan et al. 2007)(van Raan et al. 2007).

So, while bibliometrics were to play a role in the RFQ, and do play a role in REF, they are used within a qualitative framework since the evaluation of research is a complex and muli-faceted problem. As Adler et al. write:

We do not dismiss citation statistics as a tool for assessing the quality of research—citation data and statistics can provide some valuable information. We recognize that assessment must be practical, and for this reason easily derived citation statistics almost surely will be part of the process. But citation data provide only a limited and incomplete view of research quality, and the statistics derived from citation data are sometimes poorly understood and misused. Research is too important to measure its value with only a single coarse tool. (Adler et al. 2009)(Adler et al. 2009)

So, a mix of qualitative and quantitative methods is used to evaluate research quality. And it turns out that a mixture of methods is also used for the evaluation the practical impact of research, a discussion of which is in the next section.

REF

The documentation published by HEFCE for the REF explained that practical impacts would include “a wide definition of impacts, including economic, social, public policy, cultural and quality of life” and that any reference to “‘impact’ or ‘social and economic impact’” implicitly included the entire wide range of impacts. (HEFCE 2009a)(HEFCE 2009a)

This did not prevent confusion about what was meant, unfortunately. In a “lessons learned” report about the HEFCE-REF pilot exercise, one of the items given as a challenge was explaining to academic groups “that socio-economic impact was a much broader concept than economic impact” (Technopolis 2010)(Technopolis 2010).

Some outside the process seem similarly unclear on the wider impact being sought by the REF. Commenting on the high ratings received by all groups taking part in the pilot exercise, Khazragui and Hudson (2015)Khazragui and Hudson (2015) write, “But in part too it is a consequence of having economic impact evaluated by non-economists." The implication being that since economic impact is the dominant feature of overall impact, an over-estimation of economic impact would inevitably cause a large-magnitude effect on the results. Therefore the high ratings can be explained by a failure to properly quantify the economic impact.

The objection of Khazragui and Hudson to a lack of quantitative methods, presumably is what led them to denigrate the nature of the evaluation process, writing, “research funders also illustrate their impact with ‘stories’” (Khazragui and Hudson 2015)(Khazragui and Hudson 2015).

The authors are presumably are referring to the “narrative evidence” that was used for the REF, since “there are limitations in the extent to which the impacts of research can be ‘measured’ through quantifiable indicators.” The REF used a qualitative process.

Rather than seek to measure the impacts in a quantifiable way, impact will be assessed in the REF. Expert panels will review narrative evidence supported by appropriate indicators, and produce graded impact sub-profiles for each submission; they will not seek to quantify the impacts. (HEFCE 2009a)(HEFCE 2009a)

Quantitive data were included in the process, however. The HEFCE directed that the case studies that were submitted by research institutions should “include a range of indicators of impact as supporting evidence”. Those indicators were expected to be quantified values, such as the research income generated from other funding sources, and accountings of collaborations with companies in the private sector (HEFCE 2009a)(HEFCE 2009a).

RFQ

In the RFQ, research would have practical impact if it created benefits in at least one of four domains:
social, economic, environmental, or cultural (Donovan 2008)(Donovan 2008)

• Social Benefit – new approach to social issues, improved policies, improved equity, improved health, safety and security, etc.
• Economic Benefit – improved productivity, increased employment, increased innovation and global competitiveness, “unquantified economic returns resulting from social and public policy adjustments”, etc.
• Environmental Benefit – reduced waste and pollution, reduced consumption of fossil fuels, improved management of natural resources, etc.
• Cultural Benefit – greater understanding of “where we have come from, and who and what we are as a nation and society”, stimulating creativity within the community, contributing to cultural preservation and enrichment, etc.

However, the definition of impact in the final version of the Australian RFQ program was the result of tensions between defining impact in terms of “the interests of industry and commerce” and defining it in relation to “broader public benefits” (Donovan 2008)(Donovan 2008). These tensions are played out in the differences between three sources: two sets of preliminary recommendations for the program and the final version of the RFQ.

For example, one of the preliminary recommendations defined impact in terms of “direct practical utility” and within the document the term ‘impact’ was at times used interchangeably with the word ‘usefulness’. It advocates measuring impact with quantitative metrics, such as business investments and returns, numbers of patents issued, etc. In contrast, the other preliminary recommendation promotes combining quantitative indicators with qualitative evidence in order to include the intangible benefits of research in the evaluation process. The final version of the RFQ included a “case study” methodology for evaluating impact, which would allow for the more holistic method of evaluation, but also included a scale for the scoring of impact that used magnitudes of impact as a criteria, which clearly favors quantitative evidence (Donovan 2008)(Donovan 2008).

Despite the shift of focus between quantitative, economic benefits, and qualitative, social benefits, the RFQ never rejected the idea that practical impact was an important part of judging the overall value of research.

The REF and RFQ each combined qualitative and quantitative methods in order to evaluate both the quality of research and its practical impact, in order to assess the overall value of research.