Peer review is exciting to study! Again.

2013-03-13 15.17.14

Roehampton campus

Sometime ago I received an invitation from the University of Roehampton to give a seminar. As it is usual in such situations we moved very fast from initial pleasantries, to acceptance and to discussing the topic of the seminar. Now, I have four ‘progressive’ research lines and my guess is that I could have offered to give a perfectly acceptable seminar in any of those – they are interdisciplinary enough and at least two of them can potentially affect a number of neighbouring research fields in the area of science and innovation policy and management.

Instead of choosing ‘acceptable’ I said:

“How about talking about peer review as organisational practice?”

 And I almost heard my kind and considerate host’s smile break on the floor. But peer review it was. I made it my mission to convince the audience that studying peer review doesn’t have to be about its distortions (though one of the most cited works on peer review is still the one about sexism and nepotism); it is not about the pain of every researchers who has submitted a paper to a journal and has had it critiqued; and it certainly isn’t about the expenses that peer review systems inevitably incur.

After the seminar: mission accomplished :)

After the seminar: mission accomplished 🙂

But let me tell you why the study of peer review is not only exciting again but also a very topical problem to play with.

I believe that the renewed interest in the study of peer review originates from three inter-related developments:

a)      The overall shift of funding modality away from block grants to research organisations to project based funding to individuals and research teams.  There are two framework modalities for funding research that target different organisational forms, use different selection principles, provide different opportunities and have different effects on the science system: undifferentiated block grants to large organisations and project based support to principal investigators (PIs) and research teams. These funding modalities usually co-exist and the ratio between them is time and place specific. Data suggests that lately a dramatic shift away from block grant funding to organisations towards project based funding for PIs and research teams (Lepori et al. 2007) has occurred.

b)     Diminished ability of universities and research organisations to provide the ‘playground’ for emergence and development of novel, high risk ideas.  Block funding to organisations, for instance, provides the ‘low entry’ idea generation facility science needs and the stability to grow and test these ideas so that selection could be carried out at a later stage when clear promise can be discerned. These opportunities are well illustrated by two recent cases where a stable organisational environment was reported to have provided the conditions for scientific breakthrough research that was recognised by the highest accolade – the Nobel Prize[1]. One of the effects of the reduction of block grants is that organisations no longer have the resources to ‘grow’ novel and risky ideas.

c)      The stated intention of some research funding agencies to support (and select) path-breaking research. Research funders have already signalled their intention to identify and support risky, path breaking research. Many existing research councils have incorporated the support of ‘path-breaking’ research amongst their missions. Furthermore, we have witnessed the establishment of the first organisation to support investigator driven risky research at European level – the European Research Council (Nedeva et al., 2012; Nedeva & Stampfer, 2012).

Here is the big question:

“Is it possible to use peer review to identify and select path-breaking potential research proposals and, if it is, under what conditions?”

What makes this questions so exciting? Couple of things.

First, it is about using peer review as organisational practice for selection. This is obviously important practically – in the current funding regime(s) what funding agencies select and support today is THE SCIENCE of the future. It is also exciting methodologically because answering this question requires one to link ‘causally’ selection practices and the long term effects of this selection (whether or not the research has come to be considered to be path-breaking).

Second, in light of the question posed one ought to un-pack the nature (and differences) of ‘excellent’ and ‘path-breaking’ research and account for the following: a) ‘excellence’ and ‘path-breaking’ are not properties but attributes of research; b) ‘excellence’ is about consensual judgement whilst ‘path-breaking’ is about selecting outliers; c) ‘excellence’ is constructed through the ‘norm’ and ‘path-breaking’ through exceptionality; and d) ‘excellence’ is about mastery and ‘path-breaking’ about novelty and risk.

In other words, ‘excellence’ and ‘path-breaking’ are not nested notions or sides of a continuum, but they describe different axis of knowledge. Hence, it is possible to have research that is:

  • Excellent but not path-breaking
  • Excellent and path-breaking
  • Not excellent but path-breaking
  • Neither

Peer review as practices at the moment could select excellent research and it could possibly cope with excellent research that may be path-breaking. But it has problems dealing with research that is potentially path-breaking but not yet excellent.

This is why I believe that studying peer review is exciting again.

[1] In 2009 three scientists shared the Nobel Prize for Chemistry – Venkatraman Ramakrishnan, Thomas A. Steitz and Ada E. Yonath – for ‘studies in the structure and function of the ribosome’. In their interviews (, last accessed 4 May 2011) they all mentioned that when they started their research ‘nature was not ready to unveil its secrets’, What allowed them the opportunity to continue was, in the words of Venkatraman Ramakrishnan, ‘the stable and supportive funding environment’ that their organisations provided. Similarly, in 2010 two scientists from the University of Manchester, Andre Geim and Konstantin Novoselov, shared the Nobel Prize in Physics for ‘ground breaking experiments regarding the two dimensional material graphene’. Two conditions enabled this discovery: the ‘play- time’ the members of the laboratory shared every Friday afternoon and a roll of sticky tape (from interview with K. Novoselov). It is also worth noting that since theory predicted that graphene is impossible work on it could have not passed peer review and is highly unlikely to have been funded by any research funding agency.


Leave a Reply

Your email address will not be published. Required fields are marked *