Title |
Studying grant decision-making: a linguistic analysis of review reports
|
---|---|
Published in |
Scientometrics, July 2018
|
DOI | 10.1007/s11192-018-2848-x |
Pubmed ID | |
Authors |
Peter van den Besselaar, Ulf Sandström, Hélène Schiffbaenker |
Abstract |
Peer and panel review are the dominant forms of grant decision-making, despite its serious weaknesses as shown by many studies. This paper contributes to the understanding of the grant selection process through a linguistic analysis of the review reports. We reconstruct in that way several aspects of the evaluation and selection process: what dimensions of the proposal are discussed during the process and how, and what distinguishes between the successful and non-successful applications? We combine the linguistic findings with interviews with panel members and with bibliometric performance scores of applicants. The former gives the context, and the latter helps to interpret the linguistic findings. The analysis shows that the performance of the applicant and the content of the proposed study are assessed with the same categories, suggesting that the panelists actually do not make a difference between past performance and promising new research ideas. The analysis also suggests that the panels focus on rejecting the applications by searching for weak points, and not on finding the high-risk/high-gain groundbreaking ideas that may be in the proposal. This may easily result in sub-optimal selections, in low predictive validity, and in bias. |
X Demographics
Geographical breakdown
Country | Count | As % |
---|---|---|
United Kingdom | 15 | 12% |
Spain | 13 | 11% |
United States | 6 | 5% |
France | 5 | 4% |
Germany | 5 | 4% |
Poland | 5 | 4% |
Denmark | 4 | 3% |
Netherlands | 4 | 3% |
Turkey | 2 | 2% |
Other | 19 | 15% |
Unknown | 45 | 37% |
Demographic breakdown
Type | Count | As % |
---|---|---|
Members of the public | 60 | 49% |
Scientists | 58 | 47% |
Science communicators (journalists, bloggers, editors) | 4 | 3% |
Practitioners (doctors, other healthcare professionals) | 1 | <1% |
Mendeley readers
Geographical breakdown
Country | Count | As % |
---|---|---|
Unknown | 71 | 100% |
Demographic breakdown
Readers by professional status | Count | As % |
---|---|---|
Researcher | 16 | 23% |
Other | 8 | 11% |
Student > Ph. D. Student | 7 | 10% |
Professor | 6 | 8% |
Student > Doctoral Student | 4 | 6% |
Other | 17 | 24% |
Unknown | 13 | 18% |
Readers by discipline | Count | As % |
---|---|---|
Social Sciences | 18 | 25% |
Computer Science | 6 | 8% |
Business, Management and Accounting | 4 | 6% |
Psychology | 4 | 6% |
Agricultural and Biological Sciences | 3 | 4% |
Other | 16 | 23% |
Unknown | 20 | 28% |