Research evaluation

From LiquidPubWiki

In this section we focus our efforts on the creation of metrics, indexes, tools, and best practices that could be included in the performance module of the LiquidPub web platform. We would like to give the user the possibility to evaluate:

• Individual authors evaluation : assessment of individuals who have authored, or have been co-authors of, one or more publications
• Reviewers evaluation : identifying biases in the behavior of reviewers with respect to factors such as nationality, affiliation, and gender, and provide the means to compensate for such biases.
• Groups evaluation : clusters of related people, such as departments, universities, research organizations, and collaborations
• Research Artifacts evaluation : published research object evaluation methods will be discussed, with particular attention on the peer review system.
• Journals evaluation : how journals can be evaluated, and how publishing in "good" journals influences the evaluation of researchers and articles.
• Conferences evaluation : similarly to the journal case, we analyze how we can assess conferences and what are the advantages of publishing in "good" conferences.

Given that data about each of the entities listed above are available, for each of them we also might have different metrics and algorithms for the assessment. Our idea is to have a common black board on which everybody could write his/her own thought about possible aspects of the evaluation that should be pointed out. Hopefully, we'll come up with possible modifications of existing metrics or with new metrics that might be suitable for our purpose. Furthermore, we would like to take advantage of the different background that characterizes every partner who joined the LiquidPub project: indeed, since different disciplines also have different habits concerning both publication and research assessment, we encourage computer scientists, philosophers, economists, etc. to contribute to the discussion reporting the common criteria applied with respect to their particular area of interest. We will proceed following two directions, trying to find how the legacy evaluation in the "solid" publication system could be improved (with respect to each of the entities listed above), and, after that, taking advantage of the experience accumulated in the assessment of "solid" artifacts, how the process can be modelled to be suitable to our "liquid" proposal.

State of the Art

In this section we will briefly list a selected set of core papers that might be used as a starting points in this research area. We want to state clearly what are the problems we are facing concerning the evaluation of research, what kind of approaches have been adopted in the recent past, good and bad points about them and, finally, which are the "real" benefits we will get by improving the current methodology followed in the research assessment. The aim here is to encourage collaboration by providing background information contained in seminal works written by the most important researchers in this field, in order to make sure that we are talking about the same things and working at the same level.

• Peer review
  • Cicchetti D. - The reliability of peer review for manuscript and grant submission - Behavioral and Brain Sciences, 14(1), 1991, pag. 119-186
  • Harnad S. - Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals - 1996, Scholarly Publication: The Electronic Frontier. Cambridge MA: MIT Press
  • Horrobin D.F. - The philosophical basis of peer review and the suppression of innovation - JAMA, 263(10), 1990, pag. 1438-1441
  • Mahoney M.J. - Publication prejudices: An experimental study of confirmatory bias in the peer review system - Cognitive Therapy and Research, 1(2), 1977, pag. 161-175
  • McNutt R.A., Evans A.T., Fletcher R.H., Fletcher S.W. - The effects of blinding on the quality of peer review. A randomized trial - JAMA, 263(10), 1990, pag. 1371-1376
  • Rowland F. - The peer-review process - Learned Publishing, 15(4), 2002, pag. 247-258
  • Wenneras C., Wold A. - Nepotism and sexism in peer-review - Women, Science, and Technology: A Reader in Feminist Science Studies, 2001, pag. 46-52

• Bibliometrics - citation-based metrics
  • Bornmann L., Mutz R., Daniel H.D. - Are There Better Indices for Evaluation Purposes than the h Index? A Comparison of Nine Different Variants of the h Index Using Data from Biomedicine - journal of american society for information science and technology, 59(5), 2008, pag. 830
  • Garfield E. - Citation indexes for science. A new dimension in documentation through association of ideas - International Journal of Epidemiology, 35(5), 2006, pag. 1123
  • Garfield E., Merton R.K. - Citation indexing-its theory and application in science, technology, and humanities - 1979, Wiley New York
  • Moed H.F. - Citation Analysis in Research Evaluation - 2005, Springer
  • Moed H.F., Burger W.J.M., Frankfort J.G., Van Raan A.F.J. - The use of bibliometric data for the measurement of university research performance - Research Policy, 14(3), 1985, pag. 117-130
  • Van Raan A.F.J. - Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods - Scientometrics, 62(1), 2005, pag. 133-143
  • Van Raan A.F.J. - Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups - Scientometrics, 67(3), 2006, pag. 491-502

Basically, the same resources have also been used as reference for the corresponding sections inside our State of the Art document.

Evaluation in the "Liquid world"

WIP

Related prototypes implemented

So far, we have implemented a prototype inspired by already existing tools (like, for instance, Harzing's Publish or Perish), which essentially works as an automated support to the evaluation of research (and, in particular, of papers and of individual authors). However, the lack of widely-accepted multidisciplinary metrics, together with the different nature of the entities involved (reviewers, authors, papers, ...), are major issues concerning research evaluation that might eventually be improved, with the help of tools and other automated means. More details about our tools can be found in the Evaluation prototypes page.