Adding productivity indicator

vtraag · vtraag · commit 06c2d8729993 · 2024-12-12T10:37:09.000+01:00
diff --git a/indicator_templates/quarto/2_academic_impact/productivity.qmd b/indicator_templates/quarto/2_academic_impact/productivity.qmd
@@ -0,0 +1,86 @@
+---
+author:
+    - name: V.A Traag
+      orcid: 0000-0003-3170-3879
+      affiliations:
+      - ref: cwts
+
+affiliations:
+- id: cwts
+  name: Leiden University
+  department: Centre for Science and Technology Studies
+  city: Leiden
+  country: the Netherlands
+---
+
+# Productivity {#productivity .unnumbered}
+
+<div>
+
+## History
+
+| Version | Revision date | Revision    | Author     |
+|---------|---------------|-------------|------------|
+| 1.0     | 2024-12-06    | First draft | V.A. Traag |
+
+</div>
+
+## Description
+
+In general, productivity estimates the amount of output relative to the amount of input. In the context of academia, outputs can be various objects, varying from publications to data, code, or peer reviews. Although productivity is an aspect of interest, it should usually be considered jointly with something like quality. That is, a higher productivity may just stimulate more, but lower quality, outputs. There is some evidence of such a type of effect [@butler_explaining_2003], although this evidence is also disputed [@van_den_besselaar_perverse_2017].
+
+Output is usually only measured for a limited set of objects, with scholarly publications being the most typical example. Nonetheless, other relevant outputs should not be ignored, and limitations of productivity based on publications should be considered. Moreover, we should be aware of certain potential differences between productivity at the individual level and the collective level. For instance, consider a research group for which one individual is tasked with data quality assurance and code review. That individual might perhaps have a lower productivity in terms of publication outputs, yet her/his activities are a boon to the other researchers in the group, whose productivity might greatly increase as a result [@tiokhin_shifting_2023].
+
+In addition, one aspect of productivity that is usually missing is the overall input [@abramo_farewell_2016]. That is, we typically do not know how many people are employed at a certain institution. Even if part of that becomes visible in authorships, not every employee's contribution will become visible in authorship. Hence, institutions that have for example more research assistants who are not acknowledges as author may seem to have relatively few authors, but in reality there are much more people active at the institution. Moreover, even if we know whether a particular author as affiliated with a certain institution, we do not know the amount of time (s)he spends at that affiliation, which is particularly challenging with multiple affiliations. Going one step further, the input could also be specified in financial terms. Unfortunately, none of this data is typically available [@waltman_elephant_2016]. Nonetheless, this is an important limitation to taken into account when considering productivity.
+
+### Avg. number of papers per author
+
+#### Measurement
+
+For a certain institutions $i$ we can count how many authors $a_i$ are affiliated with institution $i$ and how many publications $n_i$ are published in a given year $y$. The ratio of $\frac{n_i}{a_i}$ then gives the average number of papers per author, which is an indicator of productivity. We typically observe an increase in productivity over time, such that in more recent years, the number of papers per author is usually larger than in earlier years.
+
+One relevant aspect in the context of counting number of papers per author is the increase in collaboration. If the total amount of publications remains the same in a given year, but more of them are co-authored, then the metric will be higher. Hence, it sometimes makes sense to use "fractional counting" for publications [@waltman2015]. This means that we can consider fractions, or weights, for all publications, based on the "fraction" of their authorship. For instance, if a publication has three authors: each has a fraction of 1/3. If two of the authors are affiliated with a single institution, say institution A, that institution will have a weight of 2/3. If, in addition, the third author would have two affiliations, one with the aforementioned institution A, and one with institution B, we could count that author as belonging to institution A for 1/2, bringing the total to 5/6.
+
+If we indicate $n_{ji}$ the fraction to which publication $j$ belongs to institution $i$, we can define $n'_i = \sum_j w_{ji}$ the number of fractionally counted publications. Similarly, if we indicate with $a_{ji}$ the fraction with which author $j$ belongs to institution $i$, we can define the fractionally counted number of authors as $a'_{i} = \sum_j a_{ji}$. The productivity can then be simply specified as $\frac{n'_i}{a'_i}$.
+
+If there is input data available, such that the total amount of budget of fte available is indicated by $f_i$, the average number of publications per currency unit or fte can be expressed as $\frac{n_i}{f_i}$.
+
+## Datasources
+
+### OpenAlex
+
+[OpenAlex](https://openalex.org/) covers publications based on previously gathered data from Microsoft Academic Graph, but mostly relies on Crossref to index new publications. OpenAlex offers a user interface that is at the moment still under active development, an open API, and the possibility to download the entire data snapshot. The API is rate-limited, but there are options of having a premium account. Documentation for the API is available at <https://docs.openalex.org/>.
+
+It is possible to retrieve the number of authors for a particular publication in OpenAlex, for example by using a third-party package for Python called `pyalex`.
+
+``` python
+import pyalex as alx
+alx.config.email = "mail@example.com"
+w = alx.Works()["W3128349626"]
+
+authors = w["author"]
+institutions = w["institutions"]
+countries = w["countries"]
+```
+
+Based on this type of data, the above-mentioned metrics can be calculated. When large amounts of data need to be processed, it is recommended to download the full [data snapshot](https://docs.openalex.org/download-all-data/snapshot-data-format), and work with it directly.
+
+OpenAlex provides disambiguated authors, institutes and countries. The institutions are matched to [Research Organization Registry (ROR)](https://ror.org/), the countries might be available, even if no specific institution is available.
+
+### Dimensions
+
+[Dimensions](https://app.dimensions.ai/discover/publication) is a bibliometric database that takes a comprehensive approach to indexing publications. It offers limited free access through its user interface. API access and access through its database via Google BigQuery can be arranged through payments. It also offers the possibility to apply for access to the API and/or Google BigQuery for [research purposes](https://www.dimensions.ai/request-access/). The API is documented at <https://docs.dimensions.ai/dsl>.
+
+The database is closed access, and we therefore do not provide more details about API usage.
+
+### Scopus
+
+[Scopus](https://www.scopus.com/) is a bibliometric database with a relatively broad coverage. Its data is closed and is generally available only through a paid subscription. It does offer the possibility to apply for access for research purposes through the [ICSR Lab](https://www.elsevier.com/insights/icsr/lab). Some additional documentation of their metrics is available at <https://www.elsevier.com/products/scopus/metrics>, in particular in the Research Metrics Guidebook, with documentation for the dataset available through ICSR Lab being available separately.
+
+The database is closed access, and we therefore do not provide more details about API usage.
+
+### Web of Science
+
+[Web of Science](https://webofscience.com/) is a bibliometric database that takes a more selective approach to indexing publications. Its data is closed and is only through a paid subscription.
+
+The database is closed access, and we therefore do not provide more details about API usage.
diff --git a/references.bib b/references.bib
@@ -12,6 +12,20 @@ @article{aagaard_considerations_2017
   pages   = {923--926}
 }
 
+@article{abramo_farewell_2016,
+  title    = {A farewell to the {MNCS} and like size-independent indicators},
+  volume   = {10},
+  issn     = {1751-1577},
+  doi      = {10.1016/j.joi.2016.04.006},
+  abstract = {The arguments presented demonstrate that the Mean Normalized Citation Score (MNCS) and other size-independent indicators based on the ratio to publications are not indicators of research performance. The article provides examples of the distortions when rankings by MNCS are compared to those based on indicators of productivity. The authors propose recommendations for the scientometric community to switch to ranking by research efficiency, instead of MNCS and other size-independent indicators.},
+  number   = {2},
+  journal  = {Journal of Informetrics},
+  author   = {Abramo, Giovanni and D'Angelo, Ciriaco Andrea},
+  month    = may,
+  year     = {2016},
+  pages    = {646--651}
+}
+
 @article{aksnes2019,
   title   = {Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories},
   author  = {Aksnes, Dag W. and Langfeldt, Liv and Wouters, Paul},
@@ -224,6 +238,7 @@ @techreport{brown2016
   langid = {en}
 }
 
+
 @article{bryan2021,
   title   = {The impact of open access mandates on invention},
   author  = {Bryan, Kevin A. and Ozcan, Yasin},
@@ -237,7 +252,6 @@ @article{bryan2021
   note    = {Publisher: MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info {\ldots}}
 }
 
-
 @article{budi2022,
   title   = {Understanding the meanings of citations using sentiment, role, and citation function classifications},
   author  = {Budi, Indra and Yaniasih, Yaniasih},
@@ -250,6 +264,19 @@ @article{budi2022
   doi     = {10.1007/s11192-022-04567-4}
 }
 
+@article{butler_explaining_2003,
+  title   = {Explaining {Australia}'s increased share of {ISI} publications—the effects of a funding formula based on publication counts},
+  volume  = {32},
+  issn    = {0048-7333},
+  doi     = {10.1016/S0048-7333(02)00007-0},
+  number  = {1},
+  journal = {Res. Policy},
+  author  = {Butler, Linda},
+  month   = jan,
+  year    = {2003},
+  pages   = {143--155}
+}
+
 @article{carlin2023,
   title   = {Where is all the research software? An analysis of software in UK academic repositories},
   author  = {Carlin, Domhnall and Rainer, Austen and Wilson, David},
@@ -340,6 +367,7 @@ @misc{codeof
   langid = {en}
 }
 
+
 @article{cohen2002,
   title   = {Links and Impacts: The Influence of Public Research on Industrial R&D},
   author  = {Cohen, Wesley M. and Nelson, Richard R. and Walsh, John P.},
@@ -355,6 +383,7 @@ @article{cohen2002
   langid  = {en}
 }
 
+
 @article{colavizza2020,
   title   = {The citation advantage of linking publications to research data},
   author  = {Colavizza, Giovanni and Hrynaszkiewicz, Iain and Staden, Isla and Whitaker, Kirstie and McGillivray, Barbara},
@@ -371,7 +400,6 @@ @article{colavizza2020
   langid  = {en}
 }
 
-
 @article{cole_societal_2024,
   title      = {The societal impact of {Open} {Science}: a scoping review},
   volume     = {11},
@@ -388,7 +416,6 @@ @article{cole_societal_2024
   pages      = {240286}
 }
 
-
 @book{cost-ben2018,
   title     = {Cost-benefit analysis for FAIR research data: cost of not having FAIR research data},
   year      = {2018},
@@ -1037,6 +1064,7 @@ @book{huyer2020
   langid    = {eng}
 }
 
+
 @article{istrate,
   title  = {A large dataset of software mentions in the biomedical literature},
   author = {Istrate, Ana-Maria and Li, Donghui and Taraborelli, Dario and Torkar, Michaela and Veytsman, Boris and Williams, Ivana},
@@ -1057,7 +1085,6 @@ @inproceedings{jackson2016
   langid    = {en}
 }
 
-
 @inproceedings{jacob2019,
   title  = {FAIR principles, an new opportunity to improve the data lifecycle},
   author = {Jacob, Daniel},
@@ -1083,6 +1110,7 @@ @article{janssens
   langid = {en}
 }
 
+
 @article{johnston2017,
   title   = {Contemporary Guidance for Stated Preference Studies},
   author  = {Johnston, Robert J. and Boyle, Kevin J. and Adamowicz, {Wiktor (Vic)} and Bennett, Jeff and Brouwer, Roy and Cameron, Trudy Ann and Hanemann, W. Michael and Hanley, Nick and Ryan, Mandy and Scarpa, Riccardo and Tourangeau, Roger and Vossler, Christian A.},
@@ -1107,7 +1135,6 @@ @book{jung2023
   url    = {https://cran.r-project.org/web/packages/scrutiny/index.html}
 }
 
-
 @article{keller2014,
   title   = {Re-use of public sector information in cultural heritage institutions},
   author  = {Keller, Paul and Margoni, Thomas and Rybicka, Katarzyna and Tarkowski, Alek},
@@ -1985,6 +2012,7 @@ @inbook{roberts2013
   url       = {https://api.taylorfrancis.com/content/chapters/edit/download?identifierName=doi&identifierValue=10.4324/9780203824696-29&type=chapterpdf}
 }
 
+
 @inbook{roberts2013a,
   title     = {Scientific literacy/science literacy},
   author    = {Roberts, Douglas A.},
@@ -2009,7 +2037,6 @@ @article{robinson-garcia2017
   url     = {https://www.sciencedirect.com/science/article/pii/S1751157717300834}
 }
 
-
 @article{robinson-garcia2020,
   title   = {Open Access uptake by universities worldwide},
   author  = {Robinson-Garcia, Nicolas and Costas, Rodrigo and van Leeuwen, Thed N.},
@@ -2248,6 +2275,20 @@ @article{tennant2016
   url    = {https://f1000research.com/articles/5-632}
 }
 
+@article{tiokhin_shifting_2023,
+  title    = {Shifting the {Level} of {Selection} in {Science}},
+  issn     = {1745-6916},
+  doi      = {10.1177/17456916231182568},
+  abstract = {Criteria for recognizing and rewarding scientists primarily focus on individual contributions. This creates a conflict between what is best for scientists’ careers and what is best for science. In this article, we show how the theory of multilevel selection provides conceptual tools for modifying incentives to better align individual and collective interests. A core principle is the need to account for indirect effects by shifting the level at which selection operates from individuals to the groups in which individuals are embedded. This principle is used in several fields to improve collective outcomes, including animal husbandry, team sports, and professional organizations. Shifting the level of selection has the potential to ameliorate several problems in contemporary science, including accounting for scientists’ diverse contributions to knowledge generation, reducing individual-level competition, and promoting specialization and team science. We discuss the difficulties associated with shifting the level of selection and outline directions for future development in this domain.},
+  language = {en},
+  urldate  = {2024-09-26},
+  journal  = {Perspectives on Psychological Science},
+  author   = {Tiokhin, Leo and Panchanathan, Karthik and Smaldino, Paul E. and Lakens, Daniël},
+  month    = aug,
+  year     = {2023},
+  pages    = {17456916231182568}
+}
+
 @article{tomkins_reviewer_2017,
   title   = {Reviewer bias in single- versus double-blind peer review},
   volume  = {114},
@@ -2288,6 +2329,20 @@ @article{traag2021
   langid  = {en}
 }
 
+@article{van_den_besselaar_perverse_2017,
+  title   = {Perverse effects of output-based research funding? {Butler}'s {Australian} case revisited},
+  volume  = {11},
+  issn    = {1751-1577},
+  doi     = {10.1016/j.joi.2017.05.016},
+  number  = {3},
+  journal = {J. Informetr.},
+  author  = {van den Besselaar, Peter and Heyman, Ulf and Sandström, Ulf},
+  month   = aug,
+  year    = {2017},
+  note    = {Publisher: Elsevier Ltd},
+  pages   = {905--918}
+}
+
 @book{venturini2021,
   title     = {Controversy Mapping: A Field Guide},
   author    = {Venturini, Tommaso and Munk, Anders Kristian},
@@ -2312,6 +2367,20 @@ @incollection{vohland_citizen_2021
   pages     = {35--53}
 }
 
+@article{waltman_elephant_2016,
+  title      = {The elephant in the room: {The} problem of quantifying productivity in evaluative scientometrics},
+  volume     = {10},
+  issn       = {1751-1577},
+  shorttitle = {The elephant in the room},
+  doi        = {10.1016/j.joi.2015.12.008},
+  number     = {2},
+  journal    = {Journal of Informetrics},
+  author     = {Waltman, Ludo and van Eck, Nees Jan and Visser, Martijn and Wouters, Paul},
+  month      = may,
+  year       = {2016},
+  pages      = {671--674}
+}
+
 @article{waltman_field_2019,
   title   = {Field {Normalization} of {Scientometric} {Indicators}},
   doi     = {10.1007/978-3-030-02511-3_11},
@@ -2404,6 +2473,7 @@ @misc{whatper
   langid = {en}
 }
 
+
 @article{wilkinson2016,
   title   = {The FAIR Guiding Principles for scientific data management and stewardship},
   author  = {Wilkinson, Mark D. and Dumontier, Michel and Aalbersberg, IJsbrand Jan and Appleton, Gabrielle and Axton, Myles and Baak, Arie and Blomberg, Niklas and Boiten, Jan-Willem and da Silva Santos, Luiz Bonino and Bourne, Philip E. and Bouwman, Jildau and Brookes, Anthony J. and Clark, Tim and Crosas, {Mercè} and Dillo, Ingrid and Dumon, Olivier and Edmunds, Scott and Evelo, Chris T. and Finkers, Richard and Gonzalez-Beltran, Alejandra and Gray, Alasdair J. G. and Groth, Paul and Goble, Carole and Grethe, Jeffrey S. and Heringa, Jaap and {{\textquoteright}t Hoen}, Peter A. C. and Hooft, Rob and Kuhn, Tobias and Kok, Ruben and Kok, Joost and Lusher, Scott J. and Martone, Maryann E. and Mons, Albert and Packer, Abel L. and Persson, Bengt and Rocca-Serra, Philippe and Roos, Marco and van Schaik, Rene and Sansone, Susanna-Assunta and Schultes, Erik and Sengstag, Thierry and Slater, Ted and Strawn, George and Swertz, Morris A. and Thompson, Mark and van der Lei, Johan and van Mulligen, Erik and Velterop, Jan and Waagmeester, Andra and Wittenburg, Peter and Wolstencroft, Katherine and Zhao, Jun and Mons, Barend},
@@ -2436,6 +2506,7 @@ @article{wilkinson2016a
   langid  = {en}
 }
 
+
 @article{wilner,
   title  = {Complete recovery of values in Diophantine systems (CORVIDS)},
   author = {Wilner, Sean and Wood, Katherine and Simons, Daniel J.},
@@ -2453,6 +2524,7 @@ @book{wood2021
   note   = {original-date: 2018-01-29T16:15:29Z}
 }
 
+
 @article{woods2022,
   title   = {Incentivising research data sharing: a scoping review},
   author  = {Woods, Helen Buckley and Pinfield, Stephen},
@@ -2466,7 +2538,6 @@ @article{woods2022
   url     = {https://wellcomeopenresearch.org/articles/6-355/v2}
 }
 
-
 @article{wuchty2007,
   title   = {The Increasing Dominance of Teams in Production of Knowledge},
   author  = {Wuchty, Stefan and Jones, Benjamin F. and Uzzi, Brian},
@@ -2492,7 +2563,6 @@ @article{yarkoni2019
   url    = {https://psyarxiv.com/jqw35/}
 }
 
-
 @article{zahedi2017,
   title   = {Mendeley readership as a filtering tool to identify highly cited publications},
   author  = {Zahedi, Zohreh and Costas, Rodrigo and Wouters, Paul},
