|
1 | | -# Open Science Impact Indicator Handbook {.unnumbered} |
| 1 | +--- |
| 2 | +title: Open Science Impact Indicator Handbook |
2 | 3 |
|
3 | | -This is the Open Science Impact Indicator Handbook by PathOS. In this handbook we cover various indicators measuring various aspects around Open Science itself, their academic, societal and economic impacts, and reproducibility. |
| 4 | +author: |
| 5 | + - name: V.A Traag |
| 6 | + orcid: 0000-0003-3170-3879 |
| 7 | + affiliations: |
| 8 | + - ref: cwts |
4 | 9 |
|
5 | | -## Executive summary |
| 10 | +affiliations: |
| 11 | +- id: cwts |
| 12 | + name: Leiden University |
| 13 | + department: Centre for Science and Technology Studies |
| 14 | + city: Leiden |
| 15 | + country: the Netherlands |
| 16 | +--- |
6 | 17 |
|
7 | | -In the PathOS project we take a causal perspective on studying Open Science. This necessitates making a distinction between impact itself, and the effect of Open Science on impact. For instance, we could very well see an Open Source research tool being used frequently by industry. In that sense, the Open Source research tool can be said to have a type of economic impact. However, it could very well be that the research tool would have been similarly used by industry had it been released as closed software under a commercial licence. We are interested in the difference between its actual impact under the Open Science principles and its counterfactual impact under a closed principle. That is, we are interested in the *causal* effect of Open Science on the impact of the science. |
| 18 | +This is the Open Science Impact Indicator Handbook by PathOS. In this handbook we cover various indicators measuring various aspects around [Open Science](sections/1_open_science/introduction_open_science.qmd) itself, their [academic](sections/2_academic_impact/introduction_academic_impact.qmd), [societal](sections/3_societal_impact/introduction_societal_impact.qmd) and [economic impacts](sections/4_economic_impact/introduction_economic_impact.qmd), and [reproducibility](sections/5_reproducibility/introduction_reproducibility.qmd). |
8 | 19 |
|
9 | | -Causal inference is not straightforward. We have included an introductory chapter on causality in science studies to explain some of the challenges around causal inference in Open Science. At the same time, this chapter also divulges the possibilities for inferring causality from observational data. Sometimes, we will learn that it will be impossible to correctly identify a certain causal effect. Although this limits our possible conclusions, we believe it is better to be clear about the impossibility of identifying a causal effect in some cases than to pretend we did identify some causal effect. |
| 20 | +# Executive summary |
10 | 21 |
|
11 | | -The challenge of causal inference also clarifies that we cannot provide straightforward indicators of effects of Open Science. There are many aspects of Open Science, including Open Access, Open Data and Open Code, but also elements such as Citizen Science, Open Science infrastructure, policies and training. At the same time, there are many different types of impacts in each domain of academia, society and economy. This leads to a combinatorial number of possible effects. For each such an effect it necessitates to carefully reason about its causal inference, and what other factors should be controlled for, or should not be controlled for. This is a daunting task, which goes well beyond our capacities. For that reason, the impact indicator handbook provides guidance on how to operationalise various indicators in order to facilitate studies on the effect of Open Science. We hope this is useful to the research community, and that we together face the challenges of causal inference in Open Science studies. |
| 22 | +In the PathOS project we take a causal perspective on studying Open Science. This necessitates making a distinction between impact itself, and the effect of Open Science on impact. For instance, we could very well see an Open Source research tool being used frequently by industry. In that sense, the Open Source research tool can be said to have a type of economic impact. However, it could very well be that the research tool would have been similarly used by industry had it been released as closed software under a commercial licence. We are interested in the difference between its actual impact under the Open Science principles and its counterfactual impact under a closed principle. That is, we are interested in the *causal* effect of Open Science on the impact of the science. |
12 | 23 |
|
13 | | -Finally, not all indicators wil be equally well-developed. Some indicators, like citation impact, are already long established and studied in scientometrics. Other indicators, such as on data usage or reproducibility are much more recent. Such indicators may be under active development, or may actually not be worked on at all yet. We still include such indicators in this handbook if we have identified this indicator to be vital for studying the causal effect of Open Science. This impact indicator handbook should therefore not only be seen as an inventory of what is possible today, but also what we believe is necessary tomorrow. |
| 24 | +Causal inference is not straightforward. We have included a separate section on causality, including an [introduction to causality in science studies](sections/0_causality/causal_intro/article/intro-causality.qmd) to explain some of the challenges around causal inference in Open Science. At the same time, this introduction also divulges the possibilities for inferring causality from observational data. Sometimes, we will learn that it will be impossible to correctly identify a certain causal effect. Although this limits our possible conclusions, we believe it is better to be clear about the impossibility of identifying a causal effect in some cases than to pretend we did identify some causal effect. |
14 | 25 |
|
15 | | -We hope this handbook will be a central hub to keep track of Open Science related indicators. Within the PathOS project the impact indicator handbook will remain in continuous development until january 2025. We hope to be able to contribute to keeping the impact indicator handbook up-to-date also afterwards. The handbook is open to community contributions. Together we may create a central resource that is useful to all. |
| 26 | +The challenge of causal inference also clarifies that we cannot provide straightforward indicators of effects of Open Science. There are many aspects of Open Science, including Open Access, Open Data and Open Code, but also elements such as Citizen Science, Open Science infrastructure, policies and training. At the same time, there are many different types of impacts in each domain of academia, society and economy. This leads to a combinatorial number of possible effects. For each such an effect it necessitates to carefully reason about its causal inference, and what other factors should be controlled for, or should not be controlled for. This is a daunting task, which goes well beyond our capacities. For that reason, the impact indicator handbook provides guidance on how to operationalise various indicators in order to facilitate studies on the effect of Open Science. In addition, we have explored three effects in greater detail, studying the effect of [Open Data on citations](sections/0_causality/open_data_citation_advantage.qmd) and on [Cost Savings](sections/0_causality/open_data_cost_savings.qmd), and [issues of inferring causality in the context of societal impact](sections/0_causality/social_causality.qmd). We hope this is useful to the research community, and that we together face the challenges of causal inference in Open Science studies. |
16 | 27 |
|
17 | | -## Abbreviations |
| 28 | +Finally, not all indicators are equally well-developed. Some indicators, like [citation impact](sections/2_academic_impact/citation_impact.qmd), are already long established and studied in scientometrics. Other indicators, such as on [data usage](sections/2_academic_impact/use_of_data_in_research.qmd) or the [level of replication](sections/5_reproducibility/level_of_replication.qmd) are much more recent. Such indicators may be under active development, or may actually not be worked on at all yet. We still include such indicators in this handbook if we have identified them to be vital for studying the impact of Open Science. This impact indicator handbook should therefore not only be seen as an inventory of what is possible today, but also what we believe is necessary tomorrow. |
18 | 29 |
|
19 | | -| Abbreviation | Description | Explanation | |
20 | | -|--------------|--------------|---------------------------------------------| |
21 | | -| APC | Article Processing Charge | Fee sometimes charged for publishing articles. | |
22 | | -| API | Application programming interface | Interface allowing to make use of certain service or program | |
23 | | -| DMP | Data Management Plan | Document describing how data will be managed | |
24 | | -| DOI | Digital Object Identifier | A DOI is a digital identifier of an object, any object --- physical, digital, or abstract. Core element of academic infrastructure. | |
25 | | -| EOSC | European Open Science Cloud | An initiative aimed at developing an infrastructure to provide users with services promoting Open Science practices. | |
26 | | -| EU | European Union | Supranational political union consisting of member states that are located primarily in Europe | |
27 | | -| FAIR | Findability, Accessibility, Interoperability and Reusability | Principles for academic data management | |
28 | | -| NCI | Normalised Citation Impact | Citation impact that is normalised for field differences | |
29 | | -| IPR | Intellectual Property Rights | Rights to intangible creations of the human mind. | |
30 | | -| NLP | Natural Language Processing | Computer techniques and algorithm to automatically process and analyse natural language | |
31 | | -| OA | Open Access | Principles and practices around distributing research outputs, free from any barriers. | |
32 | | -| OS | Open Science | Practices and principles for making science more open in various ways | |
33 | | -| OSF | Open Science Framework | Platform to support and enable collaboration | |
34 | | -| REST API | Representational state transfer API | Type of API often used for web services | |
35 | | -| UNESCO | United Nations Educational, Scientific and Cultural Organization | Specialized agency of the United Nations (UN) aimed at promoting world peace and security through international cooperation in education, sciences, culture, communication and information. | |
| 30 | +We hope this handbook will be a central hub to keep track of Open Science related indicators. We hope to be able to contribute to keeping the impact indicator handbook up-to-date as part of the PathOS project. The handbook is open to community contributions. Together we may create a central resource that is useful to all. |
36 | 31 |
|
37 | | -## Acknowledgements {.unnumbered} |
| 32 | +# Acknowledgements {.unnumbered} |
38 | 33 |
|
39 | | -This project has received funding from the European Union’s Horizon Europe framework programme under grant agreement No. 101058728. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor the European Research Executive Agency can be held responsible for them. |
| 34 | +The PathOS project has received funding from the European Union’s Horizon Europe framework programme under grant agreement No. 101058728. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor the European Research Executive Agency can be held responsible for them. |
0 commit comments