Open data badges: Pilot results, Challenges, and Opportunities
In 2018, Springer Nature launched an Open data badges pilot. The initial pilot has now concluded and we thought it worthwhile to share the lessons learned as well as the opportunities and challenges for rolling out badging more widely throughout Springer Nature.
In 2018, Springer Nature launched an Open data badges pilot. From September 10, 2018 to May 31, 2019, 210 papers were assessed and 65 badges awarded, meaning that 31% of papers were eligible for a badge. As a comparison, we examined the papers published in the three month period before the pilot and found that 20% would have been eligible for a badge. This indicates that there was possibly a positive impact on data sharing due to the pilot, although the majority of badges awarded were for papers in disciplines where there are community expectations that data should be deposited in a publicly available repository (in this case, genetic data). Two badges awarded to papers where data sharing was not mandated by a research community, funder, or institution, in comparison to the pre-pilot assessment where no authors had shared their data without being mandated to do so.
We also examined usage statistics for papers with and without badges to determine if there was a different level of engagement with papers awarded badges. This was based on paper accesses and downloads, click-through rates, and bounce rates (which indicate the percentage of visitors who leave the site after visiting only one page). To assess this, 31 papers with badges were compared to a randomly selected group of 31 papers without badges all published at similar times. We found that overall, papers with badges had higher engagement with a lower bounce rate (60% bounce rate for papers with a badge, and 70% for papers without) and a higher download rate (53 clicks to download papers with badges versus 12 clicks). Papers with badges also had a higher access rate (695 accesses on average) than those without badges (402 accesses on average).
Awareness of Pilot
In February 2019, the Open data badges team enlisted the help of the Market Intelligence team and conducted an online survey of authors whose papers had been evaluated for data badges in BMC Microbiology. The goal was to determine what impact the pilot had on data sharing practice. Authors were asked about their awareness of the pilot and their opinions on data sharing generally.
Respondents to our author survey largely supported data sharing practices (17 of 18 stated it was very or moderately important to them). Moreover, 10 of 18 stated that the Open data badges would make them more likely to submit their research to the journal. However, those surveyed were largely unaware of the pilot or their paper’s badge status. Only 1 of the 18 survey respondents stated they had seen messaging about the pilot in the letters they received from the journal. Additionally, 8 stated they were unaware of the pilot, and 6 stated they had seen it mentioned on the website or submission guidelines (1 respondent did not answer this question). More interestingly, only 1 of the 8 of the badge recipients surveyed indicated that they were aware that their paper had received a badge. This led the working group to re-evaluate the current state of the pilot and look into ways to increase the visibility to ensure authors were more aware of the pilot and the badge criteria.
In the second phase of the pilot, we aimed to establish whether increasing the visibility during the submission and revision process would impact the number of badges awarded to papers. To that end, we moved the pilot announcement’s placement in the submission and revision letters to closer to the top of page (it has originally been at the very end) and we added a question to the Author Questionnaire requiring them to acknowledge that they were aware of the badge requirements.
From July 11th 2019 through January 31, 2020 a total of 165 papers were evaluated for badges and 29.6% were awarded badges. We could confirm that 68 of these authors had received the additional notifications in the submission system. Of those papers, a higher percentage were awarded badges (35.4%). While this looked encouraging, we also found that all of the papers awarded badges had existing community mandates for data sharing. No badges were awarded for non-mandated data.
At this point, we decided to discontinue the pilot as it became clear that we were unable to positively determine the impact of the pilot on author data sharing behaviour. We plan to expand to badging to a second journal with a different disciplinary focus, after exploring how automated checking could assist scaling.
This pilot has raised several challenges that will need to be addressed when providing Open data badges, or any other digital badging initiative we launch.
- Authors must be made aware of the badging at submission and there must be follow up communication about the badge criteria to ensure they have the opportunity to comply and to receive a badge.
- Messaging around the badging should provide authors with additional resources they can access for further guidance with respect to appropriate repositories for their data and best practices.
- Authors should be notified of their paper’s badge status.
- Visibility of papers with badges should ideally be improved within the online platforms.
- If possible, data availability statements should be reviewed before a manuscript is accepted so it can be queried if necessary, and authors can be given the opportunity to refine or amend it.
As noted above, next steps will focus on investigating options for automating parts of the data availability statement checks and the resourcing needed to continue issuing badges that are verified as meeting the stated criteria. Future badging initiatives will need to address the challenges relating to the visibility of the badge on published papers and ways of measuring engagement with the data sets.
We believe the potential benefits of Open data badges merits further investigation. Based on the feedback from our survey, and our analysis of other journals offering badging for data sharing, we can see that there is clearly an appetite for badging. We presented on the Open Data Badge pilot at two conferences last year: the Force11 conference and the IDCC. Our submission to the IDCC was also accepted for publication, a preprint can be found here. In both cases, there was a lot of interest from various stakeholders in the research community to have badging that does not only rely on self-reporting by authors.
As leaders in open science and open access, Springer Nature wants to distinguish our own badging through validation and verification so our readers can feel confident they will be able to access and use the data. Badging is also an opportunity for Springer Nature to spotlight the data support and curation resources already available for our authors. It is our hope that through further exploration of scaling and automation, we can find ways to encourage more authors from a wide range of disciplines to engage with Open Science, and that we will be able to develop a badging workflow that can potentially be rolled out on a full portfolio of our journals.