Mega-consortia in the COVID-19 pandemic
The COVID-19 pandemic has transformed people's lives, with the need for lockdowns and social distancing. The researcher's practice is no exception to this. With restrictions on in-person lab work, research meetings and conferences, in 2020 many researchers have turned to making use of previously gathered data, and remote collaborations. In this situation, digitalization of various aspects of research practice and the development of information and communication have become key. In the past few decades, digital transformation has been steadily progressing. But never has the change in the research practices of many researchers been as dramatic as in the past year. This includes remote experimentation, teleconferencing, information sharing, collaborative document editing, and more. These changes in research practices have facilitated the implementation of mega-collaborations that bring together many researchers who remain at home.In the field of psychology, huge projects have been launched in response to COVID-19, such as The Psychological Science Accelerator's Rapid-Response COVID-19 Project (PSACR) and MB-AtHome project of ManyBabies. One of these projects is COVIDiSTRESS (For more information about this project, please refer to our previous post). In this post, we will discuss the methods and tools used, the various lessons learned, and possible solutions in the process of managing and participating in the COVIDiSTRESS consortium.
Considerations for massive, impromptu international collaborations
The following tools were used in the COVIDiSTRESS project:
- Slack & email for communication
- Doodle for scheduling meetings
- Zoom for online meetings
- Google Documents and Spreadsheets for collaborative writing
- Qualtrics for data collection
- Open Science Framework (OSF) and Github for data storage
While digital platforms make large scale, international and interdisciplinary collaborations such as the COVIDiSTRESS Consortium possible, the free (or already institutionally purchased) platforms (such as Zoom and Qualtrics) that we utilized require design thinking to harness them for collaboration.
A preference for staggered communication
The premise of our collaboration was to be as transparent as possible in all communication so all who would like to contribute could do so in all stages of the project. While platforms such as Zoom made large-scale remote meetings feasible, differences in time zones meant that most collaborations were conducted via staggered messages in digital work platforms, such as Slack, or via email. Compared to e-mail, Slack was viewed as ideal both for allowing topical discussions to take place in dedicated channels and for transparency of the communications. However, the nested and time-order nature of communications in the Slack interface meant that additional management was needed to facilitate whole group communication - such as manually connecting discussions by referring to and consolidating with relevant discussions elsewhere. Furthermore, the free version of Slack only allowed us access to the most recent 10,000 postings, so we lost our conversations during the first couple of months of the project period that might contain useful information about how we worked on the initiation of the project.
Online meetings, on the other hand, were viewed as essential “putting a face to a name” and engendering trust. The few online meetings we did conduct generally had varied attendance. At such a large scale, moreover, it soon became evident that essential, technical decisions had to be made in smaller-group deliberations. Notably, most participants are either part-time or full-time academics who use online platforms (Zoom particularly) on a daily basis due to their teaching duties. Considering that they have been living in either partial or full lockdown and spending most of their time in front of the screen, additional Consortium-related meetings had the potential to contribute to the “Zoom-fatigue” - this likely contributed to smaller numbers of attendees.
In contrast, other aspects of the project such as writing and analysis went comparatively smoothly. We used Google Sheets for survey design, Google Documents for writing, and Github for data cleaning. These tools worked well when a smaller group of researchers collaborated on a more specific task. Division of labour into these smaller groups facilitated the process and helped the project to progress. Based on individual skills and preferences, researchers efficiently decided on their own individual responsibilities, and formed smaller teams for data cleaning, data analysis, etc.
The inclusiveness of the initial structure also led to a security issue - at one point survey items in a shared working document were observed to be reordered at random by an unknown editor. This incident caused the initial open format to be revised in favor of using a few “gatekeeper” team members who held editorial rights to the documents, whereas all other collaborators made inputs as comments and suggestions. However, these editorial adjustments led to additional administrative work. With an ideal tool, such problems may better be resolved with well-developed version control.
Cultural differences in initiative- and credit-taking
While the language of communication was English, differences in understanding, training, and accepted norms across disciplines and cultures lead to some misunderstandings throughout the work. For example, the norms of turn-taking in interaction differ between cultures. This could lead to some members feeling interrupted and not being heard when someone talked at the same time as them, even if there was no negative intent from the person interrupting. In most instances, misunderstandings were resolved with team members sensitively stepping in to clarify, placate, and problem solve. It is possible that not all misunderstandings were aired, resulting in some members ceasing participating due to language barriers or unfamiliarity with the work structure and platforms. In hindsight, some of the miscommunication could have been mitigated with more clearly spelled-out rules or guidelines of collaboration and of the chosen work environment. In terms of communication, we want to emphasize that the need for patience and a positive, shared collaborative spirit should not be underestimated.
For a project involving a large number of collaborators, personal initiative both for conducting the work and for claiming credit, is often required from individual participants. Credit structure needs to be clearly communicated. While rules for authorship are generally clearly laid out by journals, more assurance as to its pragmatic implementation is necessary for fostering trust among diverse teams of researchers. This is particularly true for a collaboration where most of the authors have never met, and as many are from disparate fields of study, are also unlikely to encounter each other in future professional occasions. In projects with large diverse collaborators, more effort must be expended by project leaders to foster an atmosphere of open communication and collegiality. We found it helpful to have clearly delineated agreements and transparent task-lists where individual contributions could be collectively observed. However, it is likely the entire effort would be impossible if not for the PI’s communications containing a very human touch. For example, in one email, Andreas, the PI of the COVIDiSTRESS consortium, addressed the group by referencing a Danish fall potato beet holiday, which was both a way of connecting through culture and boosting morale.
Morale boosting mechanisms were necessary to maintain momentum. For example, one of the early stages of the work involved making a logo-collage of all the collaborator’s institutions (see the bottom part of our website: https://covidistress.github.io/about/), and regular posting of the data-collection numbers as they began increasing. We found that in terms of clarity and morale boosting, assurances both from other participants and project leaders that members are pulling their weight are necessary. Due to differences in professional/cultural background, it cannot be assumed that all participants are aware of what constitutes their right to be included, nor that they are proficient in putting themselves forward for recognition.
In addition, while consortium papers of this nature are one of the most effective and valuable methods to obtain normalized, consistent data across cultures and regions, some consortium members feared that their considerable contributions would not be recognized by their own institutions. We argue that in order to obtain a data-set of this caliber, a number of researchers each had to put forth the same amount of effort that would have been required on a series of their own local projects. The amount of collaborative work required here was greater, and therefore, the team was larger than average. We firmly believe that each team member should receive recognition for their individual contributions. To ensure recognition, we made sure to only submit papers to journals that accept and fully recognize consortium authors. In addition, we set tiers for authorship based on contribution. People with greater contributions were considered to be in a higher tier and were placed higher on the author list. In addition, we have standardized and visualized each contribution according to the CRediT (Contributor Roles Taxonomy). By considering recognition like this, we hope to improve the incentive for individual researcher participation in consortium projects, which in turn, will improve the quality of diverse, inclusive, and reliable global research.
Working with a large international consortium brings about several ethical issues. Firstly, even though the project as a whole will obtain an umbrella ethics approval, some countries or home institutions require additional ethical approval of their own. Secondly, even though the scales and items used are developed to be as universal as possible there are wording and concept issues - particularly as most scales and items are developed by western researchers for a western context. This in itself can create an ethical issue, as we should not ask participants to spend time on concepts that are not clear, understandable, or relevant to them. For example, having a political item asking participants to rate themselves on a left-right wing political scale does not correspond conceptually for all countries. In trying to avoid these issues to be able to account for as many countries as possible some exclusions of details are inevitable and the resulting data set will be more general than specific. Furthermore, some items that could be problematic from a legal perspective (due to differences in national laws) need to be excluded from the entire survey or from local questionnaires if it only applies to one or a few countries.
Localization and translation
The survey also needed to be translated into one of the languages of each country in which it would be administered, a total of 47 languages for our study. In order for results to be comparable across languages, it was critical that translations were equivalent, which requires a team of translators. This is another benefit of a large-scale consortium. The consortium was divided into teams for each language. We then followed the WHO recommended guidelines for translation (https://www.who.int/substance_abuse/research_tools/translation/en/). This included three sets of translators: one sub-team who created a forward-translation from English into another language, one who then translated the new survey back into English, and a third team who verified that the two English versions were equivalent before the team agreed on final adjustments. These teams of translators ranged from at least three, but sometimes quite large teams. We used validated scales across languages wherever possible.
In some cases, a given language was different across countries, consequently, we needed to create multiple versions of the same language. For instance, there were Spanish versions for Argentina, Colombia, Cuba, Mexico, and Spain. In a similar way, researchers from Bosnia, Croatia, and Serbia translated the survey together and adapted small language differences to fit the national contexts.
Another minor but significant barrier was the involvement of html in the text to be translated. For example, when creating a form in Qualtrics, it is better to create the text with html tags to get a better appearance, but this gets in the way when translating. Translators want to process only the plain text, but form creators then have to implement 100s of items for dozens of languages. Our coders tried to give the text to the translators with html so that it could be easily copied and pasted after translation. This forces the translators to do a careful and tedious job of entering the translated text into cells on the spreadsheet while avoiding html tags (and possibly CSS). Furthermore, translators from all over the world do not always understand these tags and codes. To improve these conditions while achieving more accurate translations, a new translation text editor dedicated to this purpose would need to be developed. In fact, Qualtrics has a module for translation, but this is for single-item revision and is not suitable for concurrent collaboration, back-translation/verification, or unified management across languages.
Future development of research environment to facilitate mega-collaborations
The onset of the COVID-19 disaster has given rise to many large-scale international collaborations, and researchers have managed to work through them by opportunistically seeking ways to run their projects more efficiently, functionally, cooperatively, flexibly, accurately, and familiarly. However, such an individual-based approach is not guaranteed to work for all time, and the more extensive the project, the more significant the loss of global resources will be if it is not successful. Clearly, we need a comprehensive and integrated environment for conducting international mega-collaborations. For example, it would include a platform for launching new projects and recruiting participating researchers: the only recruitment channels we have now are social networking sites and word of mouth. A seamless system that integrates online survey and experiment forms for data collection, large and international crowdsourcing panels, editors dedicated to translation, diverse programming libraries and analysis processes should also be implemented. A registration system should be available to check the core hours of participating researchers at a glance; knowing who was able to work during specific hours was extremely important for a team crossing many time zones. While it is unclear whether such technological reforms can be put into practice immediately, we believe they represent an opportunity well worth pursuing to fundamentally improve psychological, social, and behavioral science.