Preregistrations

By. Sara A. Hart, Ph.D.

​            Many people are interested in preregistrations but are worried they don’t know how to do them or will do them wrong. What’s interesting about preregistrations is pretty much everyone who has been to graduate school has completed at least one preregistration: a prospectus! In your prospectus, you lay out your research questions, hypotheses, and plan for how you will collect and analyze your data, and often, you include a power analysis. Preregistration is when a researcher publicly posts their study plans before conducting the study, including aspects such as research questions, hypotheses, sampling plan, independent and dependent variables, and analysis plans. The preregistration is then posted openly on the Internet, with a timestamp, before the study is started.
              You might have also heard of registered reports, which are a type of preregistration that occurs with peer-review. Many of our field’s journals now have registered reports (e.g., Journal of Educational Psychology, Developmental Science; a full list is https://cos.io/rr/). With a registered report, you write the introduction and the methods section, including the proposed analysis and power analysis (making them just like a prospectus), before conducting the study. You then submit this to a journal for Stage 1 review. Review then occurs normally but rather than being focused on the results, it’s focused on the importance of the research question and the proposed methods. If the journal accepts your Stage 1 manuscript, the journal accepts the paper “in principle.” Then, you go and complete the study, write up the remainder of the paper (not changing the introduction or methods other than tense changes), and submit it as a Stage 2 manuscript. At this stage, the reviewers are contacted again to make sure you followed your plan and that the results are appropriately discussed and reported. But a Stage 2 manuscript cannot be rejected *because* of the results, helping reduce publication bias and protecting you if you get null effects (which are unfortunately hard to publish normally).
              Preregistering your study has numerous benefits. First, you might be required to do it if you are a federally funded investigator (it is one of the SEER principles!). Second, preregistration makes your a priori hypotheses clear, allowing you to differentiate between confirmatory and exploratory analyses. Third, preregistration reduces questionable research practices, such as peeking at data, selective reporting (e.g., looking for an effect over many related outcome variables and selecting to publish the one that was statistically significant only), HARKing (Hypothesizing After Results Known), p-hacking, and the like. Fourth, when you do a registered report, you get the benefit of additional smart people working through your research questions and methods with you, increasing the chance of useful reviewer comments that improve your science.
              Are you interested in trying out a preregistration?  Any type of study can be preregistered, including secondary data analyses, reviews, and qualitative work. Fortunately, there is a growing repository of templates you can fill out to help you complete your preregistration. Before starting, try to Google and see if you can find a template to help guide you through the process. I’ve also found other people’s preregistrations and used them as templates when I couldn’t find a standard one. A very handy list of templates that might help you is available https://osf.io/zab38/wiki/home/, and for meta-analysis, check out https://www.crd.york.ac.uk/prospero/ (which walks you through questions and then posts your preregistration in one place).  Complete your preregistration, and then post it anywhere that will time stamp your document, and hopefully give you a DOI, making it citable. I’ve used OSF (https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration), figshare (https://figshare.com/), Prospero (linked above), Society for Research on Educational Effectiveness (SREE) (https://sreereg.icpsr.umich.edu), and AsPredicted (www.aspredicted.org). Then write up your study, using your preregistration as your guide for your analyses! If you’ve made a mistake with your preregistration, just note in your manuscript what is not preregistered and continue on your way.
              My own experiences with preregistrations have been mostly positive. When doing them, it feels as though you are slowing your research process and that can be frustrating. It frontloads a lot of work which we normally reserve for later, and it means a study is not “starting.” But when it comes time to write up the study paper, writing the methods and results is a breeze. I’ve also found that it really can be remarkable how reflecting on your analysis steps can be refreshing, especially firmly being able to differentiate between exploratory and confirmatory analyses. Preregistering every part of your analysis plan can be very difficult, especially for complex analyses. I have a preregistered paper published where I laid out a plan for if my variables were skewed but forgot to give a plan for if they were kurtotic! All I did was simply say explicitly in the paper that I didn’t preregister what I would do, but that I would follow my plan for if I had had skewed data (which fortunately worked to correct my kurtotic variable). I have also found that a preregistration can protect you during the review process from reviewers who want to change your paper with analyses you didn’t plan to do. In a different era, I might have felt tempted to work them into the manuscript, and maybe try to tell a story about why they were there. Now, I can either point to my preregistration and say that those analyses are not part of this study, or I can do the analyses, clearly label it as exploratory (and I’ve even explicitly said that a reviewer requested it in the text!), and just leave it there, not woven into the story. I encourage you to try out doing a preregistration. It will not be as foreign as you might think before doing your first one. It is the same research process you are used to, just in a different order than you are used to.

More resources on preregistrations, and open science more broadly:
van Dijk, W., Schatschneider, C., & Hart, S.A. (2021). Open science in education sciences. Journal of Learning Disabilities, 54(2), 139-152. https://doi.org/10.1177/0022219420945267
​
Cook, B.G., Fleming, J.I., Hart, S.A., Lane, K.L., Therrien, W., van Dijk, W., Wilson, S. (in press). A how-to guide for open-science practices in special education research. Remedial and Special Education. doi: https://doi.org/10.1177/07419325211019100 (open access version, https://edarxiv.org/zmeba/)