How to Review Registered Reports

What are Registered Reports?

Registered Reports are manuscripts that are reviewed and approved in principle prior to data collection and/or analysis. 

The Registered Reports model is an extension of a pre-registration study and refers to a type of research article. In the format, manuscript writing and review occurs in two stages. There are different forms of Registered Reports, depending on what is being registered (e.g., a planned analysis on data that already exist vs. planned analysis on data that have yet to be gathered).

Stage 1: Peer review of the study protocol, including introduction, methods and analysis plan, occurs before the research is conducted (Chambers, 2019). The decision to publish is made before the study is run and is based on the importance of the research question and rigour of the methods. Authors receive an in-principle acceptance (IPA), a commitment from the journal to publish the study irrespective of the results.   

Stage 2: Providing the researcher follows the Stage 1 protocol and obtains an IPA, the researcher can complete their study knowing that it will be published regardless of the results. Following completion of the study, authors will complete the manuscript, including Results and Discussion sections, and it will be sent for review. Reviewers will check whether the researcher has adhered to the Stage 1 plans, and whether the conclusions reflect the data.   

Stage 2 manuscripts will more closely resemble a regular research article format.

How to review Stage 1 reports

Stage 1 submissions that are judged by the editorial board to be of sufficient quality and rigor will be sent for peer review. In considering papers at the registration stage, you will be asked to assess:

What is the main question being addressed in the study and how important is it to the field? 
Is everything clearly and transparently presented so it is clear what will be done?  
What are the key independent and dependent variable(s), and how they will be measured? 
What are the authors’ hypotheses, if applicable? 
Whether the data can test the authors’ proposed hypotheses  
Is there a coherent connection between theory/hypotheses and methods and analyses? If not, how can they be strengthened? 
How many and which conditions will participants/samples be assigned to? 
How many observations will be collected and what rule will be used to terminate data collection? 
What are the study inclusion criteria? 
What are the data exclusion criteria? 
What positive controls or quality checks are in place to provide a fair test of the stated hypotheses? 
Which analyses will be used to examine the main question/hypothesis? 
For empirical registered reports, where possible, authors should submit the code or syntax they plan to run on the data. That way, data-analytical steps can be better traced by reviewers, and errors can still be spotted. 
Are the authors proposing to collect new data or analyse existing data? This distinction is important for transparency.

Following Stage 1 peer review, manuscripts will be either rejected outright, offered the opportunity to be revised, or accepted. Manuscripts that pass peer review will be issued an IPA, indicating that the article will be published pending successful completion of the study according to the exact methods and analytic procedures outlined, as well as there is a defensible and evidence-bound interpretation of the results, writing is coherent and clear, and the manuscript is formatted according to the journal’s author guidelines.

Please note that any deviation from the stated study procedures, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript. In cases where the pre-registered protocol is altered after IPA due to unforeseen circumstances (e.g. change of equipment or unanticipated technical error), the authors must consult the Editor immediately for advice, and prior to the completion of data collection. Minor changes to the protocol may be permitted according to editorial discretion. In such cases, the IPA would be preserved and the deviation reported in the Stage 2 submission. If the authors wish to alter the study procedures more substantially following an IPA but still wish to publish their article as a Registered Report then the manuscript must be withdrawn and resubmitted as a new Stage 1 submission. Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in a final manuscript (see below).

How to review Stage 2 reports

The resubmission will ideally be considered by the same reviewers as in the Stage 1 submission but could also be assessed by fresh reviewers. In considering papers at Stage 2, you will be asked to decide:

Whether the introduction, rationale and stated hypotheses are the same as the approved Stage 1 submission (required) 
Whether the authors adhered precisely to the registered experimental procedures 
Where appropriate, floor and ceiling effects, positive controls or quality checks, etc, should be reported. Failure to pass these conditions may lead to manuscript rejection.  
If there were many and/or serious deviations from the pre-registration
How the findings impact the overall paper and conclusions
Where applicable, whether any unregistered exploratory statistical analyses are justified, methodologically sound, and informative.  
Whether the authors’ conclusions are justified given the data. Please note that editorial decisions will not be based on the perceived importance, novelty, or conclusiveness of the results.  
If the paper written in a coherent and sound manner so as to clearly and efficiently communicate the research procedures and insights.

Crucially, as a reviewer you should note that editorial decisions will not be based on the perceived importance, novelty, or conclusiveness of the results. Thus, while you are free to enter such comments on the record, they will not form a valid basis for editorial decisions.

References

Chambers, C. 2019. What's next for registered reports?. Nature 573(7773), pp. 187-189. (10.1038/d41586-019-02674-6)

Note

Our thanks to Dr Cathy Manning, Registered Reports Editor at Autism, Professor Chris Chambers and the European Journal of Personality’s Editors for their input in these guidelines.

Registered Report Workflow

Diagram showing Registered Reports workflow

You are here