graph LR
A[Initial idea] --> B[Draft analysis plan]
B --> C[Propose project to team]
C --> D[Develop IRB / Preregistration, etc.]
Getting started
Analysis plan
An analysis plan is the very first part of any project. A good analysis plan should describe the project in broad strokes, including project motivation, hypotheses, data description, and proposed methods. Though not set in stone–an analysis plan is meant to be flexible–an analysis plan should bear a striking resemblance to the front end of the final article. The point of the analysis plan is to force the team working on it to think deeply about the topic, data, and methods they have at hand and how they can realistically bring those forces to bear to answer an scientific question.
We have created a set of template analysis plans for the Research Ring that cover observational and experimental research projects, the two major types of projects we are likely to undertake. Download the appropriate template below and follow the instructions provided whenever you start a new project.
While not every analysis will require all of the elements in a given template, these templates will be an excellent resource for getting started. Additionally, they will force you to formalize all of the information needed for other downstream documents needed for the study, like study preregistrations and Internal Review Board (IRB) applications. We’ll talk about those below but first we need to talk about what the analysis plan is actually used for: proposing the project to the wider team.
Proposing a project
Usually a project begins with an idea, a quick Google search, and an overabundance of enthusiasm. After confirming with other team members that the project is worthy of pursuit, an analysis plan should be drafted. After an analysis plan is drafted, it should be shared with the whole team in the form of a “project proposal” (as in, the analysis plan is your proposed approach to accomplishing the project). The process looks like this:
Proposing a project like this is useful for several reasons. First, it makes you think concretely about the project (enough to propose it). Second, getting feedback early on (i.e., prior to submitting IRBs or preregistration) is immensely helpful for identifying issues early on (e.g., the core issue is mis-conceptualized) or avoid duplication of effort (e.g., another team mate or research group is already doing something similar). Third, proposing a project allows you to solicit contributions from interested team members.
NoteParticipation
As a rule, anyone on the team is allowed to request participation in a project. As another rule, the lead author should try to accommodate these requests where it makes sense. Here’s the rub: participation should be based on contributions. Project teams should be kept lean, with every person making a substantive contribution (no freebies).
At the end of every project proposal (i.e., after discussing/ironing out all of the major details), the proposer should should provide two things: 1) tentative timeline and 2) a needs statement. Regarding the latter, a needs statement is just when the proposer says what they will need to complete the project (e.g., materials, funds, support from the team). This is also the point when the proposer lists all of the team members they envisioned helping on the project and what their role would be. This is a crucial step because this is where vision meets reality. For instance, if you imagined someone helping on the analysis but they do not have the bandwidth to help, plans will need to be adjusted–either a new team mate will need to take on the analysis role or the timeline can be adjusted.
After the initial project proposal
So you’ve proposed a project, received feedback, and gotten the green light to move forward–well done! The next step is usually to formalize your analysis plan in one of two ways (and possibly both!). Analysis plans should be used to 1) post a preregistration for the project and/or 2) submit an IRB application. We’ll breifly talk about each.
Preregistration
For experimental studies, additional steps will be required to develop the analysis plan into a preregistration (this is not optional). We will use the Open Science Framework (OSF) website. The preregistration must be posted publicly prior to any data collection. However, this is more than just a public analysis plan; preregistrations contain several additional elements that are not required for observational studies (i.e., surveys, interviews, secondary data). Extra elements include:
- Power analysis
- Confirmatory hypotheses
- Criteria for positive/negative results
When posting a preregistration on the OSF website, make sure to use the ALERRT-research account. OSF offers several resources that can help in formalizing a preregistration, much like the templates above. However, we will generally not use these as they can be too rigid and typically contain alot of material that is not actually needed for a good preregistration–they are generally designed fit many possible research designs so they end up fitting no particular research design well. Instead, use the analysis plan you began at the start of your project and add in the elements listed above to ensure that it covers all of the necessary topics.
IRB applications
Most observational studies and all experimental studies will require IRB approval. Development of the analysis plan will almost always provide all the required elements for an IRB application—conversely, anything required in an IRB application should be included in a project’s analysis plan. To submit an IRB proposal to Texas State University, you’ll need to access the Kuali System.