U3.5 Experimental Design
Master AP Statistics experimental design: identify units, factors, treatments, and response variables, apply comparison, randomization, and replication, and tell apart the three key designs.
What you'll do in this lesson
A voice-first session with the Crimsora tutor on U3.5 Experimental Design, then targeted practice and FRQs — with the tutor adapting to where you get stuck.
What this lesson covers
Experiments let statisticians do something surveys cannot: establish cause and effect. But that power only holds when the design is built correctly. In this lesson you will learn to dissect any experiment into its parts — the experimental units, the factors and their levels, the treatments, and the response variable — and then judge whether the study honors the three pillars of good design: comparison, randomization, and replication.
You will also learn to distinguish the three designs the AP exam loves to test: completely randomized, randomized block, and matched-pairs. Getting these vocabulary words and structures right is worth easy points on both multiple-choice and free-response questions, and it sets up the inference logic you will use later in the course.
You will also learn to distinguish the three designs the AP exam loves to test: completely randomized, randomized block, and matched-pairs. Getting these vocabulary words and structures right is worth easy points on both multiple-choice and free-response questions, and it sets up the inference logic you will use later in the course.
The Anatomy of an Experiment
Every experiment can be broken into a fixed set of parts, and the AP exam expects you to name them precisely.
The experimental units are the individuals or objects to which treatments are assigned. When the units are people, we call them subjects. A factor is an explanatory variable that is deliberately manipulated. Each factor has levels, the specific values it takes. A treatment is a specific combination of factor levels applied to a unit. The response variable is the outcome measured to judge the treatment's effect.
Consider testing whether caffeine dose (0 mg, 100 mg, 200 mg) and study method (flashcards, rereading) affect test scores. There are two factors. Caffeine has three levels, study method has two, so there are treatments. The response variable is the test score.
A common mistake is confusing factors with treatments. If there is only one factor, its levels equal the treatments. With multiple factors, treatments are the combinations. Always count carefully.
The experimental units are the individuals or objects to which treatments are assigned. When the units are people, we call them subjects. A factor is an explanatory variable that is deliberately manipulated. Each factor has levels, the specific values it takes. A treatment is a specific combination of factor levels applied to a unit. The response variable is the outcome measured to judge the treatment's effect.
Consider testing whether caffeine dose (0 mg, 100 mg, 200 mg) and study method (flashcards, rereading) affect test scores. There are two factors. Caffeine has three levels, study method has two, so there are treatments. The response variable is the test score.
| Part | Definition | Example |
|---|---|---|
| Experimental unit | What receives a treatment | Each student |
| Factor | Manipulated explanatory variable | Caffeine dose |
| Level | A value of a factor | 100 mg |
| Treatment | Combination of factor levels | 100 mg + flashcards |
| Response | Measured outcome | Test score |
The Three Principles of Good Design
A well-designed experiment rests on three principles. The exam frequently asks you to name them and, more importantly, to explain why each matters in a specific context.
Comparison means using at least two groups so you can attribute differences to the treatment rather than to time or chance alone. Often one group receives a control — either no treatment, a placebo, or the standard treatment — to serve as a baseline.
Randomization means using a chance process to assign units to treatment groups. This does not guarantee identical groups, but it balances out both known and unknown confounding variables in the long run, creating groups that are roughly equivalent before treatment. Randomization is what lets us make cause-and-effect conclusions.
Replication means applying each treatment to enough experimental units so that real effects can be distinguished from natural variability. Replication is about adequate sample size within the experiment, not about repeating the whole study (though that matters for science generally).
A misconception: students think randomization eliminates confounding. It balances confounders across groups on average; it does not remove them. Another trap is confusing replication with a large population sample — in experiments, replication refers to multiple units per treatment. When an FRQ asks you to 'explain the purpose,' tie the principle to the specific variables in the prompt, not a generic definition.
Comparison means using at least two groups so you can attribute differences to the treatment rather than to time or chance alone. Often one group receives a control — either no treatment, a placebo, or the standard treatment — to serve as a baseline.
Randomization means using a chance process to assign units to treatment groups. This does not guarantee identical groups, but it balances out both known and unknown confounding variables in the long run, creating groups that are roughly equivalent before treatment. Randomization is what lets us make cause-and-effect conclusions.
Replication means applying each treatment to enough experimental units so that real effects can be distinguished from natural variability. Replication is about adequate sample size within the experiment, not about repeating the whole study (though that matters for science generally).
A misconception: students think randomization eliminates confounding. It balances confounders across groups on average; it does not remove them. Another trap is confusing replication with a large population sample — in experiments, replication refers to multiple units per treatment. When an FRQ asks you to 'explain the purpose,' tie the principle to the specific variables in the prompt, not a generic definition.
Control, Placebos, and Blinding
Beyond the big three principles, several techniques improve an experiment and appear in exam questions.
A control group provides a comparison baseline. A placebo is a fake treatment indistinguishable from the real one; the placebo effect is a real response to a treatment simply because the subject expects an effect. Using a placebo lets researchers separate the true treatment effect from the psychological effect of being treated.
Blinding prevents expectations from biasing results. In a single-blind study, subjects do not know which treatment they received. In a double-blind study, neither the subjects nor the people measuring the response know. Double-blinding guards against bias from both directions — subjects reporting what they expect, and researchers unconsciously scoring outcomes favorably.
A frequent AP misconception is that every experiment needs a placebo. Not so: a control could receive the current standard treatment instead of a placebo, which is more ethical when withholding treatment would be harmful. Also, blinding is separate from randomization — you can randomize without blinding and vice versa.
When an FRQ asks you to describe an experiment, mention control and blinding where relevant, but always lead with random assignment. Random assignment is the single feature that distinguishes an experiment from an observational study and permits causal claims.
A control group provides a comparison baseline. A placebo is a fake treatment indistinguishable from the real one; the placebo effect is a real response to a treatment simply because the subject expects an effect. Using a placebo lets researchers separate the true treatment effect from the psychological effect of being treated.
Blinding prevents expectations from biasing results. In a single-blind study, subjects do not know which treatment they received. In a double-blind study, neither the subjects nor the people measuring the response know. Double-blinding guards against bias from both directions — subjects reporting what they expect, and researchers unconsciously scoring outcomes favorably.
A frequent AP misconception is that every experiment needs a placebo. Not so: a control could receive the current standard treatment instead of a placebo, which is more ethical when withholding treatment would be harmful. Also, blinding is separate from randomization — you can randomize without blinding and vice versa.
When an FRQ asks you to describe an experiment, mention control and blinding where relevant, but always lead with random assignment. Random assignment is the single feature that distinguishes an experiment from an observational study and permits causal claims.
Completely Randomized, Block, and Matched-Pairs Designs
The exam tests three structures. Choosing the right one depends on whether units are similar or vary in a known way.
In a completely randomized design (CRD), all experimental units are assigned to treatments purely by chance, with no grouping beforehand. This is the default and simplest design.
In a randomized block design, units are first sorted into blocks of similar units based on a variable expected to affect the response (like sex, age, or field location), and then randomization happens separately within each block. Blocking removes the variability due to that variable, making treatment effects easier to detect. The slogan: 'block on what you know, randomize on what you don't.'
A matched-pairs design is a special case of blocking with block size two. Either you pair similar units and randomly assign one of each pair to each treatment, or a single unit receives both treatments in random order (a before/after or repeated-measures setup).
A key misconception: blocking is not the same as stratifying. Stratifying is a sampling technique to reduce estimate variance; blocking is an experimental technique to reduce unexplained variability in treatment comparison. Same idea, different context.
In a completely randomized design (CRD), all experimental units are assigned to treatments purely by chance, with no grouping beforehand. This is the default and simplest design.
In a randomized block design, units are first sorted into blocks of similar units based on a variable expected to affect the response (like sex, age, or field location), and then randomization happens separately within each block. Blocking removes the variability due to that variable, making treatment effects easier to detect. The slogan: 'block on what you know, randomize on what you don't.'
A matched-pairs design is a special case of blocking with block size two. Either you pair similar units and randomly assign one of each pair to each treatment, or a single unit receives both treatments in random order (a before/after or repeated-measures setup).
| Design | Structure | When to use |
|---|---|---|
| Completely randomized | Random assignment of all units | Units fairly homogeneous |
| Randomized block | Randomize within blocks | A known variable affects response |
| Matched pairs | Blocks of size 2 or same unit twice | Natural pairs or repeated measures |
How the AP Exam Tests This Topic
Design questions appear on both multiple-choice and free-response sections. Multiple-choice items typically ask you to count treatments, identify the response variable, or name why randomization is used. Read carefully: a question may describe two factors and ask for the number of treatments, expecting you to multiply the levels.
Free-response prompts commonly ask you to 'design and describe' an experiment. A full-credit answer must do three things: describe how random assignment is carried out (name a specific mechanism like a random number generator or drawing names from a hat), state the treatments clearly, and identify what response is compared between groups. If the prompt hints at a variable like gender or plot location, graders expect you to block on it and explain why blocking reduces variability.
Another frequent task: given a scenario, identify which design was used and justify it. Watch for the word 'paired' or before/after measurements — those signal matched pairs. Phrases like 'divided by region, then randomly assigned within each region' signal a block design.
Avoid vague language. Writing 'randomly assign subjects' without describing the mechanism often loses points on the description component. Instead say something like: 'Label the 60 subjects 01 to 60, use a random number generator to select 30 for treatment A; the rest get treatment B.' Precision earns the points.
Free-response prompts commonly ask you to 'design and describe' an experiment. A full-credit answer must do three things: describe how random assignment is carried out (name a specific mechanism like a random number generator or drawing names from a hat), state the treatments clearly, and identify what response is compared between groups. If the prompt hints at a variable like gender or plot location, graders expect you to block on it and explain why blocking reduces variability.
Another frequent task: given a scenario, identify which design was used and justify it. Watch for the word 'paired' or before/after measurements — those signal matched pairs. Phrases like 'divided by region, then randomly assigned within each region' signal a block design.
Avoid vague language. Writing 'randomly assign subjects' without describing the mechanism often loses points on the description component. Instead say something like: 'Label the 60 subjects 01 to 60, use a random number generator to select 30 for treatment A; the rest get treatment B.' Precision earns the points.
Key terms
- Experimental unit.
- The individual object or person to which a treatment is applied; called a subject when it is a person.
- Factor.
- An explanatory variable that is deliberately manipulated in an experiment; each factor has two or more levels.
- Treatment.
- A specific condition applied to units, formed by a combination of one level from each factor.
- Response variable.
- The outcome measured to assess the effect of the treatments.
- Randomization.
- Using a chance process to assign units to treatment groups so confounding variables are balanced across groups on average.
- Replication.
- Applying each treatment to enough experimental units so that real effects can be distinguished from chance variation.
- Blocking.
- Grouping similar experimental units before randomizing within each group to reduce variability from a known variable.
- Double-blind.
- A design in which neither the subjects nor those measuring the response know which treatment each subject received.
Worked example
A researcher wants to test whether a new fertilizer increases tomato yield compared to the standard fertilizer. She has 40 tomato plants: 20 in a sunny field and 20 in a partly shaded field. Sunlight is known to strongly affect yield. Design an appropriate experiment, name the design, and identify the factor, treatments, and response variable.
First identify the parts. The factor is fertilizer type, with two levels: new and standard. Because there are two treatments and one factor, the treatments are simply 'new fertilizer' and 'standard fertilizer.' The response variable is tomato yield, measured in weight of tomatoes per plant. The experimental units are the 40 individual plants.
Next, recognize that sunlight is a known variable that affects the response — the plants are already in two clearly different light conditions. This is a signal to block. We use a randomized block design with the two fields as blocks.
Within the sunny field, label the 20 plants 01 to 20 and use a random number generator to select 10 to receive the new fertilizer; the remaining 10 receive the standard. Repeat the identical randomization separately within the shaded field: label those 20 plants, randomly choose 10 for the new fertilizer, 10 for the standard.
After the growing season, measure the yield of each plant. Compare the mean yield of new-fertilizer plants to standard-fertilizer plants within each block, then combine.
Why block? Blocking on sunlight removes the large yield differences caused by light so that any remaining difference between fertilizer groups can be attributed to the fertilizer. This satisfies comparison (two groups), randomization (assignment within blocks), and replication (10 plants per treatment per block).
Next, recognize that sunlight is a known variable that affects the response — the plants are already in two clearly different light conditions. This is a signal to block. We use a randomized block design with the two fields as blocks.
Within the sunny field, label the 20 plants 01 to 20 and use a random number generator to select 10 to receive the new fertilizer; the remaining 10 receive the standard. Repeat the identical randomization separately within the shaded field: label those 20 plants, randomly choose 10 for the new fertilizer, 10 for the standard.
After the growing season, measure the yield of each plant. Compare the mean yield of new-fertilizer plants to standard-fertilizer plants within each block, then combine.
Why block? Blocking on sunlight removes the large yield differences caused by light so that any remaining difference between fertilizer groups can be attributed to the fertilizer. This satisfies comparison (two groups), randomization (assignment within blocks), and replication (10 plants per treatment per block).
Practice questions
An experiment tests two factors: soil moisture (low, high) and light exposure (2 hours, 6 hours, 10 hours). How many treatments does this experiment have?
- 3
- 5
- 6
- 9
Answer: 6
The number of treatments equals the product of the levels of each factor. Moisture has 2 levels and light has 3 levels, so there are treatment combinations. A common error is adding the levels () instead of multiplying.
A study measures each of 25 runners' times before and after a new training program, then compares each runner's two times. Identify the type of design and explain why it is appropriate here.
Answer: This is a matched-pairs design, a special case of blocking where each block is a single unit measured under both conditions.
Because each runner is measured both before and after, the two measurements are naturally paired within the same person. This design controls for individual differences in baseline fitness — each runner serves as their own control — so the comparison isolates the training effect. Randomization would apply to the order of conditions when possible; here the before/after order is fixed by the nature of the treatment.
Explain why random assignment of subjects to treatment groups allows researchers to make cause-and-effect conclusions, while random sampling does not.
Answer: Random assignment balances confounding variables across treatment groups, so differences in the response can be attributed to the treatment; random sampling only makes the sample representative of the population.
Random assignment distributes both known and unknown lurking variables roughly evenly among groups, so the groups are comparable before treatment. Any significant post-treatment difference is then likely caused by the treatment. Random sampling addresses how subjects are selected from a population, which supports generalizing results but does not by itself create comparable groups or establish causation.
FAQ
- What is the difference between a factor and a treatment?
- A factor is an explanatory variable being manipulated, and its possible values are levels. A treatment is a specific combination of one level from each factor. With a single factor, each level is a treatment. With multiple factors, treatments are all the combinations, found by multiplying the numbers of levels.
- When should I use a block design instead of a completely randomized design?
- Use a randomized block design when there is a known variable, such as sex, age, or location, that you expect to affect the response. Blocking on that variable removes its variability from the comparison, making the treatment effect easier to detect. If units are fairly homogeneous, a completely randomized design is fine.
- Is matched pairs the same as blocking?
- Matched pairs is a special case of blocking where each block has exactly two units, or where a single unit receives both treatments. The logic is identical: control for a variable by comparing within similar pairs. The difference is just the block size of two.
- How do I describe random assignment for full credit on an FRQ?
- Name a specific mechanism and show the mechanics. For example: label the subjects with numbers, use a random number generator to select a set for one treatment, and assign the rest to the other. Vague phrases like 'randomly assign them' without describing how usually lose points.
Learn this with a teacher, not a page
The Crimsora tutor teaches U3.5 Experimental Design live — explaining on a whiteboard, asking you questions, and adapting to where you get stuck.