Pilot Studies

Pilot Studies and the Aims of Piloting

A pilot study is a small-scale, preliminary version of a planned research study, conducted before the main study begins. It uses a small sample of participants (who are not part of the main study's sample) to test and refine the research design, materials, and procedure. Piloting is a routine and important stage of research design, enabling researchers to identify and correct problems before investing the full resources of the main study.

Why Conduct a Pilot Study?

A pilot study serves several distinct functions:

• Testing the procedure: checking that the planned procedure runs smoothly, that instructions are clear, and that the session fits within the intended time frame. Timing problems — sessions running much longer or shorter than planned — are commonly identified at pilot stage.
• Checking the materials: verifying that stimulus materials (word lists, vignettes, images, questionnaire items) are appropriate in difficulty, clarity, and length. A questionnaire that pilot participants find confusing, offensive, or excessively long can be revised before the main study.
• Assessing the measures: checking that the dependent variable measures actually produce usable data — for example, that a reaction time measure is sensitive enough to detect the predicted differences, or that a rating scale is not subject to ceiling or floor effects (where all participants score at the maximum or minimum, leaving no variability to analyse).
• Identifying demand characteristics: pilot participants can be debriefed and asked what they thought the study was about. If they correctly identify the hypothesis, the procedure may need to be modified to reduce cues that reveal the study's purpose.
• Training researchers: piloting provides practice for the researcher and any confederates or assistants, ensuring the procedure is delivered consistently before the main data collection begins.
• Estimating effect sizes for power analysis: pilot data can provide preliminary estimates of effect sizes, enabling a formal power analysis to determine how large the main study's sample needs to be to have adequate statistical power to detect the expected effect.

What Changes After a Pilot Study?

Based on pilot findings, researchers may: rewrite unclear instructions; shorten, simplify, or add questionnaire items; adjust the difficulty of tasks; change the timing of events; modify the cover story to reduce demand characteristics; or reconsider the choice of dependent variable measure. Importantly, the pilot sample is not included in the main study's data — it is used solely to refine the procedure.

Limitations of Pilot Studies

Pilot studies add time and cost to the research process. A very small pilot (e.g. three to five participants) may not reveal all potential problems. If pilot participants share information about the study with potential main study participants, the integrity of the main study may be compromised. Despite these limitations, piloting is considered standard good practice in experimental and questionnaire-based research.