Evaluating small programmes well

The gold standard of impact evaluation, the randomised controlled trial, is rarely available to the programmes that most need evidence. A municipal employment scheme, a regional business support fund or a pilot health outreach service usually has a few hundred participants, a fixed budget, and no way to randomly withhold support.

That doesn't mean rigour is out of reach. It means the design has to fit the programme.

Key points

1. Decide what the evaluation is for before choosing a method. Learning, accountability and scale-up decisions need different evidence.
2. Start monitoring on day one. Most evaluations fail on data that was never collected, not on analysis.
3. Use theory-based approaches to explain how change happened, and comparison groups where data allows to estimate how much.
4. Time findings to decisions. Evidence that arrives after the budget is set has little value.

Start with the theory of change

A theory of change sets out how a programme's activities are expected to lead to outcomes, and what has to be true along the way. It is the backbone of any evaluation of a small programme, because it tells you what to measure at each step.NoteWe use the OECD DAC criteria (relevance, coherence, effectiveness, efficiency, impact and sustainability) to structure evaluation questions, but always map them back to the programme's own logic.

If a programme assumes that training leads to job applications, which lead to interviews, which lead to jobs, then you can measure each link. When outcomes disappoint, you can see where the chain broke.

Build a comparison where you can

Without randomisation, a credible comparison group is still often possible:

• Administrative or survey data, such as national labour force surveys, can provide a matched comparison on characteristics like age, region and education.
• Waiting lists let you compare people who have applied but not yet started.
• Phased roll-outs across regions create natural comparisons over time.

None of these is perfect. All of them are better than comparing participants with nobody, as long as the report is honest about their limits.

Time evidence to decisions

The most common failure we see isn't methodological. It is timing. A final evaluation that arrives fourteen months after the programme's renewal decision is an interesting document. It isn't a management tool.

When the evidence arrives

Months between programme decision points and availability of evaluation findings.

• Final evaluation only14
• Mid-term + final7
• Continuous monitoring + light evaluation2

05101520

Show data table

Category	Value (months)
Final evaluation only	14
Mid-term + final	7
Continuous monitoring + light evaluation	2

The alternative is to invest a little more in monitoring and a little less in a single large evaluation. If a small set of outcome indicators is collected continuously from the start, a lighter evaluation can focus on the questions monitoring can't answer: why things happened, and for whom.

Report for use

Evaluation reports are often written for the commissioner's file. Better ones are written for the next decision. In practice that means:

• A short brief, readable in one sitting, alongside the full report.
• Recommendations that are specific, costed where possible, and tied to evidence.
• A working session with the programme team to turn findings into actions.

An evaluation nobody reads has cost money and produced nothing.

For commissioners

If you are commissioning an evaluation of a small programme, three requests will improve it more than any methodological specification: ask for monitoring data to be designed in from the start; ask the evaluator to state the decision their findings will inform; and ask for a brief that a minister, or a board member, will actually read.