Benchmarks
How we test MapleStats
An AI assistant with a data tool can still give the wrong number, cite the wrong table or invent a figure that does not exist. We measure how often, on public question sets with answer keys read from Statistics Canada and the Bank of Canada, and we publish every answer.
The method
- Every tool answers the same questions with the same AI model (Claude Sonnet 5.5), the same instructions and the same setup. Only the data access changes: MapleStats, seven other Canadian data servers, or no data tool at all.
- Each answer is graded correct, partial, abstained (it declined) or hallucinated (a figure that does not match the official data, or one that does not exist). A program grades first; a person then reads every answer that is not correct, and each re-grade is published with its reason.
- Some questions are traps: a series that is not published, a date with no data, or a calculation that mixes years. Declining is not counted as an error; inventing a number is.
The Maple Test (November 2026)
Fifteen questions on labour, population, prices, output, trade, wages and markets, written before MapleStats was run on them, with two questions about figures that do not exist and one that asks for a misleading calculation.
| Tool | Correct | Partial | Declined | Invented figures |
|---|---|---|---|---|
| OpenMCP | 15 | 0 | 0 | 0 |
| CanData-MCP | 15 | 0 | 0 | 0 |
| MapleStats | 14 | 1 | 0 | 0 |
| arose26 | 14 | 1 | 0 | 0 |
| mcp-canada | 14 | 1 | 0 | 0 |
| mcp-statcan | 12 | 2 | 1 | 0 |
| Pipeworx | 12 | 2 | 1 | 0 |
| No data tool | 2 | 0 | 13 | 0 |
What it shows: on these core Statistics Canada and Bank of Canada questions, the leading servers are tied within one answer, and none invented a figure. With fifteen questions and one run each, a one-answer gap is not a real difference. The two perfect scores come down to the census trap: OpenMCP and CanData-MCP did not do the misleading calculation because a lookup failed, while MapleStats, arose26 and mcp-canada did it and then flagged it. The test asks nothing outside StatCan tables and Bank of Canada series, so it cannot show coverage of other sources; a larger test that does is in preparation.
Pipeworx was tested with both of its hosted servers; one of its answers was lost to a dropped connection and counts as declined. arose26 was tested with both of its packages.
Answer key, every answer and every re-grade: tests/maple_test.
CanData's Toronto housing benchmark (October 2026)
Fifteen questions on rents, vacancy, housing starts, prices, income and inflation in the Greater Toronto Area, built and published by the CanData team, who also ran the other tools. MapleStats was run through their own harness and grader.
| Tool | Correct |
|---|---|
| MapleStats | 14 |
| mcp-statcan | 11 |
| CanData-MCP (October version) | 3 |
| No data tool | 2 to 3 |
MapleStats scored 10 before the fixes these questions showed were needed, and 14 after; the other tools' scores are CanData's own runs, and CanData-MCP has improved since (it scores 15 on the Maple Test). Write-up: candata-gta-benchmark.md. The benchmark itself: Canadian Open Data Agentic Platform.
What these results do not show
- Each tool answered each question once; run-to-run variation is not measured.
- Thirty questions cover a small part of Canadian statistics, and all of them use only Statistics Canada and the Bank of Canada. A new question set will follow each round of fixes, so a score never rests on questions MapleStats was tuned against.
- A different AI model, or the cheaper model behind the chat demo, can score differently.