Chatbots built on a company’s documents still answer wrong, or make things up, when they should say they don’t know. Tally is my research into cutting that error rate. This page is its running record.
Project
Tally, my own research project
Type
Evidence-checked AI assistant
Role
Solo: the research and the test design
Field
Applied AI
LLM
Decision model
Round
This round: Model A vs Model B. Next: Model C.
Sources
Static only: documents, policies and historical data
Status
Research phase
Updated
Challenge
Chatbots that answer from a company’s documents still get facts wrong, or make them up, instead of saying they don’t know. A confident wrong answer is worse than none.
Approach
First, compare the two common designs, whole documents in context and retrieval, on the same questions. Then test my own: a decision model will gather and assess the evidence, and the LLM (the language model) will only word the answer.
Findings
In published studies, neither design wins every test, and both still answer questions they should decline. There are no Tally results yet. Those will come from my own tests, and I’ll publish how I ran them.
01
Why “I don’t know” matters
“Sometimes the right answer is ‘I don’t know.’”
A company’s chatbot is only useful if people can trust what it says. When the documents don’t hold the answer, or the model misreads them, most reply anyway, in the same calm tone they use when they’re right.
On a policy or a procedure, a wrong answer is worse than no answer, because people act on it. When the chatbot is wrong, the business is wrong. So Tally has one goal: far fewer wrong answers, and an honest “I don’t know” when the evidence isn’t there.
Case in point
In February 2024, a small-claims tribunal in British Columbia (2024 BCCRT 149) found that Air Canada’s website chatbot had given a customer wrong information about bereavement fares. It rejected the idea that the airline wasn’t responsible for its chatbot, and ordered the airline to pay about 812 Canadian dollars.
Three kinds of error
I’ll count three kinds of error. A wrong answer misreads the source. A made-up answer claims something no document says. A scope failure answers a message the chatbot should have recognized as out of scope, or as no real business inquiry at all.
All three count against the chatbot. An honest “I don’t know” isn’t an error. My tests will also include 200 attempts to steer the chatbot off the business, and 50 real questions phrased like attacks.
Case in point
In December 2023, a dealership’s chatbot couldn’t tell off-topic or adversarial requests from real customer inquiries. It wrote code for one user and, told by another to agree with anything, “agreed” to sell an SUV for one dollar. Nothing was sold; the bot was removed (Business Insider; AI Incident Database).
02
Three ways to build an answer engine
There are two common ways to make a chatbot answer from a company’s own documents, and a third I want to test. All three use an LLM to write the answer. They differ in what the LLM gets to read, and in what checks its work.
The third rests on my thesis. In it, a decision model will do the gathering and assessment: what’s asked, what evidence there is, and whether that evidence answers the question. The LLM will only do the language and communication. If that works, the LLM’s quality should matter little, and I’ll test how much.
This round of research compares the first two. The third is the next round, and I make no claims about it until I’ve tested it.
Models A, B and C
One question, three designs. Each diagram shows what the LLM reads before it answers, and what, if anything, checks the answer.
Model A
Model A: Whole documents in context
Illustration
Every document goes into the LLM’s prompt with the question, and the LLM reads it all before answering.
Nothing is left out, as long as every document fits in the prompt.
Accuracy drops as the text grows, and facts in the middle can get missed.
In a direct test on questions with no answer, it made answers up more often than retrieval.
Model B
Model B: Retrieval (RAG)
Illustration
A search step picks the passages that look most relevant, and the LLM answers from those alone.
The LLM reads a few passages, not everything, so the document set can be large.
Nothing checks whether a retrieved passage really answers the question.
When search brings back the wrong passage, the LLM often answers anyway.
Model C
Planned · next round
Model C: A decision model plus an LLM
Illustration
A new kind of AI will do the gathering and assessment. The LLM will only word the answer, and the decision model will check it.
Scope: Sorts each message first, so it answers only what’s in scope and knows when facts are needed.
Evidence: Finds the supporting evidence, or says there isn’t any. No evidence, no guess.
Verify: Checks the answer against the question and the evidence, then sends, revises or holds it back.
Endings: An answer. Without evidence: an offer to escalate if the stakes are high, otherwise “no evidence found”.
Not tested yet. I make no claims about how well it works until it is.
03
What published studies found
Before testing anything myself, I read the published studies that compare A and B. None of them settles the question for one company’s private documents, and they often disagree with each other.
What they do agree on is useful. Each design fails in its own way, and neither one is safe on its own. That’s why Tally will run its own tests, and why Model C comes after them.
What’s known, and what isn’t
Each finding comes from published studies, not from Tally. The notes name the studies and give their numbers.
What the studies found6
Neither design wins every test. Reading everything and retrieving the relevant parts each fail in their own way.
When the documents fit and the model is strong, A usually gets more questions right than B.¹ (note 1)
A loses accuracy as the text grows, even when every relevant fact is in front of it.² (note 2)
B stays focused, but when it retrieves the wrong passage, the model often answers anyway.³ (note 3)
On questions with no answer, B declined correctly more often than A. Both still made answers up.⁴ (note 4)
Counting, comparing and multi-step questions trip up both designs more than simple lookups do.⁵ (note 5)
What they don’t settle6
How A and B compare on one company’s private documents, with today’s strongest models.
How often each design is wrong versus making things up: most studies give a single score.
Whether either design’s confidence matches how often it’s actually right.
Questions about dates and change over time, which no study I found compares directly.
Whether B’s edge on “I don’t know” survives a better search step.
Very large document sets, which few studies test with both designs.
¹Note 1: Public long-document tests, 2024–25: A ahead of B by about 2 to 16 percentage points on average, depending on the model (Li et al., EMNLP 2024; Li et al., 2025). On 2026 safety manuals: A 73.1% correct, B 65.4% (Hamilton et al., under review).
²Note 2: Accuracy fell by 13.9–85% as the input grew, even when every relevant fact was retrieved (Du et al., Findings of EMNLP 2025). Facts in the middle of a long input were missed more often (Liu et al., TACL 2024).
³Note 3: With too little evidence retrieved, most models declined 50–73% of the time but still gave a wrong answer 15–40% of the time (Joren et al., Google, 2024). Passages that look relevant but aren’t pull accuracy down (Jin et al., 2024).
⁴Note 4: LaRA (ICML 2025), 11 models, questions the text can’t answer: at 32K tokens (tokens are the word pieces a model reads), B declined correctly 81.7% of the time and A 71.3%; at 128K tokens, 72.1% and 49.7%.
⁵Note 5: Comparison questions: A ahead of B by about 15 percentage points (LaRA, ICML 2025). Counting: most long-context models under 50% at 128K tokens (Bertsch et al., Oolong, 2025). Multi-step: B’s first search often misses the second step (Li et al., EMNLP 2024).
A and B in published studies
A and B in published studies
Topic
Model A: whole documents
Model B: retrieval
Single fact
Model A: whole documentsAbout even; slightly ahead with the strongest models
Model B: retrievalAbout even; ahead on longer documents, mostly with open models
Multi-step
Model A: whole documentsAhead with the strongest models
Model B: retrievalThe first search often misses the second step
Comparison
Model A: whole documentsClearly ahead, by about 15 percentage points⁵ (note 5)
Model B: retrievalClearly behind
Counting
Model A: whole documentsWeak: most models under half right on very long inputs
Model B: retrievalNo direct test found; likely weak too
Dates
Model A: whole documentsWeak at adding up changes over time
Model B: retrievalDate-aware search helps; no direct comparison found
No answer
Model A: whole documentsMakes answers up more often as the text grows
Model B: retrievalDeclines correctly more often, but still guesses
Very large
Model A: whole documentsCan’t run once the documents outgrow the prompt
Model B: retrievalStill runs, but only as good as its search
The studies can’t settle this for Tally, so my own tests will. A and B will answer the same questions about the same documents, including questions the documents can’t answer. Wrong, made-up and out-of-scope answers will be counted apart from honest “I don’t know”s.
Saying “I don’t know” is never an error, but when the answer was in the documents, it costs a correct answer. So the rule for a winner is fixed before the run. The design with fewer errors wins only if the gap is clear, and it is clearly under 5 percentage points behind on correct answers.
Then I’ll run A and B again with low-cost LLMs from two providers, and later with a messier copy of the documents that keeps every fact.
Method note
Both designs answer every question. Errors are compared with McNemar’s test for paired answers, correct answers with a paired confidence interval. A 100-question pilot checks how often the two disagree. If, as is likely, it’s more than planned, I’ll add questions before the run, so the 5-point rule holds.
The planned test, step by step
Illustration
1
Documents
One fixed set of documents, frozen, so both designs read exactly the same text.
2
Questions
About 1,000 to start, drafted by several AI models. One in five has no answer.
A person checks every answer key, and every no-answer question in full. Those include near misses, false premises and out-of-scope questions. Tuning and pilot questions are kept separate.
3
Both designs
A and B answer every question three times, with the same LLM and the same instructions.
All three use one top-tier LLM. A later test repeats it with low-cost ones.
4
Every answer
An AI judge from a different model family labels every answer. People check a share.
Six labels: correct, wrong, made up, scope failure, an honest “I don’t know”, and one when the answer was there. Plus wording errors: added claims, softened conditions, misquoted figures.
5
The rule
Fixed in advance: clearly fewer errors, and clearly under 5 points behind on correct answers.
Of 800 answerable questions, 5 points is 40 answers. Win: clearly fewer errors, clearly under 5 points behind. Trade-off: clearly fewer errors, not clearly under 5. Otherwise: no clear difference.
6
Method
I’ll publish how I measured A and B, alongside the results.
Cost per correct answer is measured alongside accuracy. It never decides a result.
Same documents, same questions, and one rule, set before the run.
05
The next round: Model C
Once A and B are measured, the next round adds Model C: a decision model, a new kind of AI, working alongside the LLM. It will check scope, then evidence, then the answer itself.
C won’t always end in an answer. Without evidence, a high-stakes question, such as one about money, a booking or safety, will get an offer to escalate to a person. Any other will get a plain reply: “I can’t find evidence in our information to answer that.”
Escalation will be scored too: correct, missed or unnecessary. I’ll test it next. Until I have, this page makes no claims about how well it works.
06
Research log
This page changes as the research goes. Every update is logged here, newest first.
, Last updated
Thesis sharpened, test plan expanded
Added scope failures as a third kind of error, two reported cases, Model C’s three endings, and tests with low-cost LLMs.
Case study published
Published this page with the research so far. Each later update gets its own entry here.
A vs B review and method
Reviewed the published studies that compare whole documents in context with retrieval, and settled how I’ll test A and B.
Earlier entries (3)
Research phase set
Put research first: measure the two common designs, then test my own.
Design goal and three checks
Set one goal, fewer wrong answers, and three checks: scope, evidence, verify. Static sources only, and each deployment will be tuned to one client’s material.
Started evaluating decision models
Began looking at decision models to pair with an LLM.
Building something that must be right?
Work with me
Tell me what you’re building and where it stands. I take on new products, rebuilds and long-term support. I’ll reply by email.
