A docs chat that refuses to guess

So there are roughly nine hundred tutorials about building a documentation chatbot, and almost all of them stop at the point where it returns something. That is the easy half. The hard half is what happens when it does not know, because a docs assistant that confidently invents a menu item is worse than no docs assistant at all. The person asking will go and look for that menu item.

This is the one we ship in Media Manager, the video platform I work on. It is live, real customers use it, and I built it to say “I don’t know” a lot more often than you would probably be comfortable with.

It started as a docs move, not an AI project

The documentation used to be an Astro Starlight site, sitting in its own repository and deployed to Cloudflare Pages. Perfectly nice setup. The problem was the gap between it and the application: you would ship a feature on Monday and update the docs on, well, eventually.

So I moved the whole corpus into the main Laravel repo as plain Markdown, served through Laradocs. The point was not the chat. The point was that docs living next to the code get updated with the code, and an agent that has just changed how captions work can go and fix the captions page in the same pull request. That alone was worth the move.

The chat came afterwards, and it came cheap, because moving the docs in also told me something useful about their size.

Seventy-three pages

The entire documentation corpus is 73 Markdown pages. Around 38,500 tokens, all of it, every page. The median page is about 357 tokens. Go and measure yours now, before you read any further, because everything below follows from that one number.

That number decides the architecture, and it is the number most RAG tutorials never ask you for. If your corpus is a hundred thousand documents then yes, you want chunking strategies and a vector database and a reranker. If yours fits in a context window twice over, most of that machinery is you solving somebody else’s problem.

So I did not chunk. At all.

The unit of retrieval is a page

Everyone glosses chunking, and it is where most of your quality actually comes from, so let me be blunt about mine: there isn’t any. A documentation page is written by a human as one coherent explanation of one thing. Chop it into 500-token windows and you take that coherence, throw it away, and then spend a fortnight of engineering effort trying to reassemble it with overlaps and metadata you would not have needed if you had left it alone.

At 357 median tokens a page, a page is a chunk. It is just a chunk somebody edited, which is more than you can say for the ones your splitter produces.

Embedding is one vector per page, over the slug, the title and the body. Storage is a PHP array. Similarity is cosine, in a loop:

foreach ($a as $i => $value) {
    $dot += $value * $b[$i];
    $magnitudeA += $value * $value;
    $magnitudeB += $b[$i] * $b[$i];
}

$magnitude = sqrt($magnitudeA) * sqrt($magnitudeB);

return $magnitude > 0 ? $dot / $magnitude : 0.0;

That is the vector database. Seventy-three dot products, in PHP, on the web server. It is not clever and it does not need to be, because 73 of anything is nothing, and you can hold the whole index in memory without noticing. The whole index is cached for thirty days under a key that includes a SHA-256 of the corpus text, so it rebuilds itself the moment anybody edits a page and never rebuilds otherwise.

I take the top 12 pages, which gives a prompt of roughly 4,750 tokens on a typical question.

The bit that stops it lying

Here is the part I actually want you to steal.

The model gets the pages and returns structured output: an answer, a boolean saying whether the pages answered the question at all, and a list of citations. Each citation is a page route plus an excerpt. The instructions are explicit that the excerpt must be copied out word for word.

Then you check it. Every excerpt is normalised (links flattened, whitespace collapsed, lowercased) and looked for in that page’s actual Markdown. If it is not there, the citation is thrown away. And then:

$answered = $answerable && ! $citations->isEmpty() && filled($answer);

An answer with no surviving citation is never shown to the user. Not softened, not shown with a caveat. Not shown.

This is the difference between asking a model to be grounded and requiring it. “Only use the provided context” is a wish you put in a prompt and hope about. Checking that the quoted sentence physically exists in the page is a test, and it fails closed. Whatever your model does next, it cannot talk its way past a str_contains.

Two smaller things in the same spirit. Changelog pages get labelled in the prompt as release history, “a record of what changed in past releases, not a description of how the product works today”, because otherwise it will cheerfully tell somebody a feature works the way it worked in v6. And if a cited page contains a > [!WARNING] block, I pull it out and surface it alongside the answer, so the caveat the docs author bothered to write does not get quietly summarised away.

When retrieval finds nothing

Two paths lead to the whole corpus. If retrieval returns no pages, or if the first attempt comes back unanswerable, I throw all 73 pages at it and try once more.

That is only affordable because 38,500 tokens is nothing, so it is a brute-force fallback that exists purely because the corpus is small, and I would not pretend it generalises to yours. But it buys something genuinely useful, which is a distinction between two failures that look identical to whoever asked:

'verified_gap' => $usedFallback && ! $answered,

If it failed with the whole corpus in front of it, retrieval was not your problem. The documentation does not cover it. That is not a bug to fix in the search, it is a page somebody needs to write, and it lands in an admin list of proven gaps.

Which is where the docs move pays off a second time. The gap is recorded in the app, the docs are in the same repo as the app, and the thing that writes the missing page can be pointed at both.

So is it any good

There is a docs:eval command with 28 fixture questions. Twenty-four have a known correct page. Four are things the docs deliberately do not cover, sat there to check it says no: Salesforce integration, scheduled publishing, exporting analytics to Google Sheets, and the refund policy.

Running it just now, against claude-haiku-4-5:

Retrieval recall:      24/24
Answered on target:    22/24
Refused when covered:  2/24  (lower is better)
  ...the model refused:      1
  ...every citation dropped: 1
Refused when uncovered: 4/4  (higher is better)
Warnings surfaced:     1/1

Retrieval is 24 out of 24, which says more about having 73 pages than about any cleverness on my part. Refusal on the uncovered four is 4 out of 4, and that is the number I care most about.

The two misses are more interesting than the hits.

“Why is this greyed out?”, asked from the webhooks page, got refused by the model. Reasonable, if annoying: the docs describe what webhooks do, not why a particular control is disabled for you right now. That is a real gap and it is now recorded as one.

“How do I create an API token?” is the one that stings. The model answered, correctly, from the right page. Then every one of its citations failed the verbatim check, so the whole answer was binned and the user got told nothing. It had the right information and lost it on a quotation error.

That is the cost of the design and I would rather be honest about it than pretend the grounding is free. You trade a small number of correct answers for approximately zero fabricated ones, and on a product where people act on what you tell them, I think that is the right side to fail on. It is still a bad afternoon for whoever wanted an API token.

What it costs

Haiku 4.5, priced in the app at $1 per million input tokens and $5 per million output.

A normal question on the retrieval path is about 4,750 input tokens and a couple of hundred out, so roughly $0.006. The whole-corpus fallback is about 38,500 in, so roughly $0.04.

Six tenths of a penny. That is the entire argument for not gating it behind a plan tier or a support queue, because at that price the question stops being “can we afford to let everyone ask” and becomes “why would I spend a week of my life building the gate”. There is a monthly token cap wired in as a backstop, currently 20 million tokens, and I measure it by summing usage straight out of the message table rather than trusting a counter somewhere to have stayed honest.

Conversation memory is capped at six messages, which is enough for a follow-up and not enough to slowly fill the context with the model’s own prose.

What I would tell you to do differently

Measure your corpus before you choose an architecture. Genuinely, before anything else. Mine was small enough that chunking, vector storage and reranking were all answers to questions I did not have, and the version without them is a few hundred lines of PHP that a colleague can read on a Tuesday afternoon without me sitting next to them.

Make your grounding mechanical. A prompt that says “do not hallucinate” is decoration and you know it. A validator that discards any claim not physically present in the source is a control, and it is the only part of this I would keep completely unchanged if I rebuilt the thing tomorrow.

Then let it refuse. Your instinct will be to squeeze the refusal rate down, because a bot that says “I don’t know” feels like a failure, and I promise you it is not. Six of the 28 fixtures end in no answer, and every one of those is either an honest gap or a question the docs genuinely do not cover. The alternative was never a right answer. The alternative was a confident wrong one, and somebody wandering off to look for a button that does not exist.

Would you have shipped the API token one, though? I still think about that.

Filed under Developer. No comments, on purpose.