Citations or It Didn't Happen: Building a Docs Chatbot That Refuses Low-Confidence Answers

The split between retrieve and rerank matters more than people new to RAG expect. Vector search is fast and recall-oriented, so it pulls back a lot of "kind of related" chunks. A reranker is slower but precision-oriented, and it gives you a calibrated relevance score per chunk that you can actually threshold on. Bi-encoder similarity scores from the first stage are not reliable enough to make an abstain decision; reranker scores are.

If you are still choosing your vector layer, I wrote about why I reach for pgvector before Pinecone for most builds ↗, and the short version is that for a single product's docs you almost never outgrow Postgres.

How do you attach a source link to every claim?

Force the model to cite chunk IDs inline, then map those IDs back to real URLs in your application layer, never letting the model write a URL itself. The model picks which chunk supports a sentence; your code owns the link. This keeps the model from inventing plausible-looking documentation URLs, which it will happily do if you let it.

I store retrieval chunks with their source metadata up front, so a citation is just a key lookup:

chunk = {
    "id": "doc_412#sec_3",
    "text": "To rotate an API key, open Settings > API ...",
    "source_url": "https://docs.example.com/api-keys#rotation",
    "source_title": "Rotating API Keys",
    "updated_at": "2026-05-02",
}

The generation prompt then constrains the model to cite by ID only:

Answer using ONLY the provided sources. After each sentence
that states a fact, add the supporting chunk id in brackets,
like [doc_412#sec_3]. If the sources do not contain the answer,
reply exactly: INSUFFICIENT_CONTEXT.
Do not use outside knowledge. Do not invent ids or URLs.

In post-processing I parse the bracketed IDs, drop any sentence whose ID is not in the set I actually retrieved (the model occasionally hallucinates an ID, and this catches it), and replace each ID with a real link from the chunk metadata. The user sees clean superscript citations; the model never touched a URL.

The honest caveat: inline-citation accuracy is not free. Models will sometimes attach a citation to a sentence the chunk does not support, a citation that points to the wrong chunk. That is what the validation stage exists to catch, and why I do not treat "the model added a citation" as proof the claim is grounded.

What confidence signal should the abstention threshold use?

Use the reranker's top-chunk score as the primary signal, with the gap to the next chunk and the validation result as secondary gates. A single similarity number is noisy; a combination of "is the best chunk relevant enough" and "did the answer survive validation" is what I trust to decide abstention.

Here is the decision logic I run after reranking and generation:

def decide(reranked, answer, validation):
    top = reranked[0].score          # cross-encoder relevance
    margin = top - reranked[1].score # how clearly best is best

    if top < ABSTAIN_THRESHOLD:
        return abstain("low_retrieval_confidence")
    if answer.text.strip() == "INSUFFICIENT_CONTEXT":
        return abstain("model_declined")
    if not validation.all_claims_supported:
        return abstain("unsupported_claim")
    if margin < MARGIN_FLOOR and validation.borderline:
        return abstain("ambiguous_sources")

    return answer_with_citations(answer, reranked)

There are three ways the bot can refuse, and each one is a real, separate failure mode. Retrieval found nothing relevant, the model itself said the context was insufficient, or the validator caught an unsupported claim. Logging which reason fired is the most useful thing you can do for tuning, because it tells you whether your problem is missing docs, a weak generation prompt, or a too-loose validator.

You set ABSTAIN_THRESHOLD empirically, not by guessing. I build a small labeled set of real questions, some answerable and some not, sweep the threshold, and pick the point where abstention catches the unanswerable ones without strangling the answerable ones. There is no universal number, because reranker scores are not comparable across different reranker models or even across query types.

How does the validation step actually catch hallucinations?

It re-reads the generated answer against the cited chunks and asks, claim by claim, whether each is supported by the text, treating the chunks as the only source of truth. This is a second model call, scoped narrowly, and it is cheaper than it sounds because the context is just the answer plus four short chunks.

You are a fact-checker. For each numbered claim, decide if it is
fully supported by the SOURCES below. Reply with JSON:
{ "claims": [ {"id": 1, "supported": true|false} ] }
A claim is supported only if a source states it directly.
Do not use outside knowledge.

SOURCES:
[doc_412#sec_3] To rotate an API key, open Settings > API ...

CLAIMS:
1. You rotate an API key from the Settings > API screen.
2. Rotated keys take up to 24 hours to propagate.

If claim 2 is not in any source, supported is false, and the whole answer abstains rather than ship a half-grounded response. I made that strict on purpose. Partial grounding is the most dangerous state, because the true parts make the fabricated part look trustworthy.

This is an entailment check, and you do not strictly need an LLM for it; a smaller natural-language-inference model can do claim-versus-source classification fast and cheaply at scale. For most SMB docs bots the volume is low enough that a second LLM call is fine, and the cost is a few cents per validated answer. At higher volume, a dedicated NLI model is where I move it to keep latency and cost down.

Does refusing answers hurt or help first-contact resolution?

It helps, as long as a refusal routes somewhere useful instead of dead-ending. First-contact resolution improves when the bot answers correctly and hands off cleanly on everything else, because a confident wrong answer does not resolve a contact, it defers and worsens it. The number that matters is correct deflection, not raw deflection.

A refusal should never be a shrug. Mine do three things: surface the closest articles the reranker found even though they scored below threshold, offer a one-click handoff to a human or a ticket form, and log the question as a documentation gap. That last part is quietly the highest-value output of the whole system, because the abstention log becomes a prioritized list of docs to write. Questions that abstain often are questions your customers have and your docs do not answer.

The tradeoff is real and worth stating plainly: