Responsible AI

« What if it makes things up? » — getting an agent to cite its sources

It is the first question in every workshop, and it is the right one. A language model produces a plausible sentence; nothing in its mechanics guarantees it is true. The craft is in making the error visible.

14 August 20266 min readBy Sébastien Joumel
You do not remove invention, you make it visibleNo design eliminates the risk. A good design makes a mistake obvious instead of letting it pass unnoticed.

The single rule: cite

Every claim an agent produces points to the document and version it came from. Not « according to our procedures » — the exact reference, clickable, checkable in three seconds.

The rule looks modest. It changes everything: it turns review from an act of trust into an act of verification. Someone who must believe reviews badly; someone who can check reviews fast.

If a sentence has no source, it has no business in the output.

Four designs that work

Answer from the documents, not from memory

The agent searches your base first, then writes from what it found. If it finds nothing, it must say so — not fill the gap with what it « knows ».

Allow « I don’t know »

An agent that is not allowed to fail will invent. The circuit must provide a « not in the base » exit that escalates to a person — and that exit should be frequent at the start.

Separate extraction from writing

Extracting a date from a document is verifiable. Writing a paragraph is not. When both are mixed, nobody knows what they are reviewing.

Log the inputs

The log keeps what the agent read, not only what it produced. That is what lets you fix the cause instead of the output.

Two false fixes

« We’ll use a more powerful model. » A better model reduces how often errors happen, not their nature. On a question whose answer is nowhere in your documents, the best model on the market will also invent — just more convincingly.

« We’ll review everything. » Nobody reviews three hundred outputs a day. After two weeks, review becomes a glance. Better to review ten per cent properly than one hundred per cent in passing.

How we check that it holds

When a system opens, we build a set of real cases — forty to a hundred — whose correct answer is known. We measure three things: the share of correct answers, the share of justified « don’t knows », and the share of silent errors. Only the third really matters.

That case set is not only for go-live: we replay it on every model or prompt change. It is what lets you say, with numbers, whether an update improved or degraded the system.

Read on the site

Other notes

All notes →

First call — fifteen minutes

We will tell you which system to open first.

Describe the task that costs you the most. We will tell you what is feasible, how long it takes and what it costs. If the answer is no, you will leave with the two reasons why.