← AICédric MerlinFREN

AI galaxy Case 5 of 8

An AI invented two wrong definitions

The problem

Asked about a business acronym, a model gave two definitions, both wrong, with total confidence. And it had been told to say “I don’t know”.

What I did

01

The instruction fails

Asking the model to be careful is not enough.

02

Only excerpts

I now give it nothing but document excerpts.

03

Otherwise: absent from the sources

With no relevant excerpt, the answer is “absent from the sources”, without even querying the model.

Diagram: from an invented answer to an answer grounded in sourcesAt the top, an answer contains two definitions crossed out in red and the “I don’t know” instruction is ignored. Below, a question goes through a database of excerpts (Qdrant) then the model, which answers with a source. For a second question with no relevant excerpt, a detour skips the model and the answer is “absent from the sources”.I don’t knowtwo wrong definitions?Qdrantmodel[1]?absent from sources

The result

A rule I apply everywhere, including to my own assistants.

Stack

  • RAG
  • Qdrant