Audit of a support assistant that made up answers
One week to measure where a RAG assistant was failing, why it kept getting more expensive and what to fix first.
This is an example project: it describes a kind of work I do, not a specific client.
- Industry
- B2B software
- Engagement
- Audit for a product team
- My role
- Technical audit
- Duration
- 1 week
- Stack
- Python, LangChain, OpenAI, PostgreSQL, pgvector
The problem
A software company has a support assistant with RAG over its documentation. Users complain about wrong answers and the model bill grows every month. Nobody knows how many answers fail or why.
What I did
- Gathered 80 real questions from the support history with the team, each with its correct answer.
- Measured the assistant as it was against that suite.
- Split the failures by cause: the right document isn’t retrieved, it’s retrieved but misused, or the question isn’t covered by the documentation.
- Reviewed the cost per conversation: context size, the model used at each step, and retries.
What I found
- Most failures came from retrieval: chunks that were too large and had lost the title of the section they belonged to.
- When it found nothing, the assistant answered anyway instead of saying it didn’t know.
- All retrieved context went to the most expensive model, even for one-line questions.
What I delivered
A report with failures ranked by impact, the eval suite ready to run on every change, and a three-step fix plan that starts with how the documentation is chunked.