---
title: "Audit of a support assistant that made up answers | Red Moon Consulting"
description: "One week to measure where a RAG assistant was failing, why it kept getting more expensive and what to fix first."
url: https://www.redmoonconsulting.es/en/case-studies/chatbot-audit/
lang: en
---

[Back to work](/en/case-studies/)

# Audit of a support assistant that made up answers

One week to measure where a RAG assistant was failing, why it kept getting more expensive and what to fix first.

This is an example project: it describes a kind of work I do, not a specific client.

Industry

B2B software

Engagement

Audit for a product team

My role

Technical audit

Duration

1 week

Stack

Python, LangChain, OpenAI, PostgreSQL, pgvector

## The problem

A software company has a support assistant with RAG over its documentation. Users complain about wrong answers and the model bill grows every month. Nobody knows how many answers fail or why.

## What I did

1. Gathered 80 real questions from the support history with the team, each with its correct answer.
2. Measured the assistant as it was against that suite.
3. Split the failures by cause: the right document isn’t retrieved, it’s retrieved but misused, or the question isn’t covered by the documentation.
4. Reviewed the cost per conversation: context size, the model used at each step, and retries.

## What I found

* Most failures came from retrieval: chunks that were too large and had lost the title of the section they belonged to.
* When it found nothing, the assistant answered anyway instead of saying it didn’t know.
* All retrieved context went to the most expensive model, even for one-line questions.

## What I delivered

A report with failures ranked by impact, the eval suite ready to run on every change, and a three-step fix plan that starts with how the documentation is chunked.

## Tell me about your project

A 30-minute call, no strings attached. If I'm not the right fit, I'll tell you.

[Book a call](/en/contact/)

Or email me at [hola@redmoonconsulting.es](mailto:hola@redmoonconsulting.es)