Replacing a hosted chatbot with RAG on local models

SkinStation's website chat used to run on Yellow.ai, a hosted chatbot platform. I rebuilt it as a system we host ourselves: the documents, the models and the conversations all stay on one server with one GPU.

A skin clinic is an awkward place for a chatbot. A wrong answer about opening hours is an annoyance. A confident wrong answer about a treatment is a real problem. So the design started from three rules:

1. The assistant answers only from documents the company has approved.
2. When it has no evidence, it says so. It does not guess.
3. A person can take over any conversation.

This is how each of those turned into code.

Three layers, cheapest first

Most visitors ask the same few things: what services there are, how to book, where the branches are. Sending those to a language model is slow, costs GPU time and adds risk for no benefit.

So a conversation passes through three layers, and each one is more expensive than the one before:

1. A guided conversation. A tree of questions and prepared answers that staff write in a visual editor. Picking a choice returns the prepared answer. No retrieval, no model, no GPU.
2. Retrieval and generation. When the visitor picks "Something else" and types a question, the system searches the approved documents and writes an answer from what it finds, with citations.
3. A person. Staff see every conversation in a shared inbox and can take over. Any step in the guided tree can also be marked to hand off directly.

Guided flows are versioned. Staff edit a draft, publish a numbered version, and a chatbot is pinned to one version. Publishing a new version never changes a live chatbot until someone points it at the new one.

Retrieval that knows when it has nothing

The stack is deliberately small. Documents are split into chunks of about 650 tokens with a 75-token overlap, keeping headings, page numbers and tables. Each chunk is stored in PostgreSQL twice: as a full-text index entry, and as a 1,024-dimension embedding from BGE-M3 in pgvector. There is no separate vector database and no orchestration framework.

A question runs both searches, keeps the top 50 from each, and merges them with reciprocal-rank fusion.

Ranking was the easy part. The hard part was deciding whether the best results are evidence at all. A vector search always returns something. Ask it about a service the company does not offer and it will hand back the five least irrelevant chunks, and a model given those chunks will happily write an answer.

So before any text is generated, the results have to clear an evidence gate. If the best semantic match is below a similarity threshold, and the keyword search does not independently support it, the system skips the model completely and returns a fixed refusal. The model never sees a question it has no material for.

Tuning it with questions that have no answer

I did not tune this by trying questions and reading the answers. I wrote a test set first: 20 questions the documents can answer and 5 they cannot. Every change was scored on both groups.

Configuration	Right source ranked first	Unanswerable questions refused
Strict keywords plus a similarity threshold	80%	5 of 5
Threshold removed	95%	0 of 5
Keywords matched with OR	95%	1 of 5
Strict keywords first, looser fallback, both kinds of evidence must agree	100%	5 of 5

The second row is the one to look at. Removing the threshold made the retrieval numbers better and made the assistant answer every question it should have refused. If I had measured only recall, I would have shipped that.

The first configuration had the opposite fault. It refused four questions it could have answered, because visitors do not phrase things the way documents do.

The version that passed uses strict keyword matching first, falls back to looser matching only when strict matching finds nothing, and then requires keyword and semantic evidence to agree. It reached the target without adding a reranking model.

The honest caveat: this is 25 synthetic questions, not a year of real conversations. It is a gate that catches regressions, not proof of quality.

One GPU, one answer at a time

Generation runs on a quantized 9-billion-parameter Qwen model served by Ollama, on a single 20 GB GPU. The same GPU serves the embedding model.

That means one answer is generated at a time, and the limits are built around that:

• One question in flight per visitor.
• 10 messages per 10 minutes per network, and 50 per visitor per day.
• At most three requests waiting. After that, visitors get a "try again shortly" response with a retry time, not a spinner that never ends.

Answers stream to the browser as they are written. Guided answers skip this queue entirely, which is the practical reason the guided layer exists: the common questions never wait behind a slow one.

What the public assistant may say

Publishing a document inside the company does not make it public. Each document version needs a separate approval before the website assistant may use it, and that approval does not carry over when the document is replaced.

That rule lives in the database query. The search joins every chunk to its document's current approved version at query time, so a hidden document cannot leak through a forgotten check in the interface.

The safety rules are fixed in code and cannot be changed from the admin screen. The assistant refuses to diagnose, to choose a treatment for someone, or to handle an emergency, and points the visitor to a contact link. It will explain what a service is, from approved material, and nothing more personal than that.

Handing over to a person

Every conversation has a support state: with the AI, waiting for staff, with staff, or resolved.

When a staff member takes over, the system cancels any answer still being generated and stops calling the model for that conversation. The visitor's next messages are stored and shown to staff. Staff replies appear in the same chat window, labelled as support. When staff mark it resolved, the visitor's next message goes back to the AI.

Live updates run over WebSockets, with one rule that saved a lot of trouble: the socket carries no content. It only says "something changed", and the browser fetches the real state from the database. A dropped connection or a missed event cannot lose a message, because the message was never in the event.

What it does not do

• It reads PDF and Word files only, and its text recognition for scanned pages is English only.
• It generates one answer at a time. More traffic means a bigger GPU or a second one.
• The staff inbox is shared. There is no assignment, no business-hours routing and no response-time target.

What I would tell someone building the same thing

• Build the refusal before the answer. An assistant that cannot say "I don't know" is not ready for customers.
• Test with questions that have no answer. Recall alone will lead you to ship the wrong configuration.
• Put the cheap, certain answers in front of the model.
• Keep permission checks in the query, where they cannot be skipped.
• Make the realtime channel a hint, and keep the database as the source of truth.