x-talk

Knowledge base for AI: why “put everything in one file” is the worst idea you could have

X⁠-⁠TALK team · · 20 min read

When companies roll out an AI Communication Center, they keep making the same mistake. They have a good model and a good platform, but a poorly prepared knowledge base. The result? The bot gives wrong answers to users’ questions. Customers get frustrated. The project is written off.

Person pointing at an email icon

The story I’m about to tell really happened. I’ll only change the company’s name, because nobody likes being held up as a bad example.

The “TechShop” story – how a knowledge base killed an AI project

The company – let’s call it “TechShop” – sells electronics online. 2,000 orders a day, 400 customer service enquiries, 8 agents per shift. They decided to roll out AI. They chose a good language model, connected it to the chat on their website and then did what most companies do: they took their internal document “Customer service knowledge base.docx” – 180 pages, everything in one file – and uploaded it to the system.

For the first week it looked promising. The bot answered simple questions. The team celebrated its AI launch.

In the second week, the problems started. A customer asked about returning Bluetooth headphones. The bot replied with the returns procedure for TVs, because a few pages further on in the same document there was a section on returning large household appliances, and the system decided it was a better match. Another customer asked about a laptop warranty. The bot gave the warranty terms for smartphones, because both passages contained the words “warranty” and “electronics”, and the system couldn’t tell the contexts apart.

In the third week, the operations director received a report: the bot answered 43% of enquiries correctly. The rest were imprecise, out of date or simply wrong. Agents were spending more time correcting the bot’s mistakes than serving customers directly. The AI wasn’t working as expected.

What went wrong? It wasn’t the model or the technology – it was how the knowledge base had been prepared.

How RAG really works and why understanding it matters for a successful rollout

Before we talk about how to prepare a knowledge base properly, we need to understand how the mechanism that uses it works. Without understanding this concept, any advice will feel abstract.

RAG, or Retrieval-Augmented Generation, is an architecture that brings together two worlds: information retrieval and response generation. It sounds technical, but the idea is simple. Imagine an expert who, instead of answering from memory, first reaches for the right handbook, finds the relevant chapter, reads it and only then formulates an answer to the question. That’s exactly how RAG works.

How does a RAG engine work?

The process goes like this. A customer asks a question – say, “I bought shoes a week ago, they’re too small, what can I do?”. The system turns this question into an embedding – a mathematical vector, a string of numbers that represents the meaning of the sentence in a multidimensional space.

This isn’t keyword search. The system isn’t looking for a document containing the word “shoes”. It’s looking for the document whose meaning is closest to the meaning of the user’s query. That’s the fundamental difference between traditional search and semantic search.

The system then compares this vector with the vectors of every document in the knowledge base. Each document has been processed the same way beforehand – split into fragments (chunks), turned into embeddings and stored in a vector database. The system finds the fragments with the highest semantic similarity: the ones that “mean” something closest to the customer’s question.

These fragments are passed to the language model as context. The model reads the customer’s question, reads the retrieved pieces of knowledge and generates an answer based on them. It doesn’t hallucinate – it answers on the basis of specific documents. Or at least that’s how it should be, if the knowledge base contains up-to-date information and is well prepared.

And this brings us to the heart of the problem.

Why one huge document as a knowledge base is a disaster for RAG

Let’s go back to TechShop and its 180-page file. What happens when a RAG system tries to use it?

First, the file is split into fragments – usually 500–1,000 tokens each (roughly half a page of text). 180 pages produce around 400 fragments. Each fragment is turned into an embedding and stored in the vector database.

Problem number one: context gets blurred

The fragment about returning Bluetooth headphones and the fragment about returning TVs have very similar embeddings, because both are about “returning electronics”. The system doesn’t know that these are completely different procedures with different conditions. All it sees is that both fragments are semantically close to the user’s query.

Problem number two: information noise

In one huge document, the returns procedure sits next to the price list, which sits next to the FAQ, which sits next to the warranty terms. Fragments from different sections get mixed up. The language model receives five fragments of “context”: two about returns, one about prices, one about warranties and one about the shop’s terms and conditions. It has to decide for itself what’s relevant – sometimes it gets it right, more often it doesn’t.

Problem number three: updating is a nightmare

Have the returns rules changed? You have to open a 180-page document, find the right section, change it and then re-index the whole file so that it contains the latest information. All 400 fragments have to be turned into embeddings again. That costs money – literally, because you pay the embedding model provider for every run.

Compare that with an approach where you have a separate document “Returns procedure – headphones and accessories” (2 pages) and a separate “Returns procedure – large appliances and TVs” (2 pages). The RAG system knows straight away which document is closer to the customer’s question and which knowledge source to use. No noise. No mix-ups. And when the headphone returns procedure changes, you re-index just that one small document, which is how you get the latest information into the system.

Anatomy of a well-prepared document for RAG

Now that we know what not to do, let’s talk about how to prepare content that makes RAG run like clockwork.

A good document for RAG has three layers:

  1. Metadata  – information about the document that helps the system with retrieval
  2. Content  – the actual knowledge, written in a way that embeddings can understand well
  3. Structure  –  a division into sections that makes chunking and precise matching easier

Layer 1: Metadata  –  search boosters

At the top of the document, you place a block of information: category, priority, keywords and symptoms (that is, how the customer describes the problem in their own words). This metadata acts as a search “booster”. When a customer says “I can’t log in”, the system doesn’t just look for documents semantically close to that sentence – it also checks whether any document lists exactly that phrase among its symptoms. If it does, that document gets a higher priority.

Keywords should include synonyms and everyday expressions. A customer won’t say “password reset procedure”. They’ll say “I forgot my password”, “I can’t get into my account”, “my login’s been blocked”. Each of these phrases should appear in the metadata of the password reset document.

Layer 2: Content – context for embeddings

And this is where RAG requires a fundamental shift in thinking. If you’ve ever built a knowledge base for a traditional chatbot or IVR system, you’re used to the “short keyword → short answer” format. Password reset → Click “Forgot password” on the login page. Account locked → The account will unlock after 30 minutes.

This format doesn’t work in RAG systems. Why? Because embeddings need context. The short phrase “password reset” isn’t enough information for the system to understand what the document is about. It could be a reset for the online shop, the admin panel, the mobile app or the office Wi-Fi. Without context, the embedding is “blurred” – it matches many queries, but none of them precisely.

A proper RAG document describes the problem in full sentences: what’s happening, why, what the causes are, what steps to take and how to check that the fix worked. The richer the context, the more precise the embedding, and the better the match with the customer’s question.

It may sound counter-intuitive – “more text = better precision?” – but that’s exactly how semantic embeddings work. A longer document describing the problem, its causes and the solution gives the engine more points to match on. The short phrase “password reset” could fit 20 different contexts. A full description of the procedure, with symptoms and steps, hits the target precisely.

Let’s show it with a real example – option A vs option B

That’s enough theory. Now let’s see how it works in practice, with an example that could happen in any online shop.

A customer calls and says: “I bought a media player from you two weeks ago. Since yesterday it won’t charge. I plug the cable in, the light blinks once and goes out. I tried a different cable – same thing.”

This is a real question. Natural language, emotion, technical details, a time frame. Let’s see how a RAG system answers it depending on how the knowledge base was prepared.

Option A: a knowledge base in the “short keyword → short answer” style

Many companies prepare their knowledge base the way they did for traditional chatbots – as terse bullet points:

  • Product return: The customer has 14 days to return. Form on the website.
  • Complaint: Fill in the complaint form. Processing time: 14 days.
  • Product exchange: The product can be exchanged within 30 days of purchase.
  • Charging problems: Check the cable and charger. Use original accessories.

What does the RAG system see? Four short fragments. The embedding of the customer’s question – which mentions a media player, charging, a light, a cable and two weeks – has to be matched to one of them. The problem is that “Charging problems: Check the cable and charger” has a semantically close embedding, but “Product return – 14 days” is also close, because the customer mentioned two weeks. And “Complaint” fits too, because the customer is describing a fault.

The system doesn’t know which to choose. The embeddings of these four entries are too short for the system to grasp the differences between them. The result? The model receives three fragments with similar scores, none of which gives it enough context to answer precisely. So it answers vaguely: “Please check the cable and charger. If the problem persists, please fill in the complaint form.”

The customer is frustrated. They’ve already checked the cable. They want to know what happens next – is it a complaint, a return or an exchange? The bot didn’t help.

Accuracy: 43% (as in TechShop’s case)

Option B: a knowledge base prepared for RAG

Now the same scenario, but with a knowledge base written the way it should be for a RAG system. Instead of four entries, there’s a separate document:

  • Document: Electronic equipment faults – diagnostics and handling path
  • Category: Technical support / Complaints
  • Products: media players, Bluetooth speakers, wireless headphones

Symptoms reported by customers: won’t charge, won’t switch on, light blinks and goes out, runs down quickly, doesn’t hold its charge, charging doesn’t work. Customers often report problems charging electronic devices. The most common symptoms are: the device doesn’t respond when the charging cable is plugged in, the charging light blinks briefly and goes out, the device won’t switch on despite charging for several hours, the battery runs down much faster than when the device was new.

Step 1. Initial diagnostics.

Ask the customer to check whether they are using the original cable and charger supplied with the device. If the customer has already tried another cable and the problem persists, go to step two.

Step 2. Decide on the handling path.

If the product was bought within the last 14 days, the customer has the right to return it without giving a reason (form on the website under “Returns”). If more than 14 days have passed but the product is under warranty (24 months from the date of purchase), the customer should submit a complaint. Tell the customer that the complaint will be processed within 14 working days.

Step 3. Complaint.

Ask the customer for their order number. Explain that once the complaint has been submitted, they will receive a return label by email within 24 hours. The device will be diagnosed by the service centre; if the fault is confirmed, the customer will receive a new device or a refund. Important: don’t suggest that the fault is the customer’s fault (e.g. “you were probably using the wrong cable”). Treat every report as a potential manufacturing defect.

What does the RAG system see now? The embedding of this document is rich in context. It contains the words customers actually use: “won’t charge”, “light blinks and goes out”, “tried a different cable”. The symptoms in the metadata match the customer’s description almost word for word. The match score is high and unambiguous, and the system is in no doubt about which document to pick.

The language model receives this one precise document and generates the answer: “I understand your media player won’t charge even with a different cable. As it’s been two weeks since you bought it, the best route is a warranty complaint. Could you give me your order number? Once you have, we’ll email you a return label within 24 hours and our service centre will diagnose the device.”

The customer gets a specific answer in real time. They know what will happen next. They feel looked after.

Accuracy: 89% (the result after optimising the knowledge base)

Why is the difference so big?

In option A, the RAG system had 30 words of context to work with. In option B, it had over 200 – with symptoms, a step-by-step procedure and a tone of voice. It comes down to the fact that semantic embeddings need anchor points: words and phrases that match the way customers describe their problems.

A customer will never say “electronic device complaint”, and that’s crucial. They’ll say “the player I bought won’t charge, the light’s blinking”. If your knowledge base doesn’t contain phrases like that, the system won’t find them. Not because the model is wrong. The problem is that you haven’t given it the right conditions to do its job well.

It’s exactly the same principle that applies to web search engines. Google doesn’t rank pages titled “Shoes”. It ranks pages describing “black leather running shoes size 43”. RAG works in exactly the same way.

Where RAG costs come from and how to keep them under control

RAG isn’t free, and using it means making informed decisions. Every element of the pipeline generates costs, and it’s worth knowing exactly where, so you can optimise them.

Cost one: embeddings

Every document in the knowledge base has to be turned into an embedding – a mathematical vector. You pay the embedding model provider for this (e.g. OpenAI, Cohere, local models). The cost per document is small – fractions of a cent – but it grows with the number of documents and how often you update them.

If you have 500 documents and update 50 of them a day, that’s 50 embedding operations a day. With large knowledge bases and frequent changes, it’s worth planning an update strategy – for example, syncing once a day rather than in real time.

Cost two: vector storage

Embeddings take up space in the vector database. These aren’t large costs, but they grow linearly with the size of the knowledge base. 500 documents is nothing. 50,000 documents in 10 languages already needs planning.

Cost three: retrieval

Every customer query requires an embedding of the question to be generated and compared with the database. This is a per-query cost. At 1,000 queries a day it’s negligible; at 100,000 it starts to add up.

Cost four, and the most important: tokens in the language model

When the RAG system finds matching fragments, it sends them to the language model as context. The more fragments and the longer the documents, the more tokens. And tokens in GPT-4 or Claude are the most expensive part of the pipeline.

That’s why a well-prepared knowledge base – with short, precise documents instead of one huge file – directly lowers costs and can bring significant benefits. Instead of sending the model 5,000 tokens of context (because fragments from a huge file are blurred and the system has to send more to “hit” the right one), you send 1,500 tokens from a precisely matched document.

Cost five: advanced RAG techniques

Query rewriting (rephrasing the query so it better matches the knowledge base), re-ranking (re-sorting search results) and query translation in multilingual settings – each of these techniques improves quality but adds latency and cost. In a text chat, an extra 200–300 milliseconds goes unnoticed. In a voice conversation, where every second of silence is felt, it’s a significant trade-off.

Cost summary

A well-prepared knowledge base doesn’t just mean better real-time answers. It also means lower operating costs. Precise documents = precise matching = fewer tokens sent to the model = a lower bill at the end of the month.

System prompt vs RAG – two different tools, one goal

There’s a question we hear from almost every client: “Can’t I just put all our knowledge into the system prompt?”

Technically you can, but in practice it’s a road to nowhere.

The system prompt is the instruction the language model receives at the start of every conversation. It defines who the agent is, how it should behave, what tone it should use and what it must not do. It’s the agent’s “personality” and “rulebook”.

The RAG knowledge base is the “handbook” the agent consults when needed. It doesn’t carry the whole thing around in its head – it reaches for it when it needs a specific piece of information.

The difference is fundamental. The system prompt is sent to the AI model with every query. Every word in it is tokens you pay for. If you put 50 pages of procedures into the system prompt, every customer conversation starts by processing those 50 pages – even if the customer is only asking about opening hours.

RAG works the other way round: it retrieves only the pieces of knowledge relevant to the specific question. Customer asking about a return? The system retrieves the returns procedure. Asking about opening hours? It retrieves the contact information. The rest of the knowledge base isn’t processed and doesn’t generate costs.

The rule of thumb is simple. If the information is fixed and concerns how the agent behaves – “always speak Polish”, “don’t promise discounts”, “escalate complaints to an agent” – put it in the system prompt. If the information changes and concerns subject knowledge – procedures, price lists, FAQs, product descriptions – put it in RAG. That way the system prompt stays light (200–500 tokens) and costs stay under control.

Multilingual support – the trap nobody talks about

If your company serves customers in several languages, Retrieval-Augmented Generation poses a challenge that most vendors would rather not mention.

A language model (GPT-4, Claude, Gemini) can generate answers in practically any language. Customer writing in German? The model will answer in German – even if the knowledge base is in Polish. This works because language models are multilingual by nature.

But RAG isn’t a large language model. RAG is a semantic search engine, and that’s where the problem starts.

Embeddings – the mathematical vectors we talked about – are language-sensitive. The German question “Wie kann ich mein Passwort zurücksetzen?” and a Polish document titled “Reset hasła użytkownika” (user password reset) are two different points in vector space. Embedding models were trained mainly on English-language data, and cross-language matching (e.g. German → Polish) scores much lower than matching within the same language.

What does this mean in practice? That a customer from Germany may not get the right answer – not because the knowledge isn’t in the knowledge base, but because the system didn’t find it.

There are three strategies for dealing with this problem:

  1. Query translation – before the question reaches RAG, the system translates it into the language of the knowledge base. Simple, but it adds latency.
  2. Dedicated knowledge bases per language – each language has its own knowledge base. The highest precision, but the highest maintenance cost.
  3. A hybrid approach – a knowledge base in the main language plus automatic query translation plus multilingual keywords in document metadata.

Each of these strategies involves trade-offs, but the worst strategy is no strategy at all – pretending the problem doesn’t exist and hoping it will “sort itself out somehow”.

Updating knowledge is a process, not a one-off task

There’s one more thing companies forget after go-live: the knowledge base is a living thing. Products change. Terms and conditions are updated. Procedures evolve. New customer questions call for new documents, the latest information and up-to-date data.

Updating a document in a RAG knowledge base isn’t a simple case of “uploading a new file”. Every change requires converting the document to a standardised format, splitting it into fragments again (chunking) and, most importantly, re-indexing the embeddings – recalculating the semantic vectors. That incurs real costs.

That’s why you need a process. Who is responsible for updating the knowledge? How often? Where do they get information about changes? How do they check that the agent is giving accurate answers after an update?

The best companies we work with appoint an “AI knowledge owner” – someone who regularly reviews agent conversations, identifies wrong answers, updates documents and tests the changes. It doesn’t have to be an engineer. It can be an experienced customer service manager who understands what customers need and can translate that into documents a RAG system can understand.

If your company uses SharePoint, Confluence or another document management system, it’s worth considering automatic synchronisation. A change to a document in SharePoint automatically updates the RAG knowledge base – no manual copying and no risk of the agent answering from an outdated version.

What TechShop learned – results after optimising the knowledge base

Let’s return to our story. After three weeks of disappointing results, TechShop didn’t give up on AI. Instead, it changed its day-to-day approach.

They took their 180-page document and broke it down into 47 separate documents from different departments. Each covered a single topic: “Returns – large appliances”, “Warranty – laptops”, “Warranty – smartphones”, “Order status — FAQ”, “Changing the delivery address”. Each document was 1–3 pages long, with clear metadata, keywords and symptoms described in the customer’s own words.

They added YAML metadata at the top of each document: categories, priorities, synonyms. They wrote the content in full sentences, with context, instead of telegraphic bullet points.

The result? Two weeks after rebuilding the knowledge base, answer accuracy rose from 43% to 81%. After a month of iterations – small tweaks based on results and analysis of real conversations – they reached 89%. Agents now handled only complex complaints and VIP customers, while the bot took care of the rest.

They didn’t change the language model. They didn’t change the platform. They changed the knowledge base – and gained outstanding customer service across their organisation.

What next?

If, after reading this article, you understand RAG better and you’re thinking “OK, I need to review our knowledge base” – good. That’s the first step. For the second, you can dig deeper into the subject with the TALK Playbook: a complete guide to building an AI Communication Center. Inside, you’ll find detailed instructions on preparing a knowledge base, formatting documents for RAG, designing system prompts, integrating with SharePoint and much more. 17 chapters packed with practical detail and best practice.

Try it on your own agent

Enter your website address and X⁠-⁠TALK Copilot builds an agent you can talk to right away, by chat and by phone. No sign-up, no card.

Build an agentBook a PoC

Read more