Prompt, knowledge base or integration? Where to put your AI agent's knowledge
Today's models have huge context windows, so the simplest option would be to paste the entire terms and conditions, price list and FAQ into the agent's instructions. That works in a demo and stops working in production. We show where each piece of information belongs, how to prepare documents for RAG and how to describe tools so the agent actually uses them.
In short
- The prompt (agent instructions) is the rulebook: role, scope, tone, the order in which to collect details, decision rules, escalation and tool descriptions. It applies in every conversation.
- The knowledge base (RAG) is the handbook: procedures, FAQ, terms and conditions, price lists, product descriptions. The agent retrieves only the fragments it needs for the current question.
- A tool (integration) is live access to a system: order status, available slots, stock levels, creating a ticket. Anything that changes by the minute or concerns a specific customer.
- Rule of thumb: how the agent works → prompt; a fact needed in some conversations → knowledge base; changing or personal data → tool. Opening hours belong in the knowledge base; "is there a free slot tomorrow at 10:00?" needs a tool.
- Everything in the prompt means a higher cost for every turn, a slower voicebot and rules that get lost among the facts.
- What makes RAG accurate is the shape of your documents: one topic per document, full sentences instead of keywords, and metadata with symptoms described in the customer's own words.
Where should an AI agent's knowledge go: the prompt, the knowledge base or a tool?
Each piece of information goes where its nature fits: rules in the prompt, facts in the knowledge base, live data in tools. In theory, you could paste almost everything into the agent's instructions. In practice, it is one of the most expensive mistakes you can make in a deployment.
| Prompt (agent instructions) | Knowledge base (RAG) | Tool (integration) | |
|---|---|---|---|
| What goes here | Role, scope, tone, rules for collecting details, decision rules, escalation, prohibitions, tool descriptions | Procedures, FAQ, terms and conditions, price lists, product descriptions, instructions | Statuses, appointment slots, stock levels, customer data, creating tickets and leads |
| How often it changes | Rarely | Weeks, months | Every minute |
| Who maintains it | The agent owner | The process and knowledge owner | IT, an integrator or the tools Copilot |
| Analogy | Rulebook and personality | The handbook on your desk | A phone call to the department that knows |
This split also divides responsibility. A price list change is uploaded by whoever owns the offer, not by whoever owns the agent's configuration. A new endpoint in the booking system is a job for IT, not a reason to rewrite the instructions.
Why not paste everything into the prompt?
Because a long prompt raises costs, slows down responses and weakens the rules. Specifically:
- The cost grows with every turn. The model processes the instructions afresh with every message. A conversation with twenty exchanges means the full terms and conditions twenty times over.
- The voicebot responds more slowly. In a phone call, a longer context means a longer silence before the answer. The customer hears it straight away.
- Rules get lost among the facts. The model treats all text equally. The rule "do not give out order details without verification" loses out to page three of the price list.
- Every price list change is a change to the agent. Instead of swapping a file, you edit the instructions and risk breaking something else along the way.
- Auditing is hard. When a customer complains about "what the agent told me", you want to know which document, and which version, the answer came from.
What is RAG in a chatbot?
RAG (Retrieval-Augmented Generation) is a mechanism in which, for every question, the system searches the knowledge base for the most relevant document fragments and the model builds its answer from them. The model does not "read" the whole knowledge base. It receives a handful of fragments.
The process looks like this:
- The system extracts clean text from the documents and splits it into fragments.
- A customer asks: "I bought some shoes a week ago and they're too small – what can I do?"
- Semantic search compares the meaning of the question with the meaning of the fragments, not just the keywords.
- A few of the best fragments are passed to the model, e.g. from "Returns procedure".
- The agent answers based on them, in the language and tone set out in the instructions.
The practical takeaway: how you write your documents determines which fragments reach the model. The best model will not answer accurately from a chaotic archive. It will answer well from a few dozen well-written documents.
In X-TALK, knowledge sources can be files (PDF, DOCX, TXT, CSV, spreadsheets, HTML, Markdown), websites (individual URLs or a crawl of the whole site), SharePoint libraries, short texts entered in the panel ("this week the helpline is open until 15:00") and images from documents, which the agent can show in its answer. Changing data is connected through integrations, not uploaded as a file.
How do you prepare documents for the knowledge base?
Split knowledge by problem, not by department, and write in full sentences. A single 200-page PDF called "Everything about customer service" is the worst option. Search returns fragments from different topics, the model gets noise, and updating one procedure means editing the whole file.
A set of short documents works better – 1–3 pages each, one topic per document:
- e-commerce: "Returns procedure", "Complaint: damage in transit", "Changing the delivery address";
- medical clinic: "Preparing for a blood test", "Cancelling and rescheduling appointments", "Referrals: when you need one";
- service desk: "Problems logging in to your account", "No VPN access", "Ordering new equipment";
- bookings: "Booking cancellation policy", "What to bring to your first lesson", "Equipment hire".
Drop the "keyword → answer" style
Older chatbots and IVRs matched keywords, so knowledge was written as terse pairs such as "Q: return / A: 14 days". Semantic search needs context. "Return" on its own could mean returning goods, a refund, a tax return or a returned parcel.
If you are migrating from an old system, do not carry the keywords over one to one. Group 20 keywords about returns, complaints and exchanges into 3–4 full documents describing the problem, conditions, steps and exceptions. Then "shoes too small" matches "wrong size", and that matches "returning goods".
Write sections that stand on their own
Search may return just one fragment. That is why every ## section in a document should make sense without the rest: repeat the context ("at shop X, returns are possible…"), use short paragraphs, lists and specific numbers. Keep prices, opening hours and contact details in a single document that is easy to update. Replace scans without a text layer with a text version.
What metadata should you add to knowledge base documents?
keywords, symptoms and summary add the most value, and for procedures, required_questions too. Markdown documents in X-TALK can start with a YAML header (front matter). The knowledge engine uses it to boost results: a document whose keywords or symptoms match the question gets a higher score.
---
procedure_id: PROC-014
priority: HIGH
category: KONTO/LOGOWANIE
summary: Co zrobić, gdy klient nie może się zalogować
keywords: [hasło, logowanie, login, dostęp, konto zablokowane]
symptoms: [nie mogę się zalogować, nie pamiętam hasła, konto zablokowane]
stt_hints: [PIN, token, Authenticator]
required_questions:
- Na jaki adres e-mail jest założone konto?
- Czy widzisz komunikat o zablokowaniu konta?
---
| Field | What it is for | Example |
|---|---|---|
keywords |
Keywords and synonyms, including colloquial ones and other languages | "parcel, package, courier" |
symptoms |
Symptoms described in the customer's own words – a strong matching signal | "I didn't get the text with the code" |
priority |
CRITICAL, HIGH, NORMAL, LOW; boosts critical procedures |
outage, security |
category, tags |
Keeping the knowledge base organised, filtering, analysis | "ORDERS/RETURNS" |
summary |
A short description of the document, for better matching and previews | "How to return goods within 14 days" |
required_questions |
Questions the agent must ask before walking the customer through the procedure | "What is the serial number?" |
stt_hints |
Names and abbreviations the agent should hear correctly in voice conversations | "VPN, PESEL (Polish national ID number), Paczkomat" |
required_questions moves some of the rules out of the prompt and into the knowledge. The rule "ask for the serial number first, then diagnose" no longer has to sit in the instructions. It lives in the procedure itself, so the prompt stays short and each procedure carries its own requirements. For details and a full document template, see the playbook chapter How to prepare a knowledge base.
When does an AI agent need a tool instead of the knowledge base?
Whenever the answer depends on the state of a system at that moment or on a specific customer. A file listing available slots will be out of date before the agent reads it. Tool calling lets the agent query a system during the conversation and write the result back to it.
| Industry | Knowledge base (fact) | Tool (live data) |
|---|---|---|
| E-commerce | Returns policy, delivery times, size chart | Order status with customer verification, tracking number, product availability |
| Medical clinic | Test preparation, list of specialities, appointment cancellation rules | Available slots, booking, cancelling an appointment |
| Service desk | Step-by-step procedures, configuration guides | Creating a ticket, ticket status |
| Bookings | Lesson prices, what to bring, terms and conditions | Instructor availability, booking a slot |
In X-TALK, you have three routes to tools: a catalogue of ready-made integrations (including HubSpot, Pipedrive, Zendesk, Freshdesk, ServiceNow, Shopify, PrestaShop, WooCommerce, InPost and Cal.com), custom tools built by Copilot for any system with an API, and a REST API and MCP for the other direction. You can see the full list on the Integrations page, and read about building tools from a description in the Custom tools section.
A good tool does one thing ("check order status", not "handle the order"). It returns short fields that are ready to be spoken and a clear error message. It requires verification, e.g. email and postcode, before returning personal data. In the voice channel, API speed matters too, because every call means a moment of silence.
How do you describe tools in the agent's instructions?
For each tool, write four things in the prompt: when to call it, what data it needs, how to present the result and what to do if there is an error. Missing descriptions like this are the most common gap in instructions. The agent has the tool but does not know when to reach for it.
An example for an online shop:
# Zamówienia
Do sprawdzenia zamówienia potrzebujesz numeru zamówienia lub e-maila
oraz kodu pocztowego. Bez weryfikacji nie podawaj szczegółów.
Użyj get_order_status. Podaj status, przewidywaną dostawę i numer
przesyłki. Jeśli narzędzie zwróci błąd, powiedz to wprost i zaproponuj
zgłoszenie, nie zgaduj statusu.
Note two things. The rule comes with a reason ("do not give out details without verification"), so the model applies it sensibly. The error instruction stops the agent from making up a status.
In a medical clinic, the equivalent section says: collect the speciality, the preferred date and whether it is a first visit, and only then check available slots. In a ski school: ask about the discipline, level, day and number of people in a single question, and only book once you have all the details.
The tool definition itself also steers behaviour. Marking a parameter as required means the agent will not call the tool until it has collected it. Describing parameters in the customer's language ("deposit as a percentage, e.g. 10") and providing a value dictionary ("snowboard" → discipline_id=2) reduce mistakes. For details, description patterns and an integration checklist, see the playbook chapter Tools and integrations.
What exceptions to the rule should you know about?
Some facts are worth keeping in the prompt even though they are "knowledge". There are three groups:
- critical facts needed in almost every conversation: the company name, the main contact number, the key opening hours;
- short dictionaries that the correct use of tools depends on: the list of cities served, appointment types or disciplines;
- descriptions of tools and their parameters: when to call them, what they require, how to interpret the result.
It works the other way round, too. A procedure with a required_questions field makes sure on its own that the agent asks the right questions, even though that is a rule rather than a fact.
How do you split knowledge between several agents?
Each agent should see a minimal but sufficient set of knowledge. In X-TALK, knowledge is divided into collections, e.g. "Offer", "Complaints", "IT procedures", and each one is assigned to one, several or all agents.
- Shared knowledge: company information, terms and conditions, contact details, opening hours, locations.
- Dedicated knowledge: the catalogue for the sales agent, complaint procedures for after-sales service, IT and HR procedures for the employee assistant, reply templates for Mailbot.
A smaller, consistent set of documents means fewer competing fragments in search. The agent is less likely to mix up the procedures of two departments.
There is a security reason too. An agent on your website or helpline talks to anyone, and anything in its collections can end up in an answer. Keep margins, internal procedures and negotiation notes in collections assigned only to internal agents. A security block in the prompt protects the instructions, but the best protection for knowledge is not to give it to a public agent in the first place. Read more about data protection on the Security page.
How do you check that knowledge is in the right place?
With tests on real questions and by reviewing unanswered questions. In the X-TALK panel, you ask the agent a test question and see both the answer and the sources it used. Q&A tests are a set of questions with expected answers, run after every change to the knowledge or instructions. Hotspots give you a list of questions the agent could not answer, and the knowledge base health view shows unavailable sources and documents nobody uses.
The agent Copilot keeps the split in order when you make changes. When you add a tool, it adds to the instructions when to use it and how to read the result. The change goes into a draft, passes the tests and only then becomes visible to customers.
Pre-launch checklist
- The prompt contains the role, scope, tone, decision rules and conditions for handing over to a human, without price lists or terms and conditions.
- Every tool has a description in the instructions: when to call it, what it needs, how to present the result, what to do on error.
- Large documents are split by problem, 1–3 pages each, with no duplicates or old versions.
- The most important procedures have
keywords,symptomsandsummary, and those that need details from the customer haverequired_questions. - Statuses, appointment slots and stock levels are connected through tools, not uploaded as files.
- Collections are assigned to agents, and internal knowledge is not available to public agents.
- Q&A tests on 20–50 real customer questions pass.
- There is a named person responsible for the knowledge and a set rhythm for reviewing hotspots.
Frequently asked questions
Doesn't a large context window solve the problem?
No. A model with a large window can fit the entire terms and conditions, but it processes them with every message, so you pay for them on every turn. In voice conversations, response times also get longer, and the rules lose weight among the facts.
Should the price list go in the knowledge base or in a tool?
Keep a fixed price list that changes every few weeks in the knowledge base, in a single document that is easy to update. If the price depends on availability, a current promotion or the specific customer, fetch it with a tool. Always use a tool for calculations, e.g. lease instalments – never leave them to the model's "mental arithmetic".
How many documents do you need to start?
Quality matters, not quantity. A few dozen well-written documents, each on one topic, work better than an archive with many versions of the same terms and conditions. Start with the topics customers ask about most often and fill the gaps based on hotspots.
Can I upload my old FAQ from a chatbot or IVR?
You can, but rewrite it first. "Keyword → answer" pairs work poorly in semantic search because they lack context. Group related keywords into full documents describing the problem, conditions, steps and exceptions.
How do I know which source the agent used to answer?
In a test conversation in the panel, you see the answer and the sources used. The internal chat for employees shows the document and fragment with every answer. On the Dedicated plan, the conversation debugger shows which sources and tools influenced the answer.
Want to test this split on your own knowledge? Build an agent with Copilot: the wizard prepares the instructions, and you add the documents and tools. For a larger deployment with integrations, book a PoC. For a full discussion of the topic, see the playbook chapter Prompt, knowledge base or tool.