"Training" an AI agent rarely means training a model. In practice it means working through four layers — instructions, examples, retrieval, and only sometimes fine-tuning — while measuring against real cases at every step. This tutorial walks all four using a running example: a support agent for a fictional online course business.
Step 0: define the job in one paragraph
Before touching any tool, write the job description: "Answer student questions about course access, billing, and certificates using only our help-center articles and policy docs. Escalate refund disputes, account-security issues, and anything not covered in the docs to a human with a summary. Reply in the student's language, under 150 words, with links to sources."
This paragraph already contains your scope, your sources, your escalation rules, and your format constraints. Most failed agent projects skipped this and paid for it later. Pair it with the foundations in the beginners' practical guide if any concept below feels new.
Step 1: sharp instructions with examples (hour 1)
Write instructions with five sections: role, task, constraints, output format, and refusal rules. Then add three to five input-output examples showing ideal behavior — including one refusal ("I can't help with that account issue, but I've prepared a summary for our team…"). Examples teach faster than rules; this technique alone resolves the majority of "the agent doesn't get it" complaints.
Test immediately on ten real historical questions. If fewer than seven come back acceptable, your instructions are vague, not the model broken. Rewrite until ten-for-ten looks right before building anything else.
Step 2: build your 50-case test set (day 1–2)
Collect fifty real questions with known-good answers: forty routine, seven tricky, three that must escalate. This set becomes your permanent benchmark — run it before and after every change you ever make. Score simply: correct, acceptable-with-flaws, wrong, missed-escalation. The last category counts double against you, because silent failures are the expensive kind.
Step 3: add RAG retrieval (week 1–2)
When the agent invents policies or cites outdated info, it needs your documents, not more instructions. The retrieval pipeline has four parts:
- Chunk your help articles and policies into passages of a few hundred words, keeping headings attached so passages stay interpretable alone.
- Index them with any standard embedding or keyword search — start simple, upgrade only if evaluation demands it.
- Retrieve the top three to five passages per question and hand them to the agent with the conversation.
- Ground the answer: instruct the agent to answer only from retrieved passages plus general knowledge clearly marked as such, citing the source article for every factual claim.
Re-run your fifty cases. Typical result: factual accuracy jumps while style stays the same — exactly the division of labor you want. If retrieval returns junk, fix chunking and coverage before blaming the model; nine times out of ten the retriever starves on badly split documents.
Step 4: the evaluation loop (ongoing)
Every change — new examples, more documents, different settings — gets scored on the same fifty cases. Group failures by pattern ("ignores the refund policy update," "too verbose on billing questions") and fix the biggest pattern first. Stop tuning a layer when two consecutive changes fail to move the score: that plateau is the signal to move up to the next layer, not to keep polishing.
Step 5: fine-tuning, only with proven need (week 3–4)
Consider fine-tuning when your scoreboard shows behavior gaps that instructions cannot close: the agent follows every rule yet writes in the wrong voice, breaks format under edge cases, or needs fifty examples in context where five should do. The procedure: write fifty to two hundred exemplary input-output pairs, keep a holdout slice you never train on, run a small tuning job, and keep the tuned version only if holdout scores clearly beat the prompt-based one. If the gain is marginal, delete it — tuned models cost more per use and complicate every future update.
Cost and operations reality check
At typical support volumes, a prompt-and-retrieval agent costs little per conversation; engineering and evaluation time dominate the budget, which is another argument for exhausting the cheap layers first. In production, add what tutorials skip: logging every conversation, sampling reviews weekly, usage and cost alerts, a kill switch that routes everything to humans, and a document-update process so retrieval never serves stale policy. An agent is a system you operate, not a file you finish.
What to build next
Once your first agent holds its scores for a month, the compounding moves are: a second agent for an adjacent job reusing the same test-set discipline, tighter retrieval as your docs grow, and connecting the agent to actions (lookups, then writes) with human approval. Structure that journey through the course catalog, and review plans when your team needs guided tracks instead of articles.
Worked mini-example: fixing the refund-policy failure
Suppose your fifty-case benchmark shows six failures, four of them sharing a pattern: the agent answers refund questions from general knowledge instead of your updated policy. The fix sequence runs: first, check whether the current policy document is even in the retrieval index — half of such failures trace to a missing or outdated chunk, not to the model. Second, improve the chunk: give the refund section its own clearly headed passage with the effective date in the first line, because dated, headed passages retrieve far better than buried paragraphs. Third, add one input-output example showing a refund answer grounded in the policy with a citation. Re-run the fifty: if those four failures clear and nothing else regresses, you fixed a systemic issue in under an hour. This is the evaluation loop working as designed — patterns, not vibes.
Retrieval details that actually matter
Three chunking decisions determine most of your RAG quality. Size: passages of roughly 200 to 400 words preserve enough context without drowning the agent; adjust only when evaluation says so. Boundaries: split at headings and keep titles attached to every chunk, never mid-paragraph on fixed character counts. Metadata: tag each chunk with document name, section, and last-updated date, and instruct the agent to prefer newer sources on conflicts. Skip exotic techniques — hybrid search, reranking, knowledge graphs — until the basics plateau on your scoreboard. Most teams never need them, and those who do will know exactly which failure pattern justifies the complexity.
Knowing when to stop
An agent project ends in one of three good states: scores plateau above your quality bar and the agent ships to production with monitoring; scores plateau below the bar and you narrow the scope (fewer question types, more escalation) until they clear it; or the task proves unsuitable and you document why, saving the next team the experiment. All three are wins. The only failure mode is endless tuning without a scoreboard — motion mistaken for progress. Set a tuning budget upfront (for example, three evaluation cycles per layer) and honor it.
Common anti-patterns to avoid
Four traps catch most first-time builders. The mega-prompt: stuffing every rule ever imagined into thousand-line instructions nobody can debug — split behaviors into separate focused agents instead. The demo dataset: evaluating on five friendly examples and declaring victory, then collapsing on real traffic — your fifty-case set exists precisely to prevent this. The silent write: letting the agent modify data, send messages, or change records without approval from day one — start read-only, add actions with human confirmation, and only then consider autonomy for narrow reversible cases. The frozen agent: shipping version one and never reviewing logs again — schedule the weekly sample review before launch, not after the first incident. Dodge these four and your agent program is already above average.
