Tool Use
Intermediate 12 min read Tool Use

Retrieval Tools

On-demand access to the corpus

A retrieval tool hands the model searchable private knowledge: embeddings, vector DBs, or internal APIs. It's the RAG primitive made available on demand — the model chooses when truth requires the corpus.

retrievalvector searchtoolknowledge accessrag tool
Prerequisites: RAG
Before we start

What you will learn

  • Expose retrieval as a callable tool
  • Set relevance thresholds and snippets
  • Prevent hallucination beyond retrieved ground
Introduction

What is this?

A retrieval tool wraps your knowledge access — vector search over embedded docs, hybrid search, or internal APIs — into an instrument the model invokes when its answer needs your corpus. It returns bounded, groundable snippets with source metadata; the answer then cites those hits.

Motivation

Why do you need it?

  • Models shouldn't 'memorize' documents at inference; they retrieve on demand.
  • Forced retrieval on every turn burns tokens; on-demand is lean.
  • Snippets + metadata make answers auditable against sources.
  • It keeps a single, testable retrieval service behind one interface.
Think about it

Mental model

Imagine

A librarian on a radio system

Think of

Retrieval tool

You don't wheel the whole library to every patron. They ask the librarian (tool) over the radio for the relevant pages; the librarian answers with the exact shelf and a few key paragraphs. Retrieval on demand — not a forklift of books.

Mechanics

How it works

  • Wrap your embed + vector-search (or search API) behind one tool.
  • Define the query parametric and result limits.
  • Return top-k snippets with source paths + scores.
  • Feed only the needed context, sanitized, into the loop.
  • Require the model to cite hits in the final answer.

Retrieval as tool.

Need
Query
Retrieve
Ground
Real world

In production

  • Support copilots query policy/product docs on demand.
  • Internal assistants search wiki, Jira and codebases via one tool.
  • Research agents loop targeted retrievals per sub-question.
Guidelines

Best practices

  • Bind k (3-6) and snippet length; more is dilution.
  • Return source metadata; cite it strictly.
  • Threshold low scores out — don't let weak hits anchor answers.
  • Scrub retrieved text of embedded instructions (injection).
Watch out

Common mistakes

  • Returning whole documents when snippets suffice.
  • Skipping the threshold; everything anchors even when nothing fits.
  • No citation gate — the model answers from 'vibes' despite a tool.
Root cause analysis

Common issues

Error
Model paraphrases without source
Root cause

Citation not enforced

Fix

Require URLs/source in output schema.

Error
Retrieval misses the right doc
Root cause

Embedding/chunk mismatch

Fix

Tune chunking and hybrid recall; test golden queries.

When things break

Debugging tips

  • Tool called but ignored → tie citations to hits in the output schema.
  • Weak hits → threshold tune or hybrid index.
  • Retrieval rarely trigger → descriptions/prompt don't mention it.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A retrieval tool should return…

2

Weak hits below threshold should still anchor answers.

Go deeper

Official documentation

File/RAG for agents (OpenAI)
docs