RAG and MCP solve different data-access problems. RAG builds a preprocessed index over documents so an agent can quickly retrieve semantically relevant context. MCP exposes live tools, resources, and application APIs so an agent can query or mutate source systems at run time. A strong enterprise assistant usually needs both: indexed retrieval for broad unstructured understanding, and live tools for structured records, current state, permissions, and actions.
The mistake is treating MCP as a replacement for vector search, or treating vector search as the only retrieval surface an agent needs. Federated tool calls can answer fresh operational questions, but they inherit each source system’s latency, schema, ranking quality, and authorization model. A unified RAG index can rank across many sources with one relevance model, but it can go stale and flatten structured business logic into text chunks.
Hybrid MCP/RAG architecture adds a routing layer. The agent first classifies the information need: broad semantic lookup, exact structured query, live status check, or action. It then chooses a pre-indexed retrieval tool, a native source-system MCP tool, or a staged combination where RAG finds candidate entities and MCP performs the precise follow-up.
A unified index crawls and preprocesses content from multiple systems into one retrieval plane. It can normalize metadata, embed chunks, add keyword indexes, and rerank results with a consistent relevance model. This is the right shape for questions that require cross-source discovery, fuzzy semantic matching, or comparing documents with different native search capabilities.
A federated tool query asks source systems directly at request time. This is useful for structured systems such as issue trackers, CRMs, databases, calendars, deployment platforms, and billing services. The result is fresh and can respect native business logic, but every source may use a different query language, ranking model, and permission boundary.
The router is the policy layer that decides which retrieval surface to use. It can be a deterministic rules engine, an LLM planner with constrained tool choices, or a hybrid classifier. Good routers reason about data type, freshness requirement, latency budget, action risk, and whether the user needs evidence citations.
Use RAG indexes when the agent needs to discover relevant context across large unstructured corpora. Use MCP tools when the agent needs authoritative current state, structured calculations, or side effects. Many workflows use both: retrieve candidate policies or records first, then call the source system for current totals, owner details, or mutations.
The first step is to classify the user’s request by data-access need. A simple router can get far with explicit categories:
from enum import Enum
class Route(str, Enum):
RAG_SEARCH = "rag_search"
LIVE_TOOL = "live_tool"
RAG_THEN_TOOL = "rag_then_tool"
TOOL_THEN_RAG = "tool_then_rag"
def classify_query(question: str) -> Route:
q = question.lower()
if any(w in q for w in ["current", "right now", "status", "available"]):
return Route.LIVE_TOOL
if any(w in q for w in ["approve", "create", "update", "send"]):
return Route.RAG_THEN_TOOL
if any(w in q for w in ["policy", "why", "compare", "similar", "past"]):
return Route.RAG_SEARCH
return Route.RAG_SEARCHProduction routers should not rely only on keywords. They should also consider source inventory, user permissions, estimated latency, and whether the answer requires citations. The important architectural point is that retrieval choice is explicit instead of hidden inside a monolithic agent loop.
For complex workflows, RAG often narrows the search space before tools provide authority. For example, a purchase-approval assistant can retrieve relevant capital-expenditure policy, similar historical requests, and vendor-contract clauses, then call finance and procurement tools for current budget and order status.
async def answer_purchase_question(question, user):
evidence = await rag.search(
question,
filters={"acl_user": user.id, "domains": ["policy", "contracts"]},
top_k=8,
)
entities = extract_entities(evidence, question)
budget = await mcp.call_tool(
"finance.current_budget",
{"department": entities.department, "user_id": user.id},
)
purchase = await mcp.call_tool(
"procurement.lookup_request",
{"request_id": entities.request_id, "user_id": user.id},
)
return synthesize_answer(question, evidence, budget, purchase)When both layers return evidence, the synthesizer should preserve provenance. Indexed chunks get document IDs and citations. Tool calls get tool names, timestamps, parameters, and authorization context. The final answer should state which facts came from indexed documents and which came from live systems.
RAG retrieves troubleshooting guides, release notes, and historical ticket patterns. MCP tools query the live ticket, subscription, entitlement, and incident systems. The agent can explain the likely issue from documents while still checking the customer’s current state before recommending an action.
RAG finds architecture docs, runbooks, postmortems, and code explanations. MCP tools query build pipelines, feature flags, observability dashboards, and issue trackers. A failure investigation can start with semantic runbook lookup, then pivot to live traces and deployment history.
RAG supplies policy context and precedent. MCP tools retrieve live budgets, approvals, vendors, and purchase-order status. The architecture avoids embedding volatile numbers into a vector index while still grounding decisions in policy text.
Maintain a registry that describes each retrieval surface. The router needs more than tool names; it needs to know freshness, latency, data type, side-effect risk, and citation behavior.
retrieval_surfaces = [
{
"name": "knowledge_base_rag",
"kind": "index",
"data": ["docs", "contracts", "runbooks"],
"freshness": "hourly",
"latency_ms": 350,
"side_effects": False,
"returns_citations": True
},
{
"name": "ticketing.get_ticket",
"kind": "mcp_tool",
"data": ["tickets", "comments", "status"],
"freshness": "live",
"latency_ms": 900,
"side_effects": False,
"returns_citations": False
}
]Separate retrieval tools from mutation tools. The agent can freely search indexes and call read-only status tools within policy, but writes such as approving a request, changing a ticket, or sending a message should require stronger authorization and often human confirmation.
Start with the cheapest broad lookup, then spend latency only when the current evidence is insufficient. A common sequence is keyword/vector search, reranking, entity extraction, source-system lookup, then final synthesis. This keeps simple questions fast while giving hard questions enough depth.