Chief AI Architect Thinking: Embracing Context Graphs in Enterprise AI Architecture

Many enterprise AI programs begin the same way. A team connects a few MCP servers to the CRM, ticketing and billing systems, and the agent’s first demos look good. Then a broader question comes in, and the setup stops working.

Ask an agent why one customer was frustrated last week and it can pull that customer’s tickets and answer. Ask which customers were frustrated over the past year and it has to page through thousands of records over live APIs. It hits rate limits and timeouts, and it never gives a useful answer.

The model isn’t the problem. SaaS APIs are built to look up one record at a time, not to answer questions across a whole dataset. Tools like Zendesk or Salesforce won’t let you ask questions in plain English across all your data, so you have to build that ability yourself.

Four layers of context

  1. Company playbook (instructions, skills and memory): how your organization defines things. “At-risk customer” might mean ticket volume at one firm and survey scores at another. These definitions tell the agent where to look before it fetches any data.
  2. Real-time lookups (live API calls): cheap and simple to build. Use them for current state and high-stakes checks, such as whether an incident is still open or an invoice has cleared.
  3. Indexed replica (cached data): a local, searchable copy of third-party data, kept in sync, so the agent can search by meaning and analyze many records at once. It costs more to build and keep fresh, but nothing else can answer questions across a large dataset.
  4. Precomputed insights (derived context): summaries, embeddings and earlier conclusions saved ahead of time. They keep later questions fast and the context window small.

The flow

A router sends each request to the right agent. The agent checks the company playbook first, then decides where to get the data. If the indexed replica is fresh enough, it uses that. If not, it makes a real-time lookup. A summarizer combines the results into a short, relevant context before generating the answer. Useful findings are saved as precomputed insights, so the next request starts further ahead.

Provenance is required

Every piece of context given to a model should carry its history: the source system, when and how it was fetched, the identity and permission scopes used, any changes made to it, and the conditions that make it stale. Vector stores support this through metadata, so store it at ingestion. Adding it later is much harder. When an agent makes a bad business decision, provenance is what lets you trace why.

Takeaways for architecture leaders

  • Connectors are not comprehension. Linking tools isn’t enough. A context graph ties data, processes and actions together across the company.
  • Precision over volume. Pull only what the question needs, and structure what you sync so it can be retrieved efficiently.
  • Show your sources. Every answer should trace back to where it came from, for debugging, audits and grounded answers.

None of this is new technology. It’s architecture, and teams that get it right early will scale agents without losing control of them.