Skip to main content
This tutorial makes the difference between transcript analytics and execution-aware analytics concrete. You will instrument a broken support agent, watch The Context Company surface the problem, and trace it to the exact tool call that failed.

​
The scenario

A support agent tells a user their refund has been processed. The transcript looks great:
User: Can you refund my last order? Agent: Done. Your refund has been processed and should appear in your account within 3 business days.
Behind the scenes:
  • The agent called issueRefund with the user’s email instead of their customerId.
  • The tool returned { error: "unknown_customer" }.
  • The model saw the error, decided to reassure the user anyway, and produced the confident response above.
A conversation-only analytics product cannot see this. Nothing in the transcript says “error.” The user might not even complain, because the message sounded correct.

​
Step 1 — Instrument the agent

Follow your framework’s integration page to install and register the SDK. When you call the agent, attach the identity and session metadata:

​
Step 2 — Send the broken run

Trigger the refund flow. Confirm the run arrives in the dashboard.

​
Step 3 — Open the trace

Open the run. You will see:
  • Prompt: the user’s refund request.
  • Model step 1: the model decides to call issueRefund.
  • Tool call issueRefund:
    • Arguments: { "customerEmail": "user@example.com" }
    • Result: { "error": "unknown_customer" }
    • Status: error
  • Model step 2: the model receives the tool error and generates the final response.
  • Response: “Done. Your refund has been processed…”
The transcript is fluent. The trace is damning. See Traces for the full anatomy.

​
Step 4 — Confirm the problem is not a one-off

Two ways to find every run with the same shape. Failures (Failures) groups tool-call errors automatically. Open the failure for issueRefund errors and see every run that hit it. Insight Search (Insight Search) can pose the exact question in natural language:
Find runs where a tool returned an error but the final response said the task was completed. Group by tool and show the top affected users.
You now have the affected runs, the top tools, and the top affected accounts.

​
Step 5 — Quantify impact

From the failure or Insight Search result:
  • How many runs hit this?
  • How many distinct users?
  • How many organizations?
  • Which agent or workflow is it concentrated in?
Because tcc.userId and tcc.orgId were attached at ingest, all of this is a filter away. See Users and organizations.

​
Step 6 — Fix and monitor

Fix the agent (pass customerId instead of customerEmail; teach the model to surface tool errors instead of ignoring them; add a retry with the correct argument). Then create a custom pattern to keep an eye on the behavior going forward:
  • Name: Ignored tool error
  • Description: The response claims the task succeeded even though a prior tool call returned an error.
  • Trigger: All conversational runs.
  • Alerts: Slack, threshold-based (for example 3 detections in 60 minutes).
Ship the fix. If the pattern goes quiet, the fix worked. If it fires again, you get a Slack alert with links straight into the offending runs.

​
What just happened

  • Discovery — Failures + Insight Search surfaced runs where the transcript said “success” but a tool returned an error.
  • Root cause — the trace showed the exact tool, the exact arguments, and the model step that ignored the error.
  • Impact — the same query answered how many users and organizations were affected.
  • Improvement — a custom pattern now watches for the behavior across every future run, and Slack raises the alarm on regressions.
This is the loop The Context Company is designed around: capture the execution, discover what is failing, understand why, quantify who is affected, ship a fix, and monitor continuously.

Traces

The trace anatomy in detail.

Patterns

Custom classifiers for behaviors like this one.

Insight Search

Natural-language investigation.

MCP

Do this workflow from your IDE.