Compare AI agent versions, prompts, and experiments in production

Every prompt change, model swap, and workflow rollout is an experiment whose real answer is in production behavior. The Context Company does not assign users to variants for you, but if you attach version or variant metadata to your runs, every analysis (Patterns, failures, feedback, tools, cost, latency) can be sliced by it.

The Context Company does not run experiments or assign users to variants. Do that in your application, then send the variant to The Context Company as metadata and compare the resulting production data.

How it works

The Context Company accepts arbitrary custom metadata on every run. You choose the field names your team already uses (agent_version, prompt_version, experiment, variant, release_sha, model_version, retrieval_version). Once attached, that field becomes a filter and grouping dimension across the entire product.

Add version metadata to your instrumentation

Send the current version or variant with every run.

await generateText({
  model,
  prompt,
  experimental_telemetry: {
    isEnabled: true,
    metadata: {
      "tcc.runId": crypto.randomUUID(),
      "tcc.userId": user.id,
      agent_version: "2026-08-21",
      prompt_version: "v14",
      experiment: "new-retrieval",
      variant: assignedVariant, // "control" or "treatment"
      release_sha: process.env.RELEASE_SHA,
    },
  },
});

with tcc.run(metadata={
    "tcc.userId": user.id,
    "agent_version": "2026-08-21",
    "prompt_version": "v14",
    "experiment": "new-retrieval",
    "variant": assigned_variant,
    "release_sha": os.environ["RELEASE_SHA"],
}):
    ...

Roll out the change

Ship the new version through whatever mechanism your team already uses: full deploy, canary, feature flag, or experiment framework. The Context Company only needs to know the variant a given run belongs to.

Compare in production

Filter or group by the version field in Insight Search, Patterns, failures, feedback, and topic clusters.

Example questions

Did prompt_version=v14 reduce frustration Pattern hits compared with v13?
Compare tool failure rate between control and treatment.
Did the new-retrieval variant improve resolution signals on the top three topics?
Which new failure groups appeared after release abc123?
Compare average tokens, cost, latency, and thumbs-down rate between variants.
Did model_version=claude-4.1 handle the refund topic better than claude-4?

What to attach

There is no fixed schema. Pick names your team recognizes and use them consistently. Common fields:

Field	Purpose
`agent_version`	The version of your agent as a whole.
`prompt_version`	The specific system prompt or template version.
`model_version`	The underlying model, if you swap models independently of the prompt.
`retrieval_version`	The version of the retrieval stack (index, ranker, chunking).
`workflow_version`	The version of a specific workflow or tool graph.
`experiment`	The name of the experiment this run participates in.
`variant`	The variant this specific run was assigned to.
`release_sha`	Commit SHA for tying runs back to deploys.

Attach as many or as few as you need. Every field is queryable.

Reading the results

Because the version field is just metadata, every analysis in the product supports it:

Insight Search: “Compare thumbs-down rate for prompt_version=v14 vs v13 on the refund topic.”
Failures: filter or group failures by variant to see which errors are net-new or fixed.
Patterns: check whether frustration or task-failure Pattern hits changed.
Recaps: highlight the release window and see which metrics moved.
MCP and the REST API: build custom dashboards or CI checks.

What The Context Company does not do

Assignment. The Context Company does not decide which user gets which variant. Use your feature flag or experiment platform for that.
Statistical significance testing. Comparisons are exploratory and directional. For rigorous experiment analysis, join the data out through the API to your existing statistics stack.
Traffic ramping and holdouts. These live in your rollout system, not in The Context Company.

Limitations

Version metadata only exists on runs where you sent it. Older runs from before you added the field are excluded from comparisons.
Metadata is set at ingestion. If you need to reclassify old runs into a new variant, re-emit them or open a request through contact.

Get started

Analyze

Investigate

Security & privacy

Help

Compare AI agent versions, prompts, and experiments in production

How it works

Example questions

What to attach

Reading the results

What The Context Company does not do

Limitations

​How it works

​Example questions

​What to attach

​Reading the results

​What The Context Company does not do

​Limitations

​Related

How it works

Example questions

What to attach

Reading the results

What The Context Company does not do

Limitations

Related