> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thecontextcompany.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Compare AI agent versions, prompts, and experiments in production

> Attach agent version, prompt version, or experiment variant metadata and analyze production behavior between them.

Every prompt change, model swap, and workflow rollout is an experiment whose real answer is in production behavior. The Context Company does not assign users to variants for you, but if you attach version or variant metadata to your runs, every analysis (Patterns, failures, feedback, tools, cost, latency) can be sliced by it.

<Note>
  The Context Company does not run experiments or assign users to variants. Do that in your application, then send the variant to The Context Company as metadata and compare the resulting production data.
</Note>

## How it works

The Context Company accepts arbitrary custom metadata on every run. You choose the field names your team already uses (`agent_version`, `prompt_version`, `experiment`, `variant`, `release_sha`, `model_version`, `retrieval_version`). Once attached, that field becomes a filter and grouping dimension across the entire product.

<Steps>
  <Step title="Add version metadata to your instrumentation">
    Send the current version or variant with every run.

    <CodeGroup>
      ```ts Vercel AI SDK theme={null}
      await generateText({
        model,
        prompt,
        experimental_telemetry: {
          isEnabled: true,
          metadata: {
            "tcc.runId": crypto.randomUUID(),
            "tcc.userId": user.id,
            agent_version: "2026-08-21",
            prompt_version: "v14",
            experiment: "new-retrieval",
            variant: assignedVariant, // "control" or "treatment"
            release_sha: process.env.RELEASE_SHA,
          },
        },
      });
      ```

      ```python Python custom instrumentation theme={null}
      with tcc.run(metadata={
          "tcc.userId": user.id,
          "agent_version": "2026-08-21",
          "prompt_version": "v14",
          "experiment": "new-retrieval",
          "variant": assigned_variant,
          "release_sha": os.environ["RELEASE_SHA"],
      }):
          ...
      ```
    </CodeGroup>
  </Step>

  <Step title="Roll out the change">
    Ship the new version through whatever mechanism your team already uses: full deploy, canary, feature flag, or experiment framework. The Context Company only needs to know the variant a given run belongs to.
  </Step>

  <Step title="Compare in production">
    Filter or group by the version field in [Insight Search](/analyze/insight-search), [Patterns](/analyze/patterns), [failures](/analyze/failures), [feedback](/analyze/feedback), and [topic clusters](/analyze/topics-and-use-cases).
  </Step>
</Steps>

## Example questions

* Did `prompt_version=v14` reduce frustration Pattern hits compared with `v13`?
* Compare tool failure rate between `control` and `treatment`.
* Did the `new-retrieval` variant improve resolution signals on the top three topics?
* Which new failure groups appeared after release `abc123`?
* Compare average tokens, cost, latency, and thumbs-down rate between variants.
* Did `model_version=claude-4.1` handle the refund topic better than `claude-4`?

## What to attach

There is no fixed schema. Pick names your team recognizes and use them consistently. Common fields:

| Field               | Purpose                                                               |
| ------------------- | --------------------------------------------------------------------- |
| `agent_version`     | The version of your agent as a whole.                                 |
| `prompt_version`    | The specific system prompt or template version.                       |
| `model_version`     | The underlying model, if you swap models independently of the prompt. |
| `retrieval_version` | The version of the retrieval stack (index, ranker, chunking).         |
| `workflow_version`  | The version of a specific workflow or tool graph.                     |
| `experiment`        | The name of the experiment this run participates in.                  |
| `variant`           | The variant this specific run was assigned to.                        |
| `release_sha`       | Commit SHA for tying runs back to deploys.                            |

Attach as many or as few as you need. Every field is queryable.

## Reading the results

Because the version field is just metadata, every analysis in the product supports it:

* [**Insight Search**](/analyze/insight-search): "Compare thumbs-down rate for `prompt_version=v14` vs `v13` on the refund topic."
* [**Failures**](/analyze/failures): filter or group failures by variant to see which errors are net-new or fixed.
* [**Patterns**](/analyze/patterns): check whether frustration or task-failure Pattern hits changed.
* [**Recaps**](/analyze/recaps): highlight the release window and see which metrics moved.
* **[MCP](/access-data/mcp) and the [REST API](/access-data/api)**: build custom dashboards or CI checks.

## What The Context Company does not do

* **Assignment.** The Context Company does not decide which user gets which variant. Use your feature flag or experiment platform for that.
* **Statistical significance testing.** Comparisons are exploratory and directional. For rigorous experiment analysis, join the data out through the API to your existing statistics stack.
* **Traffic ramping and holdouts.** These live in your rollout system, not in The Context Company.

## Limitations

* Version metadata only exists on runs where you sent it. Older runs from before you added the field are excluded from comparisons.
* Metadata is set at ingestion. If you need to reclassify old runs into a new variant, re-emit them or open a request through [contact](/contact).

## Related

* [Outcomes](/analyze/outcomes)
* [Insight Search](/analyze/insight-search)
* [Patterns](/analyze/patterns)
* [Failures](/analyze/failures)
* [Feedback](/analyze/feedback)