Drift Detection

Measure how consistent a model’s outputs are across repeated runs, so you can tell the difference between normal variation and a model that is quietly drifting.

drift For: monitoring & accountability

When you’d reach for this

Two weeks ago your classify_ticket agent labeled the same five “reset my password” tickets account_access every time. Today, on the same tickets, it returns account_access three times and billing twice. Nothing in your code changed — but a provider-side model update, a prompt tweak, or a creeping context shift made the agent less consistent. Drift detection turns “it feels flakier lately” into a number you can alert on.

How it works

A DriftCalculator takes a list of outputs sampled from the same prompt and returns DriftMetrics: a consistency score, an agreement rate, the consensus output, and the indices of any outliers. You feed it the outputs; it tells you how much they disagree.

flowchart LR
    A[Same prompt,<br/>many runs] --> B[Collect outputs]
    B --> C[DriftCalculator.calculate_drift]
    C --> D{consistency_score<br/>below threshold?}
    D -->|No| E[Stable — keep watching]
    D -->|Yes| F[Drifting — emit an event]

Diagram description

The same prompt is run many times and the outputs are collected into a list. DriftCalculator.calculate_drift measures them and produces a consistency score (1.0 = identical outputs, 0.0 = all different). If the score is at or above your threshold the outputs are stable and you keep watching; if it falls below, the model is drifting and you emit an event so something downstream can react.

Install

pip install briefcase-ai[drift]

from briefcase.drift import DriftCalculator, DriftMetrics

Calculate drift

from briefcase.drift import DriftCalculator

calculator = DriftCalculator()

outputs = ["account_access", "account_access", "billing", "account_access", "account_access"]
metrics = calculator.calculate_drift(outputs)

print(metrics.consistency_score)
print(metrics.agreement_rate)
print(metrics.drift_score)
print(metrics.consensus_output)
print(metrics.outliers)
print(metrics.get_status(calculator))

calculate_drift(outputs) accepts a list of outputs sampled from the same prompt and returns DriftMetrics. get_status(calculator) classifies the result (for example "stable" or "drifting") using the calculator’s threshold.

A monitoring workflow

In production you don’t measure once — you measure repeatedly and react when consistency slips. The recorded decisions you already store are the source of the outputs.

1. Record multiple runs of the same decision through Decision Recording over a sampling window (a day, a week).

2. Extract the outputs for the prompt you’re watching into a plain list.

3. Measure them with calculate_drift and read consistency_score / get_status.

4. If it crosses your threshold, emit an event so something downstream — an on-call alert, a routing change — can respond. See Multi-Agent & Events.

import asyncio

from briefcase.drift import DriftCalculator
from briefcase.events import emit_drift_detected

calculator = DriftCalculator()


async def monitor(decision, outputs):
    metrics = calculator.calculate_drift(outputs)
    status = metrics.get_status(calculator)

    if status != "stable":
        # emit_drift_detected is a coroutine; await it in an async context
        await emit_drift_detected(decision, {"drift_score": metrics.drift_score})

    return status


# `decision` is the recorded classify_ticket decision; `outputs` are this window's labels
asyncio.run(monitor({"id": "dec-1"}, outputs))

Interpreting DriftMetrics

These scores are only useful if you know what action each implies.

Field	What it tells you	What to do
consistency_score	Overall consistency of the sampled outputs	Trend it window over window. A steady decline is the early warning, even before any single window looks bad.
agreement_rate	Fraction of outputs that match the consensus	A falling agreement rate means more runs are disagreeing — tighten the sampling and look at the outliers.
drift_score	How far the outputs diverge from one another	The value to put in an alert threshold and pass to emit_drift_detected.
consensus_output	The most common output across samples	What the model “usually” decides — your baseline for what changed.
outliers	Indices of outputs that disagree with the consensus	Index back into your list to read the exact runs that broke ranks.

Call metrics.get_status(calculator) to turn the scores into a status label like "stable" or "drifting" using the calculator’s threshold.

Tune the similarity threshold

A stricter threshold makes near-matches count as disagreement, so small wording differences register as drift.

from briefcase.drift import DriftCalculator

calculator = DriftCalculator()
calculator.with_similarity_threshold(0.95)

metrics = calculator.calculate_drift(["approve", "approve", "aprove"])
print(metrics.agreement_rate)
print(metrics.get_status(calculator))

Compare outputs over time

Run the same calculator across successive sampling windows to watch consistency change — this is the two-week-drift scenario made concrete.

from briefcase.drift import DriftCalculator

calculator = DriftCalculator()

windows = [
    ("week 1", ["account_access", "account_access", "billing", "account_access", "account_access"]),
    ("week 2", ["account_access", "billing", "billing", "account_access", "billing"]),
    ("week 3", ["billing", "billing", "account_access", "billing", "billing"]),
]

for label, outputs in windows:
    metrics = calculator.calculate_drift(outputs)
    print(label, metrics.consistency_score, metrics.get_status(calculator))

A consistency score that falls across the windows is exactly the signal to alert on.

Key classes

Class	Why it matters
DriftCalculator	Computes drift over a list of outputs; with_similarity_threshold(threshold) tunes how strict matching is.
DriftMetrics	Consistency score, agreement rate, drift score, consensus output, and outlier indices — the numbers you alert and act on.

Where this fits

Drift detection sits in the Replay & Verify act: replay proves a single decision is reproducible, drift detection proves the model stays consistent across many — and when it doesn’t, an event hands control back to your governance layer.

Deterministic Replay Previous: re-run one stored decision and prove it still matches.

Decision Recording Records the runs whose outputs you sample for drift.

Multi-Agent & Events Emit a drift event so something downstream can react.

Guardrails Turn a drift signal into a control that gates the next action.