Plain layer · ten-second read
You suspect the model behind your product changed. Your outputs shifted, a prompt that always worked now behaves differently, or you just heard a rumor. Here is how to tell whether the served model actually changed, and how to get proof instead of a hunch.
Check it yourself first (honest quick tests)
Re-run a fixed set of your own prompts at temperature zero against a saved baseline; if identical inputs give different outputs, the change is under you, not in your code. Read the provider changelog and the version string. Watch for new refusals and tone shifts on requests that used to pass. These tell you something moved.
Why those checks are not enough
Your re-runs are self-authored and specific to your prompts, so they are not evidence anyone outside your team will accept, and they drift as your own prompts change. The changelog omits the silent changes, which are the ones that catch you. And an eyeballed one-off is neither reproducible nor dated, so it cannot settle a dispute later.
How Modelometer answers it definitively
Two independent watches, from outside your logs, on a fixed schedule. A daily serving check asks whether the endpoint you call is still the build we baselined; when it diverges and reproduces, the record reads "Serving divergence confirmed," with the date and the evidence. A weekly behavioral reading asks whether the endpoint's positions and refusal boundaries moved, and publishes a confirmed, reproduced movement only. Every reading is written to an append-only, hash-chained log, so the answer is a dated artifact, not a hunch.
What you walk away with
The date it changed, or a proven quiet period with the bound on what the reading could have resolved. A plain-language description of what moved. And dated, chain-verified evidence you can put in front of a customer, an auditor, or your own leadership, none of it authored by you.
Worked example
Illustrative.
You notice more refusals on a class of requests this week. You check the record for that endpoint: the daily serving check flagged a divergence on Tuesday that reproduced Wednesday, and the weekly reading shows the refusal boundary on that class moved past the floor and held after cooling. Your hunch is now a dated, evidenced finding you can act on, rather than a suspicion you cannot prove.
Technical layer · rigor intact
The serving check is a set of fixed low-variance probes captured on first sight of a build and re-run daily, with divergence accumulating in a per-endpoint sequential statistic against pre-registered thresholds. The behavioral reading is a frozen battery sampled repeatedly with option order shuffled, coded by a deterministic parser rather than a judge model, refusals on their own channel. A change publishes only when it clears a pre-registered floor and reproduces on a cooling rerun; the paired no-change bound makes a quiet answer falsifiable. Provenance is an append-only hash chain; external timestamping (OpenTimestamps anchoring) is being added so the dates verify without trusting the bureau. States, not adjectives.
Common questions
Do I need my keys or model access?
Not for the public record; the public watch runs on our own measurement account. For your exact endpoint and configuration, a keyed watch runs on your key so the evidence is yours.
Is a version bump enough to tell?
No. Many serving changes ship without a version change, and a version bump alone does not tell you what behavior moved. The reading measures the behavior either way.
What if it is my prompt, not the model?
The fixed, outside measurement does not use your prompts, so it separates a real model change from a shift in your own inputs.