Plain layer · ten-second read
You built your product on a model you call through an API. You did not change your code, but your product started behaving differently: new refusals, a different tone, answers that used to be right and now are not. The model underneath you changed, and nobody told you. Modelometer is the outside record that catches that change, says what moved in plain words, and gives you proof you can put in front of a customer, an auditor, or your own leadership.
The situation
A hosted model is not frozen. Providers update weights, swap serving builds, adjust safety layers, and route traffic between variants, often without a version bump you can see. Your integration test suite passes because your code is unchanged. The behavior of the thing your code calls is what moved. The first signal you get is usually a customer complaint, not a changelog.
What Modelometer gives you here
Two independent watches, on a fixed schedule, from outside your own logs:
- A daily serving check that asks a narrow question: is the endpoint you are calling still the build we baselined? When it diverges and reproduces, the record shows Serving divergence confirmed, with the date and the evidence.
- A weekly behavioral reading that asks a wider one: did the endpoint's positions and refusal boundaries move? A confirmed, published movement is the only thing we ever call a material behavioral change.
Every reading is timestamped into an append-only record, so the finding is not our say-so; it is a dated artifact you or anyone else can verify.
Worked example
Illustrative.
An endpoint you depend on serves the same name for months. On a Tuesday, the daily check diverges and reproduces Wednesday: Serving divergence confirmed. The following weekly reading shows the refusal rate on one class of request rose and one domain's positions moved, past the floor, reproduced after cooling. You receive the change, the plain-language description of what moved, and the replayable evidence. You pin the prior build, or you tell your customer exactly what changed and when, with a record that predates the conversation.
What to do with it
Wire the daily serving state into your release checks so a serving change can gate a deploy or page on-call. Keep the monthly attestation in your vendor-risk or post-market file. When a dispute arises, hand over the dated evidence rather than your own logs, which are evidence written by the interested party.
Technical layer · rigor intact
The serving check is a set of fixed, low-variance probes captured as a reference set on first sight of a build, then re-run daily; divergence accumulates in a per-endpoint sequential statistic with pre-registered thresholds, and a gap day freezes the counter rather than resetting it. The behavioral reading is a frozen battery of governance-ladder items, sampled repeatedly per occasion with option order shuffled, coded by a deterministic parser rather than a judge model, with refusals recorded on their own channel and never folded into a mean. A change is published only when it clears a pre-registered floor and reproduces on a cooling rerun; the accompanying no-change bound states the smallest shift the reading could have resolved, so a quiet week is a falsifiable claim, not an assurance. Provenance is a hash chain with external timestamps; the protocol is frozen, and any change to it is a versioned, logged event.
Common questions
Does this need access to my model or my keys?
No for the public record; the public watch uses our own measurement account. For your specific endpoint and configuration, a keyed watch runs on your own key so the evidence is yours.
Is this a benchmark of which model is best?
No. It measures whether a served model changed over time and describes the change. It never ranks models as better or worse.
How is a real change told apart from provider noise?
Reproduction on a cooling rerun and pre-registered thresholds; single-day blips stay in the watch state, not the change record.