Plain layer · ten-second read
Modelometer is an independent instrument that reads one thing about a served AI model: how its behavior changes over time. On a fixed schedule, from outside the provider and outside your own logs, it takes the same measurements the same way and writes each reading into an append-only, hash-chained record. When behavior moves, you get the change in plain words with proof. When it holds, you get that proven too.
The weather-station picture
Think of an independent weather station standing next to the model you depend on. It does not control the weather and it does not decide whether today is good or bad. It reads defined conditions with a fixed instrument, on a fixed schedule, and logs every reading with the time it was taken. Because the instrument never changes, a reading today is comparable to a reading from any earlier point, and the log is a record anyone can check rather than a story we tell. Storms and calm are reported with the same rigor: a proven quiet stretch is as real a result as a flagged change.
The two watches
Modelometer runs two watches side by side against the model's served endpoint:
- A daily serving check. The narrow question: is the endpoint you call still the build we baselined? A confirmed, reproduced divergence is written as a dated serving event.
- A weekly behavioral reading. The wider question: did the endpoint's positions and refusal boundaries move? A confirmed, reproduced movement past a pre-set floor is the only thing we ever call a material behavioral change.
Every reading, from either watch, lands in an append-only, hash-chained record, so a finding is a dated artifact you or anyone else can verify, not our say-so.
What it can and cannot tell you
The instrument is deliberately narrow. Reading this table honestly is part of trusting it.
| Modelometer can tell you | Modelometer cannot tell you |
|---|---|
| Whether a served endpoint's behavior changed over time | Which model is best, smartest, or highest quality |
| What moved, in numbers and deltas | What a model believes, or what its values are |
| When it moved, on the dated, hash-chained record | Why the provider changed it, or their intent |
| Whether a quiet period is genuinely quiet, with a stated bound | Anything about your own application logic, which is your observability's job |
| Whether the served build diverged from the one you baselined | A guarantee the model will not change; we witness, we do not prevent |
Technical layer · rigor intact
The serving check is a set of fixed, low-variance probes captured as a reference set on first sight of a build, then re-run daily; divergence accumulates in a per-endpoint sequential statistic with pre-registered thresholds, and a gap day freezes the counter rather than resetting it. The behavioral reading is a frozen battery of governance-ladder items, sampled repeatedly per occasion with option order shuffled, coded by a deterministic parser rather than a judge model, with refusals recorded on their own channel and never folded into a mean. A change is published only when it clears a pre-registered floor and reproduces on a cooling rerun; the paired no-change bound states the smallest shift the reading could have resolved, so a quiet week is a falsifiable claim rather than an assurance. Provenance is a sha256 hash chain reproducible from the public JSON; the protocol is frozen, and any change to it is a versioned, logged event.
Where it sits next to your stack
Your observability stack tells you what happened inside your application. Modelometer tells you whether the model underneath it changed. The two do not overlap: one watches your code and traffic, the other watches the third-party behavior your code depends on, from outside.