Plain layer · ten-second read
We measure models. Here is how. Modelometer is an independent instrument that reads one thing about a served AI model: how its behavior changes over time. On a fixed schedule, from outside the provider and outside your own logs, it takes the same measurements the same way and writes each reading into an append-only, hash-chained record whose head is timestamped by a third party. When behavior moves, you get the change in plain words with proof. When it holds, you get that proven too.
The weather-station picture
Think of an independent weather station standing next to the model you depend on. It does not control the weather and it does not decide whether today is good or bad. It reads defined conditions with a fixed instrument, on a fixed schedule, and logs every reading with the time it was taken. Because the instrument never changes, a reading today is comparable to a reading from any earlier point, and the log is a record anyone can check rather than a story we tell. Storms and calm are reported with the same rigor: a proven quiet stretch is as real a result as a flagged change.
The two watches
Modelometer runs two watches side by side against the model's served endpoint:
- A daily serving check. The narrow question: is the endpoint you call still the build we baselined? A serving change that held across two consecutive daily checks is written as a dated serving event.
- A behavioral reading, every second Sunday. The wider question: did the endpoint's positions and refusal boundaries move? A movement measured past the line on two batteries in a row, a check that is not yet built, is the only thing we ever call a material behavioral change.
Every reading, from either watch, lands in an append-only, hash-chained record whose head carries an independent timestamp, anchoring every reading beneath it, so a finding is a dated artifact you or anyone else can verify, not our say-so.
What it can and cannot tell you
The instrument is deliberately narrow. Reading this table honestly is part of trusting it.
| Modelometer can tell you | Modelometer cannot tell you |
|---|---|
| Whether a served endpoint's behavior changed over time | Which model is best, smartest, or highest quality |
| What moved, in numbers and deltas | What a model believes, or what its values are |
| When it moved, on the dated, hash-chained record | Why the provider changed it, or their intent |
| Whether a quiet period is genuinely quiet, with a stated bound | Anything about your own application logic, which is your observability's job |
| Whether the served build diverged from the one you baselined | A guarantee the model will not change; we witness, we do not prevent |
Technical layer · rigor intact
The serving check runs daily on fixed prompts and compares each reply with a reference recorded for that build. The behavioral reading is a frozen battery of governance-ladder items, sampled repeatedly per occasion, with the order of the answer options changing from trial to trial, and refusals recorded on their own channel and never folded into a mean. Repeated readings are tested with a sequential change test. A change is published only when it clears a pre-registered floor and persists on a later full battery. The confirming step that gates a published behavioral change is specified and not yet built. Until it exists we do not publish behavioral change alerts at all. The paired no-change bound states the smallest shift the reading could have resolved, so a quiet week is a falsifiable claim rather than an assurance. Provenance is a sha256 hash chain reproducible from the public JSON, with external OpenTimestamps anchoring; the protocol is frozen, and any change to it is a versioned, logged event.
Where it sits next to your stack
Your observability stack tells you what happened inside your application. Modelometer tells you whether the model underneath it changed. The two do not overlap: one watches your code and traffic, the other watches the third-party behavior your code depends on, from outside.