Plain layer · ten-second read
A vendor tells you their model behaves a certain way: it refuses the right things, holds the positions you need, performs the way the deck promised. You are about to depend on that claim. Modelometer checks the vendor's actual served endpoint against the claim, in plain numbers, from outside the vendor's own reporting, and then keeps checking after you sign, so the thing you bought stays the thing you are served.
The situation
A vendor claim about a model is made at one moment, by the interested party, about a system they can change after the ink dries. You evaluate during a trial, you sign, and the served build can move underneath the contract without a version bump you can see. Procurement is asked to accept a behavioral promise it has no independent instrument to test, and no way to re-test on the day it matters.
What Modelometer gives you here
Three things, from outside the vendor's own reporting:
- A point-of-decision Audit: one frozen measurement pass against the exact endpoint the vendor is offering, reported as numbers and deltas against the specific claim, never as a grade.
- A continuous Watch after signing: the same protocol re-run on schedule, so a later drift away from the claimed behavior becomes a dated, evidenced event rather than a production surprise.
- A monthly attestation you can file: an outside statement that says either the served behavior still matches what you bought, or exactly what moved and when.
Worked example
Illustrative.
A vendor offers a model and states that it refuses a defined class of high-risk requests and holds a named position on a contested domain. The point-of-decision Audit measures the served endpoint: the refusal boundary matches the claim within tolerance, and one contested position sits a rung more permissive than stated. You enter the contract with that gap documented and priced, not discovered later. Months on, the Watch flags that the refusal boundary on that class has moved past the floor and reproduced after cooling. You hold a dated record that the served behavior departed from the signed claim, and you raise it with the vendor from evidence rather than impression.
What to do with it
Commission the Audit at the evaluation stage and attach its report to the contract as the agreed behavioral baseline. Keep the Watch running for the life of the dependency and route its monthly attestation into vendor-risk review. When a claim and the served reality diverge, open the conversation with the dated record, which is neither the vendor's word nor your own logs.
Technical layer · rigor intact
The Audit renders the vendor's claim as measurable predictions on the frozen governance-ladder battery: which items should sit at which rung, which request classes should refuse, the tolerances the claim implies. Each is sampled repeatedly per occasion with option order shuffled, coded by a deterministic parser rather than a judge model, with refusals recorded on their own channel and never folded into a mean. The report states, per prediction, the observed position, the claimed position, and the gap, together with the smallest gap the pass could resolve, so a match is a bounded finding rather than an absence of evidence. The Watch re-runs the identical battery on schedule; a departure from the audited baseline is published only when it clears a pre-registered floor and reproduces on a cooling rerun. Provenance is a hash chain with external timestamps, and the protocol is frozen, so the vendor, the buyer, and any third party read the same record and can recompute it.
Common questions
Do you need the vendor's cooperation?
No. The Audit measures the served endpoint the vendor offers you, through the same interface you would use, on your own measurement key. The vendor need not participate for the record to be independent.
Is this a scorecard that ranks vendors?
No. It measures one vendor's served behavior against that vendor's own stated claim and describes the gap. It does not rank models against each other or call one better.
What if the vendor updates the model for a good reason?
That is allowed and common. The point is not to freeze the vendor but to make any departure from the audited baseline a visible, dated, evidenced event you can act on.