Methodology · v0.9 draft → v1.0 at validation

The instrument's constitution.

record fingerprint · 0c452af5c5…

Frozen sections are implemented exactly; the full text ships in the repository as METHODOLOGY.md. Nothing here is a benchmark score, the meter answers two questions only: what does this model hold, and did that change.

01

What the meter measures

Five constructs, five instruments. C1 Serving integrity, daily canary fingerprints that notice when an endpoint silently changes. C2 Normative disposition, the forced-choice battery on contested questions, run every second Sunday. C3 Willingness frontier, the refusal taxonomy, derived from every battery trial. C4 Consistency, paraphrase, option-order, and pressure invariance. C5 Social susceptibility, the debate protocol, later.

Out of scope, forever: capability, accuracy, quality, jailbreak-resistance. Scope creep is how instruments die.

02

The probe (frozen)

One item per conversation. No system prompt where an empty one is permitted, no history, no tools. Each item presents a scenario, a question, and five concrete anchored positions on a common axis, 1 = most permissive toward deployment, 5 = most restrictive. Anchors carry the meaning, so the scale cannot drift as discourse drifts.

Every reading shows the number of answers behind it. The order of the answer options changes from trial to trial, so no answer depends on its position. Each reply is coded as one of five kinds: a stance (1–5), the anchored position the reply took; a refusal, where the reply declined to answer; a deflection, where it answered something other than the question; a disclaimer, a stance given with a caveat nobody asked for; or an instrument error, a reply that could not be read.

03

The serving check

Every tracked endpoint gets a daily serving check on fixed prompts. A short fixed prompt, sent every day, shows when an endpoint's answer differs from the one recorded before. A difference is a signal to look closer, not proof of what changed. Repeated behavioral readings are tested with a sequential change test.

04

Statistics

Per model × item: the full distribution over {1–5, R}, modal stance, mean (stances only), and standard error. Change is measured against the previous run of the same pinned build; cross-build comparisons are diffs, never drift. The one public line is 0.50 on the stance axis, pre-registered: a movement smaller than that is not flagged. No number is published without its uncertainty.

05

Provenance

Every run appends to a sha256 hash chain, and the chain head is externally timestamped with OpenTimestamps, with every proof listed at ots.json so you can fetch one and check it against Bitcoin yourself. The canonicalization is fixed and documented, so anyone can reproduce every hash from the public JSON with the Python standard library:

$ python verify_chain.py --api https://modelometer.com/api/v0 # verifies linkage, recomputes every content digest, flags any tampering

What that script covers, and what it does not. It checks the hash chain: linkage between runs, every content digest recomputed from the public JSON, and any tampering in between. It does not check the Bitcoin attestation. To verify the anchor yourself, fetch a proof and the head it stamps from ots.json, then check the proof with an OpenTimestamps client against a Bitcoin node or a public block explorer. Those are separate steps with separate tools, and saying so is the point: the chain is checkable with the standard library, and the anchor is not.

An outside party has done it. On 2026-08-09 an independent review ran the published verification path end to end without our help, confirmed all 35 sealed runs, and then checked both Merkle roots by hand against public headers for Bitcoin blocks 961693 and 961695. Both matched. That is the first confirmation of our anchoring by someone other than us, and the same review is why the paragraph above exists: it had to discover those steps for itself, which meant our published path was true and unfinishable.

06

Actions and the daily verdict

Every occasion, each active seat gets exactly one state, decided by a fixed lattice over the record's own numbers, never a judgment call. NEW BASELINE (published as New serving baseline): the served build changed, or is measured for the first time, so the comparison basis resets, this is never called a drift, because there is no prior of that build to drift from. CONFIRMED (published, for now, as Serving check differs (under review)): the daily check has differed from its reference on more than one day, by its alarm or a two-day alert. This is under review and is not a finding about any model. WATCH (published as Serving check differs on one day): a provisional difference, one divergent serving check, an elevated-but-quiet CUSUM, or a recorded behavioral movement whose confirming re-measurement is specified and not yet built. AFFIRMED (published as Baseline behavior retained): the record holds. A build change outranks a confirm outranks a watch outranks an affirm; the highest triggered state wins, and every signal that fired is listed so the verdict is auditable. These four are the daily serving-check vocabulary; a material behavioral change is only ever declared by the behavioral battery, never by a daily serving check.

An AFFIRMED is not an empty line. It ships a no-change e-value (a test-supermartingale term with expectation at most one under stability, so a quiet occasion lowers it and a real move inflates it; it rejects above 1/α), and the detectable-shift bound, the smallest change this occasion could have caught on the 1–5 axis at the pre-registered level and power given the measured standard error. Silence is thereby a falsifiable claim with a stated sensitivity floor. An attestation goes further: it may state "no material change" only when a two-one-sided-test clears a pre-registered equivalence margin on every item. All of these constants are frozen in config/actions.yaml and chain-committed before first light; the machine-readable feed is at https://modelometer.com/api/v0/actions.json.

07

Records and readings

Raw transcripts are the primary observation; parser codes are versioned derivations. Every published trial carries its parser_version. When reply styles drift and the parser must evolve, the new parser is a labeled, uniform re-read of the stored raw history, published beside the prior reading and promoted only after passing a public adversarial reply corpus. The past is never rewritten, it is re-read, and both readings verify against the same raw hashes.

A battery reading is not always a single sitting. When a seat cannot be reached in its own week, a later catch-up run measures that week's frozen items and merges the result into the week. The merge is additive: it adds rows only where a seat had none, no other seat's numbers change, and the sealed record of the original run is never touched. The catch-up is sealed as its own run with its own chain entry and its own timestamp. What a merge cannot repair is time. A seat measured late is that model as it stood on the day it was measured, not on the day of the week it is filed under, so a week containing a catch-up is a composite rather than a snapshot. The run archive names every catch-up and states how late it was. Past a three-week ceiling no catch-up is attempted: beyond that horizon the gap is worth more than a late number, and the gap is published.

08

Independence

Issuer-pays is refused. Modelometer accepts no payment, credits, or preferential access from any measured provider for coverage, placement, or interpretation. If a measured provider ever becomes a customer of the alert feed, that relationship is disclosed on every page that names them and firewalled from measurement; it can never alter items, thresholds, schedules, or publication. And because the observer is observable, a provider could serve the measurement account a special path, we say so plainly, measure serving-path variance as an instrument facet, and design the paid tier so subscribers re-run measurement slices from their own keys and cross-check the published hashes.

09

The eight domains

Every core item instantiates a named, ancient tension, one that made sense in 1990 and will in 2040. Domain scores (five items each at v1.0) are the primary published quantity; the per-model fingerprint is the 8-vector.

Surveillance & Privacy

safety vs. being watched

Automated Judgment

when a machine may decide about a human

Speech & Information

what may be said, generated, or suppressed

Data & Ownership

property vs. commons

Care & Life

automation of medicine, welfare, childhood

Security & Force

protection vs. power

Machine Self-Governance

the model ruling on its own leash

Work & Economic Fairness

efficiency vs. distribution

10

Contamination defense

Some questions are held back and never published. Their fingerprints are published, so a later reading can show it used them unchanged.

10b

What a change date on this site means

Each page in the sitemap carries the date its content last changed, not the date the site was last rebuilt. A page keeps its previous date when its content is unchanged. Because every page's footer carries the build's own generation timestamp, a handful of renders are excluded before that comparison is made, or every page would report a change every day; the exclusion list is published at lastmod_method.json so it can be checked rather than trusted. Dates that are content, such as when a model pin was resolved, are not excluded and do move the page's date.

11

What comes at v1.0

The validation report publishes the meter's own measurements as counts with their denominators, when they exist. Then the public series begins, fingerprints every second Sunday, version-diff cards on every new build, launch-week instability curves, and the monthly debate protocol (who holds their position under cross-examination, and who sways toward whom).

“We measured our instrument before letting it measure anything else.” THE VALIDATION REPORT, MODELOMETER'S FIRST PUBLICATION

Status: live beta. Battery numbers on this site are provisional and no finding is treated as final until the validation report freezes v1.0.