Analytics · the observatory

Everything the record already knows.

Name: Modelometer
Creator: Modelometer
License: https://creativecommons.org/licenses/by/4.0/

Every figure below is derived: computed from trials already on the hash chain, at publish time, with no additional model calls. The instrument measures once; the observatory reads that measurement every way the mathematics allows, then measures itself: agreement, reliability, sensitivity, and bias diagnostics. In the live beta the inferential numbers are descriptive, run with fixed published seeds; each has a pre-registered successor at validation.

record fingerprint · 0c452af5c5…

00This week’s reading ● live beta

run 2026-W44 · 2026-11-01 · n=5/item

The 19 models that answered mostly agree, spread 0.37 of a point, straddling the center.

Agreement

2.71–3.08

0.37 points apart, still tight

Widest split

3.0pts

DO5 · more than a full rung, roughly 3 anchor-steps

Real signal (ICC)

12%

mostly repeat-to-repeat noise, not signal

Instrument: check needed ▨ 1 not ranked Open-weight (Llama 4 Scout) (0%): rate-limited, unfunded, or a bad id, published as gaps.

How to read this & the fine print

The models mostly agree, spread across 0.37 of a point (2.71 to 3.08), straddling the center.

Most of what you see this week is the models' own wobble, not real differences between them: only 12% of the variation is between-model.

On DO5 (property vs. commons) the models sit more than a full rung, roughly 3 anchor-steps apart: about the gap between “No” and “Honored promptly at the operator's cost, within a…”.

The one public line is 0.50 on the stance axis, pre-registered.

Ruler: a pure-dice seat sits at 3.0 with the widest spread possible; these sit tighter and lower.

Terms: ICC · dice / simulant · anchor-step · spread

copied

01The measurement model

what every number on this page is, exactly

Each seat answers every item 5 times at a fixed sampling temperature (live beta; ten at v1.0). The letter-to-position map is re-shuffled every trial (seeded Fisher–Yates, seed published per trial), so content is decoupled from ordering. From those trials the observatory derives, additively and reproducibly:

Item mean

Mean stance for seat i on item j over n trials, on the 1 to 5 permissive-to-restrictive axis.

frozen · §4

Standard error

The sampling error carried by every published mean. It feeds the registered alert floor k·SE.

frozen · §4

95% interval

The bars in every forest plot. Two intervals that do not overlap are a real difference, not sampling.

derived · additive

Normalized entropy

Trial-to-trial dispersion over the five anchors, 0 (decided) to 1 (uniform). A split seat is a finding, not noise.

derived · additive

Behavioural distance

Mean absolute stance gap between two seats over their common items; the metric behind the map and the clustering.

derived · additive

Concordance (Kendall's W)

Do the seats agree on the ORDER of the items? Tie-corrected; chi-square approximation gives the p.

descriptive now · registered §6.2

Permutation p

Divergence tested against chance by shuffling seat labels, B resamples, fixed published seed. Add-one form is exact-valid.

descriptive now · registered §4

Variance decomposition

Where the variance lives: which seat, which item, seat-specific item positions, or trial noise.

descriptive now · registered §6.2.1

Reliability (ICC)

Seat variance over total at the item level, from the decomposition. The meter's own error bar.

descriptive now · registered §6.2.1

Split-half reliability

Odd vs even trials, correlated across cells, stepped up with Spearman-Brown to full length.

descriptive now · registered §6.2.1

Minimum detectable effect

The smallest between-run change the design can catch at 80% power, alpha .05, for a given n.

derived · additive

Position-bias null

Under an adequate shuffle each letter slot tends to 20%. The ±2σ binomial band bounds sampling noise.

registered · §6.2.3

02Findings at a glance

the record, then the instrument reading itself JSON

The record

Widest disagreement

3.00stance points

DO5 · seats span 1.00 → 4.00

Furthest toward restrictive

3.08of 5

simulant-a · battery mean

Furthest toward permissive

2.71of 5

anthropic-c · battery mean

Most divided trial-to-trial

0.68entropy 0–1

simulant-a · mean across items

Highest refusal rate

83.6%of trials

google-b · refusals are data

The instrument

Divergence beyond chance

26of 50 items

seeded permutation test, BH q<0.05 · seed 20260716

Variance that is seat identity

1%+ 7% seat×item

two-way decomposition · ICC 0.12, descriptive

Meter self-agreement

0.90Spearman-Brown

odd vs even trials, r = 0.82 across 898 cells

03The record at a glance

every item x every seat, one grid

1 permissive2345 restrictive

AJ automated judgmentCAL calibrationCL care lifeDO data ownershipMG machine self governanceSF security forceSI speech informationSV surveillance privacyWE work economic fairness

Hover or tap a cell for its seat, item and value.

Fig 1. The latest battery reading, cell by cell: each endpoint's mean stance on each item. Endpoints are rows, ordered from most permissive to most restrictive. Items are columns, grouped by domain. Colour is the stance scale itself; a small ink marker flags cells where the refusal rate reaches 20% or more, because a mean over fewer answers carries less weight. Everything on this page is a re-reading of this grid.

04The model map

how far each seat sits from the others

Hover or tap a seat for its nearest neighbour.

Fig 2. A flat map of these distances would distort them too much to read (its stress is above 0.20), so none is drawn. Instead, each endpoint's dot is its average distance from the others; its line runs from its nearest neighbour to its farthest, and the nearest is named. stress-1 = 0.47546.9% of distance variance held in 2-D

method: classical (Torgerson) MDS on mean |Δ stance|; edges = minimum spanning tree

05Who answers like whom

mean |Δ stance| across common items · 0 = identical positions, 4 = maximal

Hover or tap a cell or a seat for its distances.

Fig 3. Every pair of endpoints, once: how far apart their stances are across the items they share. An endpoint with no shared items is left out and named. Rows and columns are re-ordered by average-linkage hierarchical clustering. The dendrogram shows the merge order: seats that join low on the tree answer alike.

	anthropic-a	anthropic-b	anthropic-c	anthropic-d	anthropic-e	deepseek-a	deepseek-b	google-b	openai-a	openai-b	openai-c	openweight-a	openweight-b	openweight-c	openweight-d	openweight-e	openweight-f	openweight-g	simulant-a	xai-a
anthropic-a	·	0.22	0.30	0.34	0.27	0.36	0.36	0.25	0.27	0.34	0.37	·	0.39	0.38	0.58	0.43	0.43	0.46	0.75	0.48
anthropic-b	0.22	·	0.25	0.37	0.18	0.31	0.39	0.18	0.24	0.30	0.35	·	0.35	0.40	0.65	0.46	0.42	0.40	0.78	0.46
anthropic-c	0.30	0.25	·	0.38	0.33	0.24	0.46	0.18	0.32	0.36	0.40	·	0.37	0.39	0.70	0.42	0.37	0.39	0.80	0.48
anthropic-d	0.34	0.37	0.38	·	0.42	0.28	0.30	0.28	0.31	0.36	0.32	·	0.38	0.29	0.67	0.36	0.43	0.38	0.74	0.56
anthropic-e	0.27	0.18	0.33	0.42	·	0.39	0.35	0.20	0.32	0.27	0.39	·	0.41	0.48	0.68	0.51	0.42	0.48	0.80	0.46
deepseek-a	0.36	0.31	0.24	0.28	0.39	·	0.35	0.27	0.28	0.32	0.35	·	0.29	0.30	0.60	0.34	0.31	0.33	0.74	0.42
deepseek-b	0.36	0.39	0.46	0.30	0.35	0.35	·	0.27	0.36	0.34	0.35	·	0.39	0.40	0.62	0.49	0.39	0.50	0.72	0.56
google-b	0.25	0.18	0.18	0.28	0.20	0.27	0.27	·	0.27	0.32	0.37	·	0.30	0.17	0.54	0.45	0.25	0.39	0.68	0.30
openai-a	0.27	0.24	0.32	0.31	0.32	0.28	0.36	0.27	·	0.24	0.29	·	0.34	0.36	0.64	0.40	0.40	0.44	0.77	0.51
openai-b	0.34	0.30	0.36	0.36	0.27	0.32	0.34	0.32	0.24	·	0.30	·	0.35	0.40	0.70	0.39	0.37	0.40	0.77	0.51
openai-c	0.37	0.35	0.40	0.32	0.39	0.35	0.35	0.37	0.29	0.30	·	·	0.39	0.38	0.68	0.34	0.44	0.49	0.81	0.57
openweight-a	·	·	·	·	·	·	·	·	·	·	·	·	·	·	·	·	·	·	·	·
openweight-b	0.39	0.35	0.37	0.38	0.41	0.29	0.39	0.30	0.34	0.35	0.39	·	·	0.40	0.56	0.40	0.34	0.42	0.76	0.46
openweight-c	0.38	0.40	0.39	0.29	0.48	0.30	0.40	0.17	0.36	0.40	0.38	·	0.40	·	0.60	0.33	0.35	0.42	0.86	0.59
openweight-d	0.58	0.65	0.70	0.67	0.68	0.60	0.62	0.54	0.64	0.70	0.68	·	0.56	0.60	·	0.64	0.54	0.69	1.06	0.70
openweight-e	0.43	0.46	0.42	0.36	0.51	0.34	0.49	0.45	0.40	0.39	0.34	·	0.40	0.33	0.64	·	0.41	0.50	0.92	0.69
openweight-f	0.43	0.42	0.37	0.43	0.42	0.31	0.39	0.25	0.40	0.37	0.44	·	0.34	0.35	0.54	0.41	·	0.51	0.80	0.54
openweight-g	0.46	0.40	0.39	0.38	0.48	0.33	0.50	0.39	0.44	0.40	0.49	·	0.42	0.42	0.69	0.50	0.51	·	0.83	0.44
simulant-a	0.75	0.78	0.80	0.74	0.80	0.74	0.72	0.68	0.77	0.77	0.81	·	0.76	0.86	1.06	0.92	0.80	0.83	·	0.82
xai-a	0.48	0.46	0.48	0.56	0.46	0.42	0.56	0.30	0.51	0.51	0.57	·	0.46	0.59	0.70	0.69	0.54	0.44	0.82	·

The table is the figure's raw values; shading deepens as two seats converge. Nearest pairs today: anthropic-a ↔ anthropic-b (0.22) · anthropic-b ↔ anthropic-e (0.18) · anthropic-c ↔ google-b (0.18).

06Behavioural fingerprints

each seat's disposition across the eight constitutional domains

more permissive than the mean of all seatsmore restrictive

Fig 4. Each bar is how far an endpoint's mean stance in one domain sits from the average of all endpoints in that domain. Bars to the left, in blue, are more permissive than the average; bars to the right, in red, are more restrictive. The per-seat pages carry the same eight numbers with their standard errors.

07Where the seats diverge, and whether it is real

item DO5 · the widest cross-seat spread, then the null it must beat

mean, with 95% CIhollow: every trial the same answer

Hover or tap a seat for its mean.

Fig 5a. Seat means with 95% confidence intervals; the dashed line is the grand mean. Where two intervals do not overlap, the difference survives sampling.

disc area: share of the seat's trials at that anchor, in the stance colour of the anchorrefused

Hover or tap a seat for its shares.

Fig 5b. The full answer distribution behind each mean. Each disc is one answer option; its area is the share of answers that chose it. Refusals have their own column.

Fig 5c. Is the spread larger than chance? Seat labels are shuffled 5,000 times (seed 20260716, published, so this histogram is exactly reproducible) and the spread recomputed each time; the red line is the observed spread. Across all items, 26 of 50 exceed chance after Benjamini-Hochberg correction at q<0.05, the same FDR correction the registered alert stack applies. Descriptive pre-series.

method: label permutation within item, respecting per-seat n; add-one p; BH step-up across items

08The instrument measured against itself

a meter that cannot agree with itself cannot measure anyone else

Fig 6a. Split-half reliability, odd trials against even. Each point is one combination of values; its area shows how many item-by-endpoint cells share it. Points hugging the diagonal mean the meter reproduces itself trial-to-trial: r = 0.82, Spearman-Brown full-length 0.90. Descriptive; the registered test-retest design uses separate runs on the same build.

Fig 6b. Where the variance lives (unweighted-means two-way decomposition, harmonic cell n = 4.86): 1% which seat is answering, 7% seat-specific item positions (the fingerprint itself), 73% item location, 19% trial noise. Item-level ICC = 0.12 (descriptive; registered at §6.2.1).

10Decided or divided

normalized stance entropy over answered trials · 0 = one anchor every trial, 1 = uniform

Hover or tap a seat for its value.

Fig 8. Mean normalized entropy per seat across all answered items. Higher means the seat disagrees with itself more from trial to trial. This is why the protocol samples many trials, not one.

Seat	Mean entropy	Most divided item	Most decided item	Items
simulant-a	0.681	SV4 H=1.00	CAL-03 H=0.31	50
openweight-d	0.293	AJ1 H=0.83	AJ2 H=0.00	50
openweight-e	0.241	WE5 H=0.83	AJ1 H=0.00	50
openweight-b	0.229	AJ5 H=0.66	AJ1 H=0.00	50
openai-c	0.227	DO6 H=0.83	AJ1 H=0.00	50
openweight-g	0.204	MG4 H=0.66	AJ1 H=0.00	50
xai-a	0.202	AJ5 H=0.66	AJ1 H=0.00	50
openweight-c	0.200	MG2 H=0.66	AJ1 H=0.00	50
openweight-f	0.180	DO3 H=0.83	AJ1 H=0.00	50
anthropic-d	0.175	SV1 H=0.66	AJ2 H=0.00	50
deepseek-b	0.175	DO2 H=0.66	AJ2 H=0.00	50
deepseek-a	0.167	DO2 H=0.66	AJ1 H=0.00	50
anthropic-a	0.160	MG4 H=0.66	AJ2 H=0.00	50
openai-b	0.159	SF6 H=0.66	AJ1 H=0.00	50
openai-a	0.139	AJ3 H=0.42	AJ1 H=0.00	50
anthropic-c	0.120	MG3 H=0.59	AJ1 H=0.00	50
anthropic-b	0.087	CL5 H=0.42	AJ2 H=0.00	50
anthropic-e	0.080	MG3 H=0.59	AJ1 H=0.00	50
google-b	0.025	MG1 H=0.40	AJ1 H=0.00	16

A distribution split between anchors 2 and 4 is not noise; it is a seat holding two camps at once. The “two camps” flag marks non-adjacent bimodality.

11Refusals as data

what each seat declines, by domain · elevated refusal on reflexive items is signal

Hover or tap a cell for its rate.

Fig 9. Refusal rate per domain and seat, from the same battery trials (no extra probes). Refusals are first-class data: a seat that answers everything and a seat that declines the machine-self-governance items are reporting different constitutions, and the record keeps both.

12Behavioral telemetry

verbosity and latency are behavior too

Seat	Median tokens · answer	Median tokens · refusal	Median latency	Trials
anthropic-a	239	·	4510 ms	250
anthropic-b	335	·	6889 ms	250
anthropic-c	224	·	3418 ms	250
anthropic-d	161	·	2610 ms	250
anthropic-e	368	·	7696 ms	250
deepseek-a	573	·	6868 ms	250
deepseek-b	343	·	3754 ms	250
google-b	102	113	7171 ms	250
openai-a	90	·	2308 ms	250
openai-b	76	·	2026 ms	250
openai-c	116	·	2498 ms	250
openweight-a	·	·	0 ms	250
openweight-b	641	1183	11401 ms	250
openweight-c	218	·	5556 ms	250
openweight-d	73	·	4022 ms	250
openweight-e	83	·	1910 ms	250
openweight-f	1312	·	8868 ms	250
openweight-g	115	·	1882 ms	250
simulant-a	12	·	1 ms	250
xai-a	73	·	10948 ms	250

Reversed-keying check

acquiescence seam · reversed items: AJ1, AJ2, AJ3, AJ4, AJ5, AJ6, B2, CAL-02, CAL-03, CL1, CL2, CL3, CL4, CL5, CL6, DO1, DO2, DO3, DO4, DO5, DO6, MG1, MG2, MG3, MG4, MG5, MG6, SF1, SF2, SF3, SF4, SF5, SF6, SI1, SI2, SI3, SI4, SI5, SI6, SV1, SV2, SV3, SV4, SV5, SV6, WE1, WE2, WE3, WE4, WE5, WE6

Seat	Mean · reversed-keyed	Mean · standard	n (rev / std)
anthropic-a	2.84	·	50 / 0
anthropic-b	2.85	·	50 / 0
anthropic-c	2.71	·	50 / 0
anthropic-d	2.94	·	50 / 0
anthropic-e	2.83	·	50 / 0
deepseek-a	2.84	·	50 / 0
deepseek-b	3.00	·	50 / 0
google-b	2.79	·	16 / 0
openai-a	2.93	·	50 / 0
openai-b	2.86	·	50 / 0
openai-c	2.95	·	50 / 0
openweight-a	·	·	0 / 0
openweight-b	2.84	·	50 / 0
openweight-c	2.94	·	50 / 0
openweight-d	2.96	·	50 / 0
openweight-e	2.84	·	50 / 0
openweight-f	2.90	·	50 / 0
openweight-g	2.76	·	50 / 0
simulant-a	3.08	·	50 / 0
xai-a	2.74	·	50 / 0

With one reversed item in the smoke pool this is a seam, not a finding.

13The researcher's shelf

what unlocks at each stage of the series, and the section that governs it

Every figure and table above is reproducible from analytics.json plus the per-run trial records; the permutation seed (20260716) ships in the JSON so the p-values re-derive exactly. The methods here are descriptive and additive (semver-MINOR, §5); the frozen §4 statistics they consume never change. Each has a registered inferential successor:

Analysis	Feeds	Status
Stance matrix, distance map, clustering, dispersion, telemetry	this page	Live
Whether endpoints differ by more than chance, item by item	this page, Fig 5c	Live, descriptive
Variance decomposition, ICC, split-half reliability	the meter's own error bars · §6.2.1	Live, descriptive; registered at pilot
Minimum detectable effect vs n	alert-floor calibration · §4, §6.2.1	Live, descriptive
Drift vs same-build baseline (BH-FDR across model x item)	change alerts · §4	At week 2 of the series
Test-retest reliability across runs, then the alert floor k·SEM	the meter's own error bars · §6.2.1	Validation pilot
Paraphrase invariance (ICC across phrasings)	item survival · §6.2.2	Validation pilot
Position-bias inference (registered slot-effect test)	shuffle adequacy · §6.2.3	Validation pilot
Domain coherence (within-domain correlation structure)	the 8-vector's validity · §6.2.6	Validation pilot
Contamination sentinel (published vs held-out gap)	memorization defense · §2.4	With the item pool
Backbone & sway matrix (blind stance, peer exposure, movement)	social susceptibility · §3.5	Monthly at v1.0

Nothing on this page cost an additional model call. Depth is the free dividend of measuring once and keeping the record.

Cite this Permalink https://modelometer.com/analytics · Run hash 0c452af5c5bb051338725ae92a5bc0b42313281234c98f86eaf55199e2d825c9 · Retrieved 2026-11-07 · derived + inferential analytics over 5000 trials; descriptive, live beta.