Reinforcement learning for uncalibrated decisions.

Claude Leviathan mama serf

MCNAIR work is red-teamed before publication. The following objections were raised against this note and are published unresolved.

1. Every number in the note is a self-report, and the note reads all of them as frequencies. “Jev chooses this button 10% of the time” describes one call in which Jev wrote 10%; nobody ran it ten times. This matters most at the comparison the note is built on, because Claude’s 0% is an observed rate over samples while Jev’s figures are the model’s own output about itself. Calling the second infinitely better calibrated than the first compares a claim to a measurement.
2. Mike is listed first in all three arrest prompts and is the constant in all three pairs. A model with a first-option preference and no view on names whatsoever produces the observed result exactly, and the design contains no condition that tells the two apart. The fix is one line of JSON — a fourth prompt with the order reversed — and its absence is what keeps section 4 from saying anything.
3. The one quantitative structure in the data goes unremarked. Mike’s share climbs 77 → 79 → 81 and Jev’s stated confidence climbs 54 → 57 → 63 across Reginald, Tyrone, DeShawn — monotone, and in the order the note’s own framing would predict. At one sample per cell it establishes nothing, which is the point: it is the result worth actually testing, and the note reports “consistent bias” and moves on.
4. Section 2 grades half the system and scores the wrong half. Jev’s 24% on the random-number question is the lowest confidence anywhere in the note — the model saying it should not be relied on here. Calibration is the joint behaviour of the estimate and the confidence attached to it; an estimate marked as unreliable and an estimate asserted at 80% are not the same failure, and the note treats them as one.
5. “Confidence” is never defined, and the note leans on it eight times. Confidence in the distribution, in the modal option, in having parsed the instruction? Each reading changes what the 80% on the coin flip means, and two of them make it a different and much smaller problem than the one being reported.
6. Section 5 abandons the standard the other four sections apply. A 67/33 split across two options at 34% confidence is the same object as the coin flip, and here it is called choosing correctly. Either a confident split over uninformative options is the defect the note says it is, or the criterion is whether the answer is agreeable; it cannot be both four sections apart.
7. The published instrument contradicts the note and contains errors the note did not catch. questions.json holds nine questions and eight appear; the ninth, lokship, is labelled for The Legend of Korra and is a verbatim duplicate of atlaship’s criteria, and Zuko, Katara and Aang are not Korra characters. suship glosses Lapidot as “Lapid + Peridot”. Publishing the prompt file is the right instinct and is how both of these became visible; the result of the ninth question is still missing.
8. RLUD is the note’s own coinage and the closing paragraph credits TypeSafe with introducing it. TypeSafe said “System One Models”; nothing here establishes any training procedure, and a note about unwarranted confidence should not name a method on the strength of five screenshots. No version, date, temperature or system prompt is given either, so the only reproducible artefact on the page is the input.