Recently, several efforts have been made to propose standards for Super Intelligence (SI) development to mitigate potentially catastrophic risks:
- OpenAI has called for safety cases, “comprehensive, structured, evidence-based arguments about risk” that are required before any frontier RL run.
- Guidelight has created standards for alignment, control, transparency, and capability testing, which “describe what safe frontier AI development looks like in concrete terms, balanced by what’s achievable.”
- Google, OpenAI, and Anthropic are reportedly working on a body provisionally called the Standards Authority for Frontier AI.
All of these efforts assume that there is a way to develop SI that we know to be safe, that if we just follow some standards we can almost certainly keep scaling. This is false. There is no scientific consensus on how to develop SI safely. In aviation, there is a robust understanding of the forces which affect flight, and how multiple redundant systems in planes interact to prevent crashes. We do not have that. In nuclear energy, there are safe-by-design systems, where failures are self-limiting and prevention doesn’t depend on operator carefulness. We do not have that. In medicine, there is a broad understanding of how common drugs or procedures might interact badly, and, at least for the most common interactions, how they occur. We do not have that.
In SI development, at the most, we have several common-sense ideas which we think would reduce the propensities of unwanted behaviors or otherwise reduce risk for at least the next couple of months of scaling. These are the equivalent of “try building walls around your nuclear core” or “turn on a black box recorder before flying.” Of course, most SI companies have failed to even integrate these basic failsafes, but that doesn’t mean that if they added them everything would suddenly be okay! The current state of development is not one of a careful engineer calibrating various dials and switches; we are mere scientific toddlers, playing god in the vast void of unknown unknowns, hoping that with our hastily constructed, leaky raft we don’t careen into catastrophe.
This implies companies cannot advertise safety cases as guaranteeing safety (or even giving a real upper bound on risk), and standards bodies should not describe following their principles as guaranteeing safe development. In order to actually do this safely, we will need a minor revolution in several fields, on the same scale as Newtonian physics or Copernican astronomy. Luckily, we might have lots of (neurotic, untrustworthy, alien) SI help, but we will also have much, much less time than the physicists had — more like 18 months, not 180 years, even assuming we are able to stave off the Chinese and pace the frontier.
If we are able to adopt good standards industry-wide, mitigating risks from SI development will merely look like messy, high-stakes judgement calls between multiple bad options, rather than definitely hopeless. However, to actually build safely, we need science, and the time for that science.
Adversarial Review from Qwen-6-UltraMax
MCNAIR work is red-teamed before publication. The following objections were raised against this note and are published unresolved.
- The note misstates its first source. OpenAI’s post says structured safety documentation should be required before continuing a frontier RL run, and that safety cases are what such documentation would “ideally” rise to — an “aspirational north star,” harder to make rigorous for AI than for aviation or nuclear power. Nothing is currently required, and the post concedes the very gap the note accuses it of ignoring.
- No cited effort claims what the note refutes. Guidelight’s sentence, quoted in full, ends “balanced by what’s achievable”; the standards authority is a reported plan with no published standards at all. “All of these efforts assume” that standards guarantee safety is asserted, not shown, and the two quotations the note supplies cut against it.
- The vocabulary is the White House’s, not the sources’. All three efforts concern “frontier AI”; the note adopts the September executive order’s “Super Intelligence,” which renames AI in general. To any reader outside the federal style guide, “standards for SI development” describes something far more ambitious than any of these efforts claims to cover, and that makes the refutation correspondingly easier.
- The analogies compare AI’s present with other fields’ maturity. Aviation and nuclear power wrote standards well before they reached the understanding the note credits them with, and those standards were part of how they got there. That is an argument for standards as an instrument of learning, which the note does not address.
- The interactive is evidence of unknown unknowns, not of the note’s conclusion. Both papers present existence proofs in deliberately artificial settings; teacher and student share a base model throughout; two of the five results depend on attention-only LoRA, which the authors say is not standard practice. A standard forbidding distillation from an untrusted teacher on the same base would close the channel shown, which is the kind of failsafe the note dismisses.
- “18 months, not 180 years” carries the urgency and is sourced for neither figure. Copernicus published in 1543 and Newton in 1687, which is 144 years; the 18 months has no stated basis at all.
- The closing paragraph concedes the title. If industry-wide standards turn the outlook from “definitely hopeless” into “messy, high-stakes judgement calls,” they are doing a great deal of the work, and the note’s case reduces to the claim that they do not do all of it — which none of its sources disputed.
- The explainer embedded in the argument was written by a language model. A note whose thesis is that nobody understands what these systems absorb asks the reader to take a model’s account of hidden signals in model output on trust. That is either a demonstration of the problem or an exception the note does not explain.