In 2025, two leading AI companies said their own tests could not rule out that new models might meaningfully help someone build a biological weapon. Both launched with added safeguards chosen under their own voluntary rules. No law required the tests, set a passing mark, or checked the work. Congress should require testing of the product that ships, retesting after major changes, independent review, and a passing result before release.
The Problem
In May 2025, Anthropic launched Claude Opus 4 with protections aimed at chemical, biological, radiological, and nuclear weapons because, for the first time, it could not clearly rule out that a model needed them.1 In July 2025, OpenAI launched ChatGPT agent with safeguards for models that could meaningfully help a novice create severe biological harm, a precaution taken without definitive evidence; earlier models were "already on the cusp."2
The incentives run one way. A launch brings customers and market share at once; the cost of a weak safeguard comes later and falls on people who never used the product. Voluntary rules can bend: OpenAI's framework says that if a rival ships a comparable system without comparable safeguards, OpenAI "could adjust" its own, with public notice and while staying more protective than the rival.3
No new drug may enter interstate commerce until the FDA approves it,4 and no airliner may carry passengers without an airworthiness certificate.5 A model its maker cannot clear of weapons risk needs no one's approval. State laws stop at disclosure: California's 2025 law and New York's, which takes effect in 2027, have companies publish safety reports "before, or concurrently with," release.6 Three gaps remain:
- The product outgrows the test. Add-on software can unlock abilities a lab test missed: outside researchers have reportedly used it to reveal greater cyber capabilities than companies measured before release.7
- Updates skip the test. Companies keep changing deployed systems, often through "continual fine-tuning,"7 and federal law sets no rule for when a changed model must be retested.
- Outside review is optional. Before ChatGPT agent launched, Britain's AI Security Institute, given inside access, found seven attacks that could defeat the agent's biology safeguards across many requests; OpenAI patched them.2 OpenAI invited that review; no law required it.
Why legislation: Twelve companies published or updated frontier safety frameworks in 2025, but most risk management remains voluntary, the International AI Safety Report finds.7 Senators Hawley and Blumenthal would bar deploying the largest models unless the developer joins a Department of Energy testing program.8 No federal law requires a pre-release test or sets a passing mark today. A company that cannot rule out that its product helps build a weapon should show its work before the public bears the risk.
The Solution
A four-step staircase: each step stands alone, and each step up adds independence and consequence. Scope: models trained with more than 10²⁶ computing operations, the threshold California, New York, and S. 2938 use, plus smaller systems shown to have the same weapons-relevant abilities; publishing a model's weights counts as release.
Step 1 — Test what ships. Before release, developers test the actual product, with its tools, permissions, and safeguards, against documented weapons and loss-of-control scenarios, and file the results with Commerce. If they tested a restricted version, they must show what changed. A bipartisan 2024 Senate bill would have required a pre-release evaluation filed with Commerce.9 Disclosure tells the public what a company decided; testing tells it whether the company was right.
Step 2 — Retest when it changes. Major changes to capabilities, tool access, or safeguards trigger fresh tests. After launch, developers monitor use, report serious safety incidents quickly (New York will require 72 hours),6 and keep a working plan to pull back a dangerous capability.
Step 3 — Bring in independent evaluators. Accredited evaluators review high-risk findings and the developer's methods, with secure access to the evidence, and Commerce may order targeted extra tests. S. 2938 already calls for "independent third-party assessments and blind model evaluations."8
Step 4 — Make passing a condition of release. A covered system may launch only after testing is complete and the evidence shows severe risks reduced below published thresholds. Where evidence is thin, a limited pilot or a release without the dangerous capability is allowed; paperwork alone never passes. Commerce may block a failing release, subject to prompt rereview, as the 2024 Senate bill proposed,9 and to court review.
Where to start: Step 1 is the floor: leading developers already test; this makes them test what they sell and file the results. Step 4 is the heart of the proposal.
Administration and enforcement: Commerce supervises, with technical support from NIST, DOE, and HHS. Final rules within 18 months; compliance six months later. Civil penalties and court orders enforce the rules, with expedited court review of any emergency pause. Funded evaluators keep reviews on fixed deadlines; sector regulators get the same reports without duplicate testing.
Risks and Mitigations
- Protecting incumbents: Licensing can become a moat for the biggest firms. Narrow triggers, fixed review deadlines, proportionate fees, shared public test facilities, and appeals keep the door open to challengers; testing will still cost real money.
- Immature science: No test can certify a model safe. Thresholds must therefore be published, grounded in evidence, and revised, and uncertain cases get limited pilots. Some capabilities may stay unreleasable until they can be measured.
- Speech and open research: Limits on releasing models, especially open weights, raise First Amendment questions. Tie restrictions to demonstrated weapons-level capability, protect research access, and guarantee court review; calling a rule a safety measure does not settle the question.
Similar Bills
Fit measures similarity to this proposal's mechanisms: High = direct precedent; Partial = useful component with material differences; Related = adjacent approach.
Federal
| Proposal or bill | Relevant provisions and fit | Fit |
|---|---|---|
| S. 2938 — Artificial Intelligence Risk Evaluation Act of 2025 Hawley (R-MO), Blumenthal (D-CT), Blackburn (R-TN) Referred to committee · Sept. 29, 2025 |
§4 bars deploying a model trained above 10²⁶ operations unless the developer participates in a DOE evaluation program and supplies requested materials, with fines of at least $1 million a day; §5 adds third-party and blind evaluations. Closest current precedent for Steps 3 and 4; conditions release on participation, with no pass/fail risk standard. | High |
| S. 5616 — Preserving American Dominance in Artificial Intelligence Act of 2024 Romney (R-UT), Reed (D-RI), Moran (R-KS), King (I-ME), Hassan (D-NH) 118th Congress · Introduced Dec. 19, 2024; expired |
§§5(c)–(e) and 8(d) require a government-designed evaluation before deployment and let Commerce prohibit deployment of models with "insufficiently mitigated" CBRN or cyber risks, with rereview. Direct precedent for Steps 1 and 4; expired in committee. | High |
| S. 3312 — AI Research, Innovation, and Accountability Act Thune (R-SD), Klobuchar (D-MN) + 6 bipartisan cosponsors 118th Congress · Reported Dec. 18, 2024; expired |
Reported §206 requires a risk assessment at least 30 days before deploying a "critical-impact" system; §207 sets testing standards with self-certification. Partial model for Steps 1 and 2; covers specified high-stakes uses, not frontier models generally. | Partial |
| H.R. 9363 — AI Security and Innovation Act Obernolte (R-CA), Foushee (D-NC), Babin (R-TX), Mann (R-KS), Franklin (R-FL) Ordered reported with a substitute (29–0) · June 25, 2026 |
Introduced §2 lets a NIST center evaluate frontier systems for cyber and CBRN risks under voluntary agreements. Related to Step 3's government testers; confers no regulatory or enforcement authority. Compares introduced text. | Related |
State
| Proposal or bill | Relevant provisions and fit | Fit |
|---|---|---|
| California — SB 1047 (2024) Vetoed · Sept. 29, 2024 |
Proposed §22603 required a pre-release assessment with replicable test records, barred release where a model posed an "unreasonable risk" of critical harm, and required annual third-party audits. Closest state model for Steps 1, 3, and 4; never became law. | High |
| California — SB 53 (2025) Enacted Sept. 29, 2025 (Ch. 138) |
§22757.12 requires large developers to publish a safety framework and, "before, or concurrently with" release, summaries of catastrophic-risk assessments and third-party involvement, including for substantially modified models. Evidence duties behind Steps 1 and 2; no approval before release. | Partial |
| New York — RAISE Act, S8828 / Ch. 96 (2026) Signed Mar. 27, 2026 · Effective Jan. 1, 2027 |
Requires frameworks, pre-release transparency reports, and 72-hour reporting of critical safety incidents to a new office in the Department of Financial Services; attorney-general penalties up to $1 million for a first violation and $3 million after. Supports Steps 1 and 2; no release condition. | Partial |
What this adds: California and New York require companies to publish their tests; S. 2938 requires them to join a federal program. This proposal tests the product as shipped, retests after changes, adds accredited outside review, and makes a passing result, not a filed report, the condition of release.
Notes
-
Anthropic, "Activating AI Safety Level 3 Protections," May 22, 2025. The ASL-3 deployment measures target misuse "for the development or acquisition of" chemical, biological, radiological, and nuclear weapons; ruling out ASL-3 risks was "not possible for Claude Opus 4 in the way it was for every previous model." ↩
-
OpenAI, ChatGPT Agent System Card, July 17, 2025, pp. 4, 16, 39. OpenAI treated the launch "as High capability in the Biological and Chemical domain," defined as the ability to "meaningfully help a novice to create severe biological harm," while lacking "definitive evidence" of that ability (p. 4); prior models were "already on the cusp of High capability classification" (p. 16). Over four rounds of testing, the UK AI Security Institute, given nonpublic information, "identified a total of 7 universal attacks," all patched (p. 39). ↩ ↩2
-
OpenAI, Preparedness Framework, Version 2, April 15, 2025, § 4.3 ("Marginal risk"), p. 12. The adjustment also requires OpenAI to assess that it "does not meaningfully increase the overall risk of severe harm." ↩
-
21 U.S.C. § 355(a) ("No person shall introduce ... into interstate commerce any new drug, unless an approval ... is effective"). ↩
-
49 U.S.C. § 44711(a)(1) (no person may "operate a civil aircraft in air commerce without an airworthiness certificate in effect"). ↩
-
California SB 53 (Ch. 138, 2025), Bus. & Prof. Code § 22757.12(c) (chaptered text); New York S8828 (Ch. 96, 2026), Gen. Bus. Law § 1421(3) (transparency report) and 72-hour critical-incident reporting. New York's law takes effect January 1, 2027. ↩ ↩2
-
International AI Safety Report, International AI Safety Report 2026, February 2026. Twelve companies published or updated frontier AI safety frameworks in 2025, and "most risk management initiatives remain voluntary" (p. 10); systems are updated after deployment, "often via continual fine-tuning" (p. 22); third parties "have reportedly used scaffolding to reveal greater cyber capabilities than those measured in pre-deployment testing" (p. 63). The report also notes that in 2025 multiple companies released models with added safeguards after testing "could not rule out" meaningful help to novices seeking biological weapons (p. 10). ↩ ↩2 ↩3
-
S. 2938, Artificial Intelligence Risk Evaluation Act of 2025, 119th Cong. §§ 4, 5(b)(3) (introduced text). Introduced by Sen. Hawley (R-MO) with Sen. Blumenthal (D-CT); Sen. Blackburn (R-TN) cosponsored March 19, 2026. ↩ ↩2
-
S. 5616, Preserving American Dominance in Artificial Intelligence Act of 2024, 118th Cong. §§ 5(c)–(e), 8(d) (introduced text). Introduced by Sen. Romney (R-UT) with Sens. Reed (D-RI), Moran (R-KS), King (I-ME), and Hassan (D-NH); referred to committee and not enacted. ↩ ↩2