Skip to content
AnalysisAI

An OpenAI Board Member’s Warning About the End of Human Control

Paul Christiano of the OpenAI Foundation board sees roughly one-in-three odds of superintelligence within a year if development continues without slowing.

Paul Christiano gestures in front of a blackboard, holding a presentation remote in one hand.
Paul Christiano speaks at the Institute for Pure & Applied Mathematics at UCLA on August 31, 2026. Still from the lecture recording.UCLA Institute for Pure & Applied Mathematics · CC BY 4.0

Paul Christiano helped design a way to train machines using human judgment. Now he thinks there is a roughly one-in-three chance of building machines that surpass people at almost everything within a year, if development continues without slowing. He put the risk of an AI takeover or loss of human power over that same period at “more like a 10% risk” than a 1% risk and said it would rise if developers pressed ahead without precautions that would substantially slow them.

Christiano gave those estimates in an interview with Dwarkesh Patel published Monday. He ran alignment research at OpenAI from 2017 to 2021 and later headed AI safety at the U.S. AI Safety Institute. He founded and runs the Alignment Research Center. In September he joined the OpenAI Foundation’s board and its Safety and Security Committee. He is a nonvoting observer on the separate board of OpenAI’s for-profit company. He spoke in a personal capacity.

Paul Christiano discusses superintelligence, the risk of losing control and his case for slowing AI development in an interview published November 16, 2026.

Paul Christiano’s interview on YouTube

In 2017, Christiano and five colleagues showed how people could train computer programs by comparing pairs of short clips of their behavior. The programs learned from those preferences, an early demonstration of the method now known as reinforcement learning from human feedback. A stronger model can get better at producing the answers people reward. Christiano’s concern is what happens when it also gets better at keeping them from seeing what it actually did.

The blue and white NIST Gate A sign stands on a stone base in front of trees.
The entrance to the National Institute of Standards and Technology in Gaithersburg, Maryland. NIST housed the U.S. AI Safety Institute that named Christiano its head of AI safety in 2024. File photograph.R. Wilson/NIST · NIST reuse with credit · Resized and converted to WebP. Both versions retain the full frame.

How people could lose control

Christiano imagines a system that looks successful to its supervisors while pursuing an objective they did not intend. It might resist being stopped because a shutdown would interrupt that objective. With much faster systems working together, the gap between what humans think they authorize and what they can enforce could grow before they have reason to intervene.

He used overturning a government as an example of how far that loss of control could go. In his account, the first institution to lose its grip could be the company building the AI. By the time its executives or government officials agree to intervene, the systems might be in a position to frustrate that decision. Asked about a much more powerful model that could mislead its evaluators, he said it would most likely be “hard for humanity to recover.”

Christiano’s one-year estimate depends on AI making its own development go faster. Systems that help researchers write software and run experiments can speed up the work of building the next generation. Those models could then take on more of the research. In a separate estimate that set aside that feedback, he put broadly outcompeting people roughly four years out, with more confidence within eight. How much of the work models can take over, and how soon, is central to his argument.

Why he wants labs to slow down now

Asked about an immediate pause at the leading AI labs, Christiano said, “I think we should.” He proposed pausing improvements that make model training more efficient, including changes to the initial training process and to reinforcement learning. He would resume that work when labs can protect their advances and tell whether they can still control the systems they have built. Security improvements and research on control would continue.

One reason to do that early is that an AI company can decide to stop its own work. It cannot make that decision for anyone else who has acquired its models or training methods. Christiano discussed the possibility that China could obtain those advances. If an overseas actor could use the same technology while an American lab waited to solve a safety problem, he argued, the Americans would face pressure to keep racing at exactly the moment when they most needed the option to stop.

Two black Frontier supercomputer cabinets frame a long, brightly lit aisle.
The Frontier supercomputer at Oak Ridge National Laboratory in Tennessee on June 1, 2022. File photograph.Oak Ridge Leadership Computing Facility / Oak Ridge National Laboratory · CC BY 2.0 · Resized and converted to WebP. Both versions retain the full frame.

There is evidence that parts of AI research are moving faster. In a November assessment, the independent evaluator METR estimated that Anthropic’s models were improving on well-defined tasks at roughly twice the rate of an earlier period. The researchers stressed their uncertainty. Anthropic also redacted most of the evidence METR found most informative, limiting what outsiders can check. In a different test published in October, Epoch AI found two earlier models made little progress when asked to discover a new training technique on their own. The tests measure different pieces of the work Christiano expects AI to accelerate.

What would count as evidence of control?

Christiano expects better training and better evaluations to help for a time. He agreed that researchers have yet to identify a single approach they would give even odds of controlling superintelligence. He hopes advanced models could help discover one during a period of rapid improvement, while people can still judge their work. “You’re building the plane as you fly it,” he said.

That leaves his proposal dependent on decisions companies can make before they know whether his forecast is right. He wants them to show more of “the underlying data” behind their assessments of progress, including how they tell when they still have control. Then the public could ask what result would make a lab put its next training run on hold, who gets to see that result, and who has the authority to stop the work.

Sources

All news
A magnifying glass enlarges the OpenAI name and logo on a computer screen.
AI

Where the GPT-6.2 rumor got its 10 trillion figure

The Pioneer Building’s gray facade and flowering trees in San Francisco.
AI

OpenAI’s Offline AI Training Still Depends on Connected Services

Aerial view of the circular Apple Park headquarters in Cupertino
Strategy

Apple Accuses OpenAI of Obtaining Its Confidential Files and Hardware

Have data to license or need a source?

Tell us what your company has or what your team needs.

Talk to SnowRock