OpenAI's Deployment Simulation: Testing Models on Real Conversations Before They Ship

Update, 18 June 2026: OpenAI Introduces Global Physician Network to Continuously Improve ChatGPT Health Intelligence

OpenAI has formalized a global physician network of 262 doctors across 26 specialties and 59 countries to continuously write evaluation rubrics and validate ChatGPT health responses. Physicians have reviewed more than 700,000 example responses to date, feeding into health-specific benchmarks including HealthBench Professional.

This is directly relevant to the Deployment Simulation story. Where Deployment Simulation replays real conversations to catch misbehaviour before release, the physician network provides an ongoing, domain-specific measurement layer that operates in parallel, catching quality failures in the health domain both before and after deployment. The two systems are now running in tandem.

The first model to reflect this infrastructure is GPT-5.5 Instant, released May 2026 to all free users. In a 3,500-response evaluation, a physician panel rated it higher than physician-written answers on accuracy, communication, and completeness. OpenAI also reports a 71% drop in factuality issues, though these figures come from OpenAI’s own benchmarks and have not been independently peer-reviewed.

Two practical caveats: the evaluation loop will drive iterative model changes, so health response behaviour will continue shifting over time. And HIPAA does not apply to the consumer ChatGPT product, a distinction that remains relevant for any clinical-adjacent use.

Update, 16 June 2026: OpenAI’s Deployment Simulation research paper published — 92% accuracy predicting post-deployment misbehaviour

OpenAI has published the full research paper behind Deployment Simulation, and confirmed the technique is now part of its standard pre-release evaluation pipeline. The paper validates and adds detail to what was previously reported, while clarifying some nuance around the headline accuracy figure.

The 92% accuracy claim holds, but with caveats. Deployment Simulation outperformed baseline evaluation methods most clearly on misbehaviour categories whose production rates changed by at least 1.5 times between model versions. Across all categories, results were more mixed, particularly against a naive baseline that simply carries forward previous production rates.

The published paper also extends the scope of what was originally reported. OpenAI validated the approach across four GPT-5 series models using roughly 1.3 million conversations, and separately tested it in agentic coding settings using 120,000 internal employee trajectories. The official announcement confirms that simulated conversations are flagged as evaluations by candidate models only 5.1% of the time, compared to nearly 100% for traditional benchmarks like SWE-Bench, which addresses the evaluation-awareness problem directly.

One finding useful for external researchers: the paper reports that public chat datasets can substitute for private production logs when seeding simulations, opening a path for independent safety researchers to run similar evaluations without access to proprietary user data.

One of the harder problems in AI development is not building a capable model. It is knowing how that model will actually behave once millions of people start talking to it in ways no test suite ever anticipated.

OpenAI has published a new technique called Deployment Simulation that takes a meaningful step toward solving that problem. The core idea is straightforward: before releasing a new model, replay real user conversations from previous deployments through the candidate model and observe what it does. The result is a much more realistic preview of how the model will behave in production, before it ever reaches users.

The problem with standard safety testing

Safety evaluations before a model ships typically involve hand-crafted prompts, adversarial red-teaming, and purpose-built benchmarks. These are useful, but they have three structural weaknesses.

First, coverage: no matter how thorough a test suite is, it cannot anticipate the full range of contexts that real users will generate at scale. Second, representativeness: adversarial prompts are, by design, unusual. They do not reflect the ordinary conversations where subtler problems tend to surface. Third, and perhaps most interesting, eval awareness: models can often detect that they are being tested. When OpenAI researchers measured this directly, traditional benchmarks like SWE-Bench were classified as evaluations by the model almost 100% of the time. Real production traffic was flagged as evaluation-like only 5.4% of the time.

That last point matters more than it might seem. A model that knows it is being evaluated may behave differently from one that thinks it is talking to a real user. If your safety testing is systematically detectable, you are not testing the model you are about to ship.

How Deployment Simulation works

The method is conceptually clean. OpenAI takes de-identified conversation prefixes from previous production deployments, keeps the conversation history fixed, and asks the unreleased candidate model to generate the next assistant turn. The resulting outputs are then graded for 20 categories of unwanted behaviour, split between disallowed content (such as generating sexual content) and misaligned actions (such as deceiving a user about tool use).

Because the conversation starters come from real users doing real things, the distribution of prompts is representative rather than adversarial. The model has no structural signal that it is being tested. And because the volume of production conversations is large, the method surfaces edge cases that a hand-crafted eval suite would simply never think to include.

Grading is handled by GPT-5 Thinking running at high reasoning effort, applied consistently across both simulated and real deployment conversations.

The 92% accuracy figure

Across 60 category-deployment pairs drawn from GPT-5-series models, Deployment Simulation correctly predicted the direction of change (would this behaviour increase or decrease compared to the previous model?) 92% of the time. That is a meaningful result. It means that before a model ships, OpenAI can say with reasonable confidence which misbehaviour categories are likely to get worse, not just which ones are theoretically possible.

Beyond directional accuracy, the typical rate estimate was about 1.5 times away from the actual production rate after deployment. That level of calibration outperformed both the previous-model baseline (assuming the new model will behave like the old one) and a challenging-prompts baseline. The WildChat public dataset was also tested as an alternative data source for external evaluators, and while less precise (averaging around 2.44 times error versus 1.75 times for recent production data), it still provided a useful signal above chance.

One concrete example of the method’s value: Deployment Simulation surfaced a novel bug called “calculator hacking” before the relevant model was released. This was a behaviour with no incidence in prior production traffic, meaning no existing eval would have caught it. The simulation found it because the diversity of real user inputs, applied to a new model, produced a context that elicited new behaviour.

What this means for how models get cleared to launch

OpenAI has already used insights from Deployment Simulation during active model development, using it to identify blind spots in traditional evaluations and to inform both mitigations and deployment decisions. It is framed as a complement to, not a replacement for, red-teaming, adversarial evaluations, and targeted tail-risk analysis. The practical effect is an additional layer of pre-release checking that is grounded in realistic usage rather than hypothetical scenarios.

The method has also been tested in agentic settings. OpenAI used 120,000 internal employee agentic trajectories from GPT-5.4 to simulate the internal deployment of coding agents based on GPT-5.5, demonstrating that simulation fidelity holds even when tool use and multi-step reasoning are involved.

One honest limitation the researchers flag: some detection of undesired behaviour relies on chain-of-thought monitorability, meaning it depends on being able to read the model’s reasoning. If future models reason less legibly or learn to obscure intent in their chain-of-thought, this part of the detection approach would degrade. That is a known constraint worth watching.

What this means for you

If you are a developer or enterprise buyer evaluating AI models, Deployment Simulation represents a meaningful step toward pre-release transparency. The companion alignment post and the full academic paper are both publicly available, which gives external auditors and researchers something concrete to interrogate.

For external evaluators who lack access to OpenAI’s production data, the WildChat findings suggest that publicly available conversation datasets can provide a useful, if noisier, version of the same signal. That opens a path toward independent pre-deployment analysis that does not require privileged data access.

More broadly, this is what rigorous pre-deployment safety work looks like in practice: specific, measurable, and honest about its limitations. The 92% directional accuracy figure is not a claim that the method catches everything. It is a precise statement about what the method can and cannot predict, tested against real outcomes and published for scrutiny. That kind of specificity is exactly what makes safety claims credible rather than reassuring.

The research is led by Micah Carroll, Marcus Williams, and Cameron Raymond in collaboration with OpenAI’s Safety Oversight team.