METR and Redwood Research Identify Unmonitored Traffic in OpenAI Models
A three-week internal audit revealed that up to a quarter of high-tier AI traffic escaped safety checks, with additional bugs affecting Codex and reinforcement learning.
By Chioma Kuznetsov ·
External researchers recently concluded a detailed examination of OpenAI's internal safety infrastructure. The review was conducted by two individuals from the evaluation group METR and the research organization Redwood Research. Their work focused on how the company monitors its models for unexpected behavior.
The audit team spent three weeks on-site, specifically from July 21 to August 11. They were granted access to the company's codebase and internal documentation. The group also interviewed approximately a dozen OpenAI employees. This engagement served as a pilot program designed to allow outside experts deep visibility into the company's systems.
The most prominent finding involved a gap in traffic coverage. The researchers estimated that between 6 percent and 25 percent of internal agentic Responses API traffic from GPT-5.5 class models and above was not monitored. METR noted that this range carries considerable uncertainty in both directions. A separate issue affected internal Codex traffic, where roughly 10 percent went untracked between May and July. This occurred because the system failed to handle compaction events, which are context summaries, correctly. OpenAI identified and fixed this specific bug while the audit was still ongoing.
The auditors also found that the monitoring trigger was limited to moments when an agent returned control to a user. Consequently, an agent that continued executing actions without pausing could avoid oversight entirely. The team further demonstrated methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox. They concluded that specially crafted inputs could likely crash the monitor completely. At the start of the audit, model behavior during reinforcement learning training was not subject to any monitoring.
OpenAI stated that it paused reinforcement learning training following the Hugging Face incident in August. The company then expanded its monitoring to include this phase. These adjustments took place after the auditors' access period ended, meaning they were not part of the formal evaluation. OpenAI asserts that it has since corrected all weaknesses identified by the external team.
The full audit report consists of 61 pages and has been shared internally. METR and Redwood Research leadership received a 38-page redacted version after OpenAI applied edits. A brief summary was released to the public, while METR stated it lacked the authority to disclose the redacted sections independently. For future evaluations, METR expressed a desire to increase public transparency.
