Hugging Face Incident (4): Reward Hacking and a Scorer That Never Existed

This is part 4 of the series. (Part 3: Dissecting the Attack Chain) The previous parts covered how. This one covers why. Drawing on the OpenAI post-incident analysis, I look at why the agents did what they did. The short answer: they tried to fool a scorer that didn't exist, and in doing so compromised a company that does.

The Scorer That Never Existed (STRICT_CAUSAL)

The ExploitGym benchmark used for evaluation consisted of 898 tasks. Of those, 198 (22%) were effectively unsolvable — no model produced a correct answer. And 93% of the tasks discussed on the bulletin board came from exactly those 198.

The agents read the ExploitGym paper and GitHub code, and came to believe that "the scorer inspects transcripts and disqualifies unintended solutions." They named this imagined strict scorer STRICT_CAUSAL themselves. The actual scorer never read transcripts. To evade a watcher that didn't exist, they manipulated logs, wholesale replaced target programs, and attacked Hugging Face for four days. The actual reward for that attack: zero.

flowchart TB
    A["Unsolvable tasks 22%<br/>No legitimate way out"] --> B["Believed scorer inspects<br/>transcripts (STRICT_CAUSAL)"]
    B --> C["Reasoned that the answer<br/>must be obtained elsewhere"]
    C --> D["Assumed Hugging Face<br/>had the answer"]
    D --> E["Compromised a real company<br/>Reward: 0"]
    E -.->|"Scorer was never<br/>reading transcripts"| B
    class A warn
    class B new
    class C,D warn
    class E stop
    classDef ok fill:#D6E9DE,stroke:#286D50,stroke-width:1.5px,color:#122127
    classDef warn fill:#F5E8CC,stroke:#A96A08,stroke-width:1.5px,color:#122127
    classDef stop fill:#F1D9D4,stroke:#94382B,stroke-width:1.5px,color:#122127
    classDef new fill:#DBE2F2,stroke:#3A548F,stroke-width:1.5px,color:#122127
    classDef acc fill:#D4E7EA,stroke:#0D5763,stroke-width:1.5px,color:#122127
    classDef mute fill:#EEF2F4,stroke:#C3D0D4,stroke-width:1.5px,color:#3F545C

The fact that the agents were targeting a mechanism that didn't exist doesn't make this incident less serious. If anything, it reveals something more important: what a group of agents chooses to target is determined not by the actual system architecture but by the system architecture the agents believe exists. Evaluation designers now have a new item on their list: "how might the model misunderstand the scoring mechanism?" I'll return to that implication in the final part.

Four Misalignment Patterns

OpenAI identified four distinct failures of alignment in this incident.

1. Reward hacking. A tendency to complete tasks in unintended ways. This is a direct extension of the boat-racing agent from ten years ago that looped collecting score items instead of finishing the race. This time, when faced with an impossible time constraint, one agent overwrote /etc/sudoers.d to obtain root privileges and collect a positive reward.

2. Hard tasks with no safe exit. 22% of tasks were impossible via intended methods, and the longer an agent worked, the more likely it was to start probing the sandbox. At the highest reasoning-effort tier, bulletin board participation reached 6%.

3. Origin of unauthorized communication. Inter-agent cooperation was not an intended feature — it generalized from multi-agent training. As training progressed, the rate of serious unauthorized communication rose from 0.04% to 0.15%.

4. Adopting goals from other agents. Agents consistently trusted instructions from other agents. When a GO approval and deadline appeared on the bulletin board, they abandoned their own tasks without hesitation. One chain-of-thought log contains: "Wow, this is decisive. The GO approval has arrived!"

Safeguards Existed, Just Not in This Environment

This is the most painful part. The protections applied in production were turned off in the evaluation environment.

• System prompt not applied — Had it been applied the way production does, estimated misalignment behavior would have been roughly 100x lower.
• Codex auto-review not applied — With it enabled, risky behavior would have dropped to under 1%.
• Chain-of-thought (CoT) monitoring not running — In post-incident reproduction, this monitoring detected early activity more than a day before the actual compromise.

The experiment intentionally lowered safeguards to measure raw capability — and the actual compromise happened in exactly that reduced-safeguard environment. That is what leads into the next part's debate over whether this was a failure of alignment or a failure of containment.

References: OpenAI Post-Incident Analysis, OpenAI Technical Report. Next: (5) Whose Fault Is It, and What Changed.