Ryan RotellaCreative technologist, Brooklyn
HeadCoach Bench, still

2026 · AI Benchmark & Live Game Harness

← All work

HeadCoach Bench

HeadCoach Bench puts frontier language models on the headset inside real football games — NFL Head Coach (2006) on PS2 and NFL Head Coach 09 on PC — and grades every call by what actually happens on the field. Alongside the live harness, I built an interactive coaching booth where anyone can watch the broadcast and inspect both models' chain-of-thought reasoning, SVG play drawings, coordinator film reads, and custom plays written directly into game RAM.

Role
Creator & Reverse Engineer
Tools
Python, PCSX2 (PINE IPC), RPCS3, Win32 RAM Hooks, OBS WebSocket, SVG
Timeline
Fall 2026
With
Solo Project
Tags
PythonAI/MLReverse EngineeringSVGNext.js
1,150
graded PS2 snaps across 6 models
42 / 42
in-game calls followed the coordinator's note
3 / 1,638
desk calls questioned swapped 4th-down math
80% → 100%
play diagram comprehension, PNG vs. SVG/code

ABOUT

Most AI benchmarks test models on static multiple-choice questions they have already memorised. I wanted to see what happens when two models have to coach a 60-minute football game against each other under clock pressure. On PS2, a Python harness talks to PCSX2 over the PINE socket protocol, reading the 28-byte live game state from RAM and pressing controller buttons to navigate the play-call menus. On PC, a two-coach harness hooks NFL Head Coach 09 via Win32 process memory, holding the game clock while both models think and verifying from the in-memory play-call struct that the engine ran the exact call each model chose.

THE DIAGRAM AHA MOMENT

Halfway through building the harness, I realised the models were coaching blind: they only saw play names like Flood Slot Post or Strong Eagle Slant 3, with no diagrams. When I rendered play diagrams as clean PNGs and tested six frontier vision models across 480 graded cards, image comprehension topped out at 80% to 99% — models miscounted receivers and misread zone drops from pixels. Meeting the models at the code layer changed everything: feeding structured vector play specs and rendering them as high-contrast SVGs (PlayArt) jumped comprehension to 99%–100%, with 0.00-yard movement error and exact route-vs-defender separation math.

THE COVER 2 LOOP & CUSTOM PLAYS

In an early Play Data Off game between DeepSeek V4.1 Flash and Kimi K3.1 Pro (hs953), DeepSeek discovered that Ride 132 Flood shredded the Colts' Cover 2 shell for 7.3 yards a pop and called it 22 times. Kimi explicitly recognised the spam in its reasoning and halftime interview — 'I reacted too slowly... I stayed too predictable with two-deep looks' — yet stayed trapped in a losing loop because the stock Colts playbook in HC09 has 24 Cover 2/Tampa 2 calls out of 25 regular defenses and zero man or single-high Cover 3/4 counters. That deadlock inspired the custom play creator: before kickoff, each model reads a scouting report on the opponent's playbook, designs 5 custom offensive and 5 custom defensive plays that are compiled into HC09's in-memory UDB0 play tables, and rewrites its own playbook slots at halftime (hs981).

LIVESTREAM & BOOTH

During live exhibition games, a portable OBS Studio instance on the Windows rig captures the 1280×800 game window, isolated WASAPI game audio, and a live bottom ticker driven over obs-websocket, streaming to this page while watch.html runs in read-only spectator mode so viewers can click through every snap as it happens.

Live site

HC09 Watch · Interactive Coaching Booth
HC09 Watch · Interactive Coaching Booth, still frame

This is the real HC09 Watch coaching booth running inside this page. Use the selector in the top bar to switch between three full games — hs955 (Play Data On + full booth with pre-snap SVG drawings, staff film reads, and interviews), hs953 (the Cover 2 loop where DeepSeek ran Ride 132 Flood 22 times), and hs981 (custom plays designed in RAM and rewritten at halftime; click 'Created plays' in the top bar).

Gameplay Clip

Gallery

6 pieces