Yao Shunyu: Let Me Go a Little Crazy! Training Models at Anthropic & Gemini, Heroism Is Over

Yao Shunyu describes frontier-model development as having moved from a phase where labs worried about whether they could reach top capability, to a phase where top labs are much closer on benchmarked performance and the harder problem is deciding what to build, how to define desired behavior, and which capabilities to prioritize. He argues that differences between Anthropic, OpenAI, and Gemini still matter in real use, but are less visible in public benchmarks. He also frames success in the current AI industry less as individual brilliance and more as disciplined execution, infrastructure, data work, judgment about product direction, and reliable teams. The transcript also contains his strong preference for precision over vague theorizing, his skepticism toward deference to seniority, and his view that earlier AI progress had clearer heroic figures than the current phase.

Key insights

1. Frontier model competition has shifted from capability catch-up to problem definition: Yao says that roughly a year earlier, labs were worried about whether they could catch up in raw model capability, especially on reasoning. Now, among Gemini, OpenAI, and Anthropic, he does not think any of them is primarily worried about failing to catch up. The harder question is what to do with the models and how to define the actual problem well.

   Why it matters: If true, advantage is moving away from obvious raw-model gaps and toward clearer product bets, better task definitions, and better translation of capability into user value.

2. Public benchmark gaps are compressing and may contain more noise than signal: He says that on benchmarks such as SWE-bench and math evaluations, leading models now look very close, with small differences that may be only one or two percentage points and often mostly noise rather than meaningful signal. He contrasts this with an earlier period when benchmark numbers made capability differences easier to see.

   Why it matters: Decision-makers should be cautious about over-interpreting leaderboard deltas. Small public benchmark wins may no longer reliably predict user-perceived superiority or durable strategic advantage.

3. Real-world model differences still exist, but are increasingly product- and use-case-specific: Despite compressed benchmark scores, Yao says users can still feel meaningful differences. In his view, Claude remains especially strong as a general-purpose tool-using agent; Codex has recently narrowed the gap in pure coding; Gemini may be stronger in pure reasoning and some everyday scenarios, while still catching up in coding and agents.

   Why it matters: Model choice should be tied to workflow and task profile, not just benchmark branding. The competitive landscape may fragment by modality, agent behavior, and product integration rather than one universal leader.

4. Prioritization and infrastructure investment shaped earlier model differences: He says earlier differences were driven heavily by what each lab chose to prioritize. For example, valuing tool use or reasoning led companies to spend more effort on infrastructure and especially data. He emphasizes that building the right data takes significant time and effort.

   Why it matters: Capabilities are not only a function of abstract research quality; they are path-dependent outcomes of sustained organizational focus, data pipelines, and support systems.

5. As models become closer, unclear task definitions increasingly determine outcomes: Yao argues that when models are numerically similar, the harder issue is defining the behavior one wants. He says some important model differences may come from factors teams would not have anticipated in advance, and only understand in retrospect.

   Why it matters: The bottleneck may be shifting from training a stronger base model to specifying, evaluating, and shaping behavior for messy real tasks. This favors organizations with better iteration loops and sharper definitions of success.

6. Some strong model behavior may come from accidental data advantages, not fully understood design insight: As an example of unintuitive causes, he says that years ago models may have learned code unusually well partly because code found on the open web, such as GitHub, was naturally higher-quality than many other web sources, even before people fully understood why coding performance emerged strongly.

   Why it matters: Observed capability can arise from data composition effects that teams do not initially understand. This implies that data provenance and quality may matter as much as headline architecture in surprising ways.

Strategic implications

• If frontier labs are near parity on many public metrics, downstream differentiation is likely to come from product packaging, task framing, agent reliability, tool integration, and post-training behavior shaping rather than raw benchmark dominance alone.
• Organizations evaluating model vendors should run workflow-specific tests instead of relying heavily on public benchmark tables, especially where score gaps are small.
• Data and infrastructure remain strategic moats. Even if architectures and headline methods diffuse, sustained prioritization can still create meaningful capability asymmetries over time.
• The industrialization of AI research may increase the value of operational excellence, research rigor, and dependable execution relative to charismatic vision alone.

Signals to watch

• Whether public benchmark leaders continue rotating while user preference remains stable, which would support the claim that benchmarks are nearing saturation as decision tools.
• How much model vendors differentiate by agent/tool use, coding workflow quality, and everyday product experience rather than by single-number benchmark wins.
• Whether large labs increasingly launch polished versions of capabilities first surfaced in rougher external or open-source demos.
• Investment patterns around data curation, infrastructure for agent behavior, and long-horizon task evaluation.

Caveats

• The transcript appears incomplete and includes AI-generated subtitles with probable errors, omissions, and mistranscriptions.
• Some terms and product names are unclear or potentially mistranscribed, including references such as 'OpenClaw' and parts of the discussion around Claude/Opus/Codex.
• Several strong claims are personal judgments by the speaker rather than independently verified facts, especially comparative assessments of labs and models.