Executive Briefing: Shifting From Pre-Training to Deliberative Inference
For seven years, AI capability gains were governed by Kaplan and Chinchilla pre-training scaling laws: spending tens of millions of dollars feeding petabytes of tokens into larger parameter networks. However, with web text corpora largely exhausted, pre-training yield curves began showing diminishing returns. The test-time compute revolution—spearheaded by OpenAI o1/o3 and Claude 3.7 Sonnet Extended Thinking—introduced an entirely new scaling dimension: allocating computational resources during inference time, enabling models to explore, backtrack, and verify chain-of-thought hypothesis trees before generating a final answer.
1. System 1 vs. System 2 Machine Cognition
Daniel Kahneman’s dual-process cognitive framework provides the conceptual foundation for test-time scaling:
2. Process Reward Models (PRMs) & Outcome Reinforcement Learning
Standard Reinforcement Learning from Human Feedback (RLHF) rewards only the final output (Outcome Reward Model). If an answer is correct by pure luck despite flawed reasoning, the model reinforces invalid logic. Process Reward Models (PRMs) evaluate and score every individual intermediate thought step. If step 4 of an equation contains an algebraic hallucination, the search tree immediately prunes that branch, backtracking to explore viable alternative hypotheses.
3. References
- Lightman, H., et al. (2023). Let’s Verify Step by Step: Process Supervision in Mathematical Reasoning. OpenAI Research.
- Snell, C., et al. (2024). Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Model Parameters. UC Berkeley & Google DeepMind.