Notes
On papers, ideas, and things I'm thinking about.
Expenditure Horizon: Measuring Optimization Ability
We propose a measure of an AI agent’s optimization ability with an “expenditure horizon.” We give an empirical illustration from the NanoGPT speedrun.
Evidence on AI R&D Progress from NanoGPT
How much are AI agents accelerating AI R&D, and how is that changing over time? The NanoGPT speedrun leaderboard is one source of evidence: a public, cumulative record of pretraining optimizations contributed by humans and, recently, by agents.
An Apple-Picking Model of AI R&D
In this note we describe a simple apple-picking model of AI R&D, to help measure the contribution that autonomous agents are making.
On the Subtleties of Software Optimization Evals
Small metric choices in software optimization benchmarks can create brittle aggregates and misleading capability conclusions. A look at how harmonic mean aggregation and correctness penalties interact in optimization evals.
Building Blocks to AI Timelines
Notes and intuitions on economic growth models, functional forms, and how they connect to forecasting AI progress. Part of a series on economics basics for AI forecasting.