Council Post: The Toddler Test: What A Child Knows That A Trillion Tokens Don't
Charles Yeomans is the chairman and founder of Atombeam.
getty
Print the words used to train a modern large language model (LLM) and, as MIT Technology Review (subscription required) recently observed, the stack of paper reaches past the International Space Station. Print out every word a child hears on the way to mastering language and the stack stands about as tall as a house.
That is the focus of this column, and I would argue it is the most important problem in artificial intelligence (AI) today: a toddler reaches grammatical fluency on 10-30 million words, while our best models consume trillions of tokens, a difference of five orders of magnitude, to reach fluency that still is not understanding.
Understanding The Gap
Scientists call this the data efficiency gap, and it’s being studied seriously. An annual competition called BabyLM challenges researchers to train models on a child’s diet of words. The results are humbling. Stanford’s Michael Frank put it memorably: train an LM on what a toddler hears and “you get a nonsense generator; you don’t get a kid.”
The obvious fixes have been tried. Feeding models data in order, simple before complex, did not close the gap. Nor did video from cameras worn by babies or letting models learn socially from other models. The gap is not a tuning problem. It’s architectural.
Why should technologists care? Two reasons, one financial and one strategic. The former: the trillion-token diet is why AI costs so much, the data-center buildout measured in the trillions of dollars, the gigawatts of power and the inference bills in the billions-per-year for a single company are the price of achieving fluency with brute statistical force.
The strategic one: the internet is finite. Researchers project that the well of high-quality training data could run dry as early as the 2030s. A paradigm that requires 100,000 times more language than a human, in a world running out of language to feed it, has a problem capital alone cannot solve. Meanwhile, every child born demonstrates that the problem is not necessary.
The Historical Context
History explains why the gap exists. In the 1950s, Noam Chomsky argued that language cannot be learned from statistics alone; children’s exposure is too thin for the richness of what they learn, so structure must come first. His side won the linguistics debate and inspired decades of AI built on hand-coded rules, which failed.
The statistical side then won the engineering war, but the five-orders-of-magnitude bill served as a reminder. Statistics at scale brought fluency while conceding the problem Chomsky raised: the models needed a hundred thousand childhoods’ worth of language precisely because they bring no structure to the task. The two paradigms we tried are rules written by hand and statistics without structure. But a third possibility is structure that is learned.
Differing Architecture
Look at what psychologists say children have that models lack. The list is architectural. Children know what they do not know and are driven to fill the gap. Children choose their own data, probing the world with little experiments rather than passively absorbing it and learn continuously. Harvard’s Elizabeth Bonawitz notes that in contrast, models pretrain once and never again, unlike minds that keep becoming.
The toddler finding marks the third time in 18 months that a different research community reached the same conclusion. Harvard and MIT researchers trained a transformer on orbital data and found it predicted trajectories superbly but had no understanding of why or Newtonian mechanics. A Princeton-led team gave research agents everything they needed to create a research paper, but they were unable to complete the task. And now we know models cannot learn from a little, while every child learns from almost nothing.
Orbits, laboratories, nurseries: three fields, one finding. What is missing is not scale, but structure—held explicitly, revisable and doing the work statistics has been asked to approximate.
What could the missing structure be? My own bet is geometry. The mathematics of learning says how much data you need depends on how many possibilities you must consider. A learner, like an LLM, that considers every possible statistical pattern needs oceans of examples.
In contrast, a learner that assumes the world has shape, persistent objects, agents with goals, causes that precede effects and space with paths through it only needs enough data to locate things within that structure. Developmental psychology has shown infants possess exactly this core knowledge months before language begins. The child is not learning a world from words but instead attaching words to one already modeled.
LLMs run the process backward, reconstructing a world from trillions of textual shadows, but never quite getting there. Physics is the proof that structure works: relativity makes predictions from a handful of observations because geometry constrains what is possible before any data emerges.
Bring structure, and data only has to tell you where you are, not what the universe is. Yes, this is a hypothesis of the developing mind. But it accounts for the failure of statistical scale conducted all at once.
Closing Thoughts
Berkeley’s great developmental psychologist Alison Gopnik offered a prediction in the MIT Technology Review piece I suspect is prophetic: current frontier labs are not racing to learn from children, and it will be the next generation of AI, whatever replaces the transformer, that takes its lessons from developmental psychology.
I agree. The systems that pass the toddler test learn much from little, know the edges of their knowledge and grow every day they run will not get there by scaling the current recipe. They will get there the way children do, and the way physics does: by holding a model of the world and letting the data be evidence about it rather than the thing itself.
A child hears a thin trickle of language and reconstructs the entire river from it. The companies that figure out how will not need a stack of paper to the space station or massive power plants and data centers. They will need what every nursery already contains: a learner intent to seek structure, notice what it lacks and never stop becoming. That is the benchmark that matters now.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?