A great article! A couple of observations and a hypothesis - would love your thoughts:
1. The number of generations from earliest life to modern day humans seems to be in the order of 10^11-10^13 (microbial eras dominate). The unit of information passing is the number of genetic mutations (<< 1 for microbe reproductions to few dozen for humans). And much to my surprise, there are few mutations in any single lineage (~a few billion).
2. Trillions of parallel experiments over billions of years in a planet-scale search found the path to intelligence, but the actual direct path that got to us required relatively fewer operations ( a few billion ~= generations * average mutations/generation). So, after all the effort we put into our own search towards AGI, the final solution might be relatively simpler than what the evolutionary compute suggests.
3. CNNs that relied on priors like locality and translation equivariance lit the fuse for computer vision, and transformers with attention mechanisms started the GenAI revolution. We need a scalable architecture with the right inductive priors to achieve human-level AGI. Some believe transformers/LLMs will get us there. Others, like Yann LeCun with I-JEPA, are proposing alternatives.
(2) this is an interesting one. I think it implies that the search is MUCH bigger than the find.... sortof like... if you're looking for gold... digging 6 feet into the earth takes like 20 minutes.... but searching the whole earth for *where* to dig can take many lifetimes.
This is a really nice analogy which emphasizes why we should carry forward all the knowledge humasn have acquired instead of re-running the search... we already have some pretty great places to dig!
Ultimately, the challenge is to know how to use human knowledge to guide the search; to figure out the architectures with the right priors, objective functions, and correct ways of using human knowledge.
After all, evolution's objective function is reproduction, and intelligence is a byproduct. Likewise, who could've predicted that next token prediction would result in all these emergent capabilities!
Extremely well written article Andrew!
so many insights.
"Each generation relearns the same lethal lessons… because they don’t have information technology."
Very interesting and helpful perspective
A great article! A couple of observations and a hypothesis - would love your thoughts:
1. The number of generations from earliest life to modern day humans seems to be in the order of 10^11-10^13 (microbial eras dominate). The unit of information passing is the number of genetic mutations (<< 1 for microbe reproductions to few dozen for humans). And much to my surprise, there are few mutations in any single lineage (~a few billion).
2. Trillions of parallel experiments over billions of years in a planet-scale search found the path to intelligence, but the actual direct path that got to us required relatively fewer operations ( a few billion ~= generations * average mutations/generation). So, after all the effort we put into our own search towards AGI, the final solution might be relatively simpler than what the evolutionary compute suggests.
3. CNNs that relied on priors like locality and translation equivariance lit the fuse for computer vision, and transformers with attention mechanisms started the GenAI revolution. We need a scalable architecture with the right inductive priors to achieve human-level AGI. Some believe transformers/LLMs will get us there. Others, like Yann LeCun with I-JEPA, are proposing alternatives.
(2) this is an interesting one. I think it implies that the search is MUCH bigger than the find.... sortof like... if you're looking for gold... digging 6 feet into the earth takes like 20 minutes.... but searching the whole earth for *where* to dig can take many lifetimes.
This is a really nice analogy which emphasizes why we should carry forward all the knowledge humasn have acquired instead of re-running the search... we already have some pretty great places to dig!
Ultimately, the challenge is to know how to use human knowledge to guide the search; to figure out the architectures with the right priors, objective functions, and correct ways of using human knowledge.
After all, evolution's objective function is reproduction, and intelligence is a byproduct. Likewise, who could've predicted that next token prediction would result in all these emergent capabilities!