index
Ch.
4

Superintelligence is closer than it may appear

Chapter 4 illustration Chapter 4 illustration

The corporate analogy — which could just as easily be that of a general and an army — is also a good guide for the sort of AI superintelligence we’re likely to first encounter.1

Most current debate around advanced AI discusses “AGI,” which goes by many definitions. Here we will define it as autonomous general intelligence2 that matches or exceeds top human experts’ intellectual capability across relevant domains.3 Timelines to AGI (assuming continued large-scale efforts) are uncertain, but many experts and developers expect it by some definition in 2-5 years, possibly less.4

Our concern here is with superintelligence, AI that significantly or even dramatically exceeds human capability across relevant domains. If AGI competes with the best individual humans, superintelligence would surpass them, and compete with human civilizational capabilities (such as “doing science”) as a whole. Why discuss superintelligence if AGI doesn’t even yet exist? Because as we’ll now explain, a “weak” version of superintelligence is likely to follow immediately from AGI, and a strong version could follow relatively quickly afterwards, on a timescale of a few years at most.5

The immediate “weak” version comes simply from using a multiplicity of AGI systems. Just as teams of humans can get more and different things done than individuals, a large collective of AGI systems could together constitute a much stronger system worth calling superintelligence. This system could further scale itself by autonomously accessing more hardware with which to run and coordinate more copies. Also immediate would be the ability to increase the speed and/or depth-of-thought of each AGI system by scaling up available computation.6

After this, there are a number of ways in which the AGI aggregate could improve itself to become steadily stronger.7 Some of these, like improved hardware on which to run, are both slow and limited by real-world timescales. Others, which the AI can do fully autonomously on its own timescale, can happen at a large multiple of human-equivalent speed. This is very crucial: the full autonomy of AGI means that it would be able to design and implement improvements far faster than even the smartest human engineers.8 Here are some self-improvement pathways.

  • Hardware optimization: Increasing the speed of individual hardware elements, on a timescale of years, limited by manufacturing processes.9

  • Model retraining: A new/bigger/better core neural network trained with methodology improved by the AGI, on a timescale of weeks—months, limited by the training compute needed.

  • Model fine-tuning: Additional model training, using data and methods devised by the AGI,10 on a timescale of hours—days limited by available compute.

  • Tool development: New capability-enhancing software tools can be custom-written and improved by AGI, on a timescale of days to weeks.11

  • Improved “scaffolding”: Modern AI systems are composed of both neural networks and increasingly-sophisticated software “scaffolds” that bind those networks together and compose them with other software and with users. AGI could innovatively improve this scaffolding on a timescale of weeks.

  • Knowledge/prompt base: The AGIs could assemble their own “how-to” instructions, manuals, and general knowledge bases providing concise and improved summaries of important facts, procedures, and methods, that are iteratively improved for greater capability, on a timescale of weeks.12

  • Social/organizational innovation: Like groups of humans, AGIs could develop and improve their own social and organizational structures to work much better as groups, with innovation on a timescale of weeks.13

  • Other: In general, the AGI would be able to pursue essentially any pathway to increase its capability that we humans could pursue, as well as others that humans could not.

These considerations make the gap between AGI and superintelligence rather narrow: unless carefully and deliberately prevented from doing so, AGI would be perfectly able (and, due to instrumental incentives, likely motivated) to enter multiple mutually-reinforcing self-improvement cycles operating on timescales of days to months. Many such cycles fit into a year, so we can expect that if unconstrained, AGI would evolve into dramatically superhuman systems on that timescale or less.

It is therefore crucial to understand how, and particularly if, such systems can be kept under meaningful human control.

Footnotes

  1. Indeed this analogy is only barely an analogy: it is almost exactly the situation in which the human supervisors of a “weak” superintelligence composed of AGI systems would find themselves. This picture has been specifically promoted as a vision for AGI by the CEO of Anthropic; and an extended and detailed depiction of this scenario has been developed in the AI 2027 piece. ↩

  2. While the “A” usually stands for “artificial,” we follow Aguirre 2025 in emphasizing that it is the triple-intersection of autonomy, generality, and intelligence that is crucial, and distinct from AI systems circa mid-2025. ↩

  3. Relevant domains are those related to scientific, technological, mathematical, planning, social, etc. capabilities that provide economic and strategic power. Lack of phenomenological awareness, qualia, real empathy, and other consciousness-related capabilities are very unlikely to be necessary for these or for AI to constitute a transformative technology. ↩

  4. Predicting technological progress is always difficult, and for AGI there is diversity of both opinion and definitions. As two concrete predictions, (a) aggregated forecasting platforms place the median arrival time of “Weak AGI” in 2027 and a strong version, including robotics, in 2033; (b) Extrapolating METR’s time horizon dataset suggests AI systems around 2028-2030 that can autonomously perform tasks that would take humans a full month of work. Public statements from leaders at major AI labs frequently suggest similar timeframes. ↩

  5. The primary uncertainties would be availability of compute, and willingness to build the superintelligence or allow AGI to do so. This is not a given, but it is an extrapolation of the current race to build AGI, and the intention of at least several companies to build superintelligence. ↩

  6. The speed at which an individual neural network can produce each token is largely limited by the GPU speed and can only be somewhat increased by efficiency tricks. However, the number of AI systems running in parallel scales directly with compute. Increasing the depth of “thinking” via chain-of-thought generally causes responses to take longer; however with appropriate techniques these chains could be run much more in parallel, allowing them to be sped up (but probably sublinearly) with compute. ↩

  7. For a general discussion, see Bostrom’s Superintelligence. ↩

  8. As we don’t know the exact architecture of the system, this speedup multiple is unknown. The below uses a representative example value of 50x drawn from the range given in the AI 2027 scenario. This improvement would be “jagged”: some capabilities might get fast and dramatic advancement, for example via the type of self-play that allowed AlphaZero to go from novice to world class in Go in 30 hours. Others — especially any that require interaction with the slower-moving human world — could take longer. But all would proceed faster than humans could make happen. ↩

  9. Eventually, manufacturing processes would be automated and sped, but probably on a longer timescale than the AGI→\rightarrowsuperintelligence transition. ↩

  10. Although there are some pathways by which synthetic data can cause “model collapse,” others — such as generating data via a simulated environment or data that pertains to verifiable tasks like proving theorems — do not, and have been key to recent AI capability advances. ↩

  11. Here and elsewhere, for improvement done via intellectual labor, estimates are given on the basis of how long humans currently take for similar tasks, dilated by our assumed 50x speedup — so in this case days—weeks for AI where humans would take months. ↩

  12. While this may sound minor, it should not be underestimated. For example, the scientific method is a set of facts, procedures, and methods, that if made available to human civilization earlier could have profoundly changed its development. ↩

  13. There are ways in which grouping AGIs may have smaller returns than grouping humans. In particular, humans with different specialties and capabilities gain dramatically from combining complementary specialties, whereas very broad AGIs would not. Diversity is also very beneficial in a multitude of ways including both stabilization and innovation from the collective. On the other hand, AI would have additional powerful means at their disposal including: AI systems can be directly copied (greater homogeneity) or diversified by modifying or evolving those copies; AGI also can directly share knowledge very efficiently by model-weight updates or other highly efficient communication protocols it devises; and AGI would likely suffer less “interpersonal conflict” caused by personality differences or weaknesses. ↩

← Previous Chapter
Next Chapter →