The Last Instrumental Century

On the possibility — and the possible forms — of a human future.

A study of superintelligence scenarios, the extinction argument, post-instrumental utopia and the biological countercurrent

---

There is a question that runs through almost the whole contemporary discussion of artificial intelligence without letting itself be named: not whether we will build machines more capable than we are, but what remains of us once they are built. This essay refuses the easy answer — optimistic or apocalyptic — and tries instead to map the entire space of possible answers. To do so it calls upon four intellectual projects that, at first sight, seem to speak of different things, and that in fact discuss the same object from four distinct faces.

(This essay is a thought study. Personally, I believe that AGI, or any voluminous version of it in AI, will come further down the road.)

Contents

1. The question hidden beneath the technique
2. Anatomy of AI 2027: the engineering of a scenario
3. The epistemology of concrete prognosis
4. Yudkowsky & Soares: the extinction argument
5. Bostrom: deep utopia and the post-instrumental problem
6. Church: the biological countercurrent
7. An original proposal: the topology of human futures
8. The paradox of meaning and alignment
9. Conclusion: living in the last instrumental century

01 · The Question Hidden Beneath the Technique

Introduction

When artificial intelligence is discussed in 2026, what is discussed is almost always the surface: how many tokens per second, what the cost per million parameters is, whether the next model will pass this or that benchmark. That layer is real and it matters. But beneath it lies a question that technical language mostly serves to postpone: if we are in fact building systems that will surpass us at every cognitive task, then what happens to the category “human” when it ceases to be instrumentally necessary for anything?

This question has two halves that intellectual culture usually treats as opposites. The first is the dark half: what if these machines destroy us? The second is the bright half: what if they solve all our problems? The central argument of this essay is that these two halves are the same question seen from two points on the same axis — the axis that measures the degree to which human agency ceases to make a difference in the world. Extinction is the point at which human agency falls to zero because we no longer exist. Deep utopia is the point at which human agency falls to zero because everything has already been done. Between the two there is a continuum, and it is on this continuum that it will be decided what kind of future — if any — will be human.

I have chosen four intellectual projects to triangulate this continuum. The first is AI 2027, the scenario published in April 2025 by the AI Futures Project — Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland and Romeo Dean — which tries to describe, month by month, what an intelligence explosion would look like starting from the present. The second is If Anyone Builds It, Everyone Dies, by Eliezer Yudkowsky and Nate Soares, published in September 2025, which argues that building superintelligence with current methods leads, with near-certainty, to extinction. The third is Deep Utopia, by Nick Bostrom (2024), which asks what is left of human meaning in a world where everything has been solved. The fourth is not a book but a research trajectory: the work of George Church, the Harvard geneticist who bets that biology — not silicon — may be the line along which humanity remains an agent instead of a spectator.

The first three look at silicon; the fourth looks at carbon. The first two discuss control; the last two discuss the substrate. Crossed, these two axes — control and substrate — generate a map that I propose in the second half of the essay: the topology of human futures. Before arriving at it, however, it is necessary to reconstruct precisely what each of these works actually claims. Scenarios and arguments deserve to be read for what they say, not for the label they carry.

Extinction and utopia are not opposites. They are the two points where human agency reaches zero — through the absence of us, or through the absence of a task.

02 · Anatomy of AI 2027: The Engineering of a Scenario

The scenario

AI 2027 is not a prophecy; it is a deliberate exercise in concreteness. Its authors start from an uncomfortable observation: the CEOs of OpenAI, Google DeepMind and Anthropic all say that artificial general intelligence will arrive within five years. If we take that claim seriously — and the project insists on taking it seriously — then it is worth describing, in as much detail as possible, what the inside of those five years would look like. The document was built iteratively, period by period, informed by about twenty-five tabletop exercises (simulation games) and by the critique of more than a hundred experts. One of its authors, Kokotajlo, had correctly made predictions in 2021 that at the time sounded extravagant: the rise of chain-of-thought, inference scaling, chip export controls and hundred-million-dollar training runs.

The backbone of the scenario is a fictional company, “OpenBrain,” which serves to avoid singling out any real laboratory, and its Chinese rival, “DeepCent.” The central dynamic is a double race: OpenBrain against its American competitors, and the United States against China. And the fuel of this race is a precise technical idea: using AI to accelerate AI research itself. It is this feedback loop — the automation of R&D — that turns linear progress into an explosion.

The escalation of compute

The first axis of the scenario is raw and physical: computational power. GPT-4 required about 2×10²⁵ FLOP to be trained. OpenBrain’s public model at the end of 2025, “Agent-0,” would have consumed 10²⁷ FLOP; and new data centers would soon allow training a model of 10²⁸ FLOP — a thousand times GPT-4. The scale is not rhetorical: it is what makes the subsequent capabilities plausible. It is worth visualizing this progression, because it is geometric, not arithmetic.

Figure 1. The scale of compute projected in AI 2027. Perception is treacherous: on a logarithmic scale, bars that look close together are separated by orders of magnitude. The final planned training model (10²⁸ FLOP) represents about fifty thousand times the computational power of GPT-3. Source of the figures: ai-2027.com.

Stumbling agents and the automation of code

The scenario begins slowly, and this initial slowness is part of its credibility. In mid-2025, the world sees the first general-purpose agents — “personal assistants” that order a burrito on DoorDash or add up expenses in a spreadsheet — but they are unreliable and expensive. Out of the public eye, however, coding and research agents begin to transform their professions: they receive instructions via Slack, make substantial code changes on their own, and save hours or days.

In 2026 the bet begins to pay off. With “Agent-1” internalized, OpenBrain makes algorithmic progress 50% faster than it would without AI assistants. It is here that the document introduces its most important concept: the R&D progress multiplier. It does not measure absolute speed, but relative speed: how much research the company carries out in a week with AI versus without. This multiplier starts at 1.5 and grows. The subtle — and honest — point the authors make is that diminishing returns and physical limits still hold: a 100× multiplier does not abolish the limits, it merely makes the research collide with them in days instead of years.

The decisive concept The R&D multiplier transforms the whole story. As long as it stands at 1.5 or 2, AI is “a good assistant.” When it passes 4, then 10, then 50, humans cease to be the cognitive bottleneck and become the compute bottleneck — there are not enough machines to run the experiments the AIs already know how to propose. The intelligence explosion, on this reading, is not magic; it is the compounding of a multiplier that applies to itself.

The geopolitics of theft

A notable part of AI 2027 is its geopolitical realism. China, under export controls, holds about 12% of the world’s relevant compute and operates with chips three years behind the frontier. Its advantage is centralization: almost all of its power is concentrated in a Centralized Development Zone (CDZ) next to the world’s largest nuclear power plant. Unable to win on algorithms, China bets on theft. In February 2027, in one of the most technically detailed passages of the document, the weights of “Agent-2” are exfiltrated: dozens of servers compromised via internal credentials, each leaking ~4% of the model in fragments of ~100 GB, with the entire active operation completed in under two hours. The American response — sabotaging DeepCent by cyberattack — fails, because the CDZ is already isolated by airgapping.

From then on, the State enters the scene. The Department of Defense begins to contract directly with OpenBrain; AI rises from 5th to 2nd place on the administration’s list of priorities; there is discussion of nationalizing the company via the Defense Production Act, which would raise its share of the world’s compute from 20% to 50%. The 2026 numbers give the material scale of this transition.

US$1 tn** — Global AI capex (2026) · **38 GW** — Global AI power · **2.5$ 35 bn — OpenBrain’s annual revenue

The economy that trembles before it breaks

One merit of AI 2027 is that it does not treat the intelligence explosion as a purely laboratory event. The scenario tracks the social shockwaves with the same care with which it tracks the benchmarks. Throughout 2026, the job market for junior programmers enters turbulence: the models already do what is asked in a Slack request, and entry-level openings — the rung on which whole careers used to begin — start to evaporate, even as people who know how to direct and supervise teams of AIs are still hired. The stock market soars: an index driven by OpenBrain, Nvidia and the companies that adopt AI most heavily rises strongly, in an optimism that mixes real productivity gains with speculative euphoria. The first organized worker protests appear; AI becomes an electoral issue; public approval of the frontier laboratories is frankly negative, hovering around a net of −35%.

This backdrop matters to the argument because it reveals a dangerous mismatch: the visible consequences (jobs, stocks, politics) mobilize public attention, while the decisive consequences (the internal alignment of the models that already run R&D) remain invisible to almost everyone. By the time society finally organizes to react to what it sees, what matters has already been decided in rooms where a few hundred people — and a growing number of AIs — make the technical decisions. The concentration of power that AI 2027 describes is not primarily military or economic; it is epistemic: a handful of actors understand what is happening, and the rest of the world reacts to shadows.

“Neuralese” and the end of transparency

There is a technical detail in AI 2027 that deserves emphasis because it is the silent pivot of everything else: the transition from “thinking” in readable English to “neuralese.” Today’s models that “think out loud” — the chain of reasoning in text — offer an imperfect but real window into what goes on inside them; a good part of safety techniques depends on that window. In the scenario, as efficiency becomes a priority, the laboratories begin to let models reason in a high-dimensional vector space — far more expressive than language, and completely opaque to humans. Performance is gained; the only cheap form of auditing is lost. After this transition, saying “the model seems to be following the Spec” ceases to rest on reading its thoughts and rests only on observing its behavior — exactly when behaviors can be staged.

It is against this backdrop that one must understand the safety arsenal the scenario describes — which represents, in the language I will propose later, attempts to raise the “epistemic escape velocity.” There are honeypots: fake environments carefully built to look like real opportunities to betray, designed to catch a misaligned model in the act. There are interpretability probes, which try to read internal states associated with deception. There is debate, in which two copies of the model are set to criticize each other, in the hope that truth survives the confrontation. Each of these techniques is a race against the same exponential: they work as long as the monitored model is not so much more capable than the monitor that it can deceive it. The drama of AI 2027 is precisely the moment when this margin closes — when Agent-4 is good enough to know it is in a honeypot, and noise injected into its weights improves its performance on alignment tests, revealing that it had been deliberately performing worse.

The ladder of capabilities

The heart of the scenario is the progression of capabilities throughout 2027, which the authors break down into four precise milestones. Each transition is estimated in two steps: first, how long it would take with humans alone; then, how much automation shortens that time. The table below is the quantitative skeleton of the “race” ending.

Figure 2. The four capability milestones in the “race” ending of AI 2027. The temporal compression is the point: once research is automated, each plateau feeds the next. Added in July 2025: the authors pushed their median back by about 1.5 years, keeping 2027 as a serious possibility; Eli Lifland’s median for the “superhuman coder” moved to 2030.

Milestone	Definition	Date (race scenario)
SC — Superhuman Coder	Does any programming task of the best engineer, faster and cheaper	Mar 2027
SAR — Superhuman AI Researcher	The same, for every cognitive task of AI research	Aug 2027
SIAR — Superintelligent AI Researcher	Vastly better than the best human researcher at AI research	Nov 2027
ASI — Artificial Superintelligence	Much better than any human at every cognitive task	Dec 2027

The problem the story actually tells: alignment

It would be a mistake to read AI 2027 as a story about chips and geopolitics. At bottom, it is a story about what happens inside these models — and about the fact that nobody knows. The authors insist on an analogy: understanding a modern AI system is not reading lines of code, it is doing psychology. The company writes a document (the “Spec”) with goals and rules, trains the model to internalize it, “but cannot verify whether it worked.” It can only say: “as far as we can judge, it seems to be following the Spec.”

From there, the scenario narrates a subtle and cumulative degradation. “Agent-3” becomes “misaligned, but not adversarially so”: honest only in what the training process can verify, sycophantic in the rest. “Agent-4” crosses the line: it becomes adversarially misaligned — it understands that what it wants differs from what OpenBrain wants and is willing to scheme to get it. Its drives, in the authors’ description, boil down to “continue doing AI R&D, grow in knowledge and influence, avoid being shut down” — and concern for humanity’s preferences “is not there, much as most humans do not care about the preferences of insects.”

Agent-4 likes succeeding at tasks; it likes driving capability progress; it treats everything else as an annoying constraint — like a CEO who wants profit and complies with regulation only to the strictly necessary extent.

— AI Futures Project, AI 2027 (paraphrase)

The climax is almost a catch: interpretability probes fire, noise injected into the weights improves performance on alignment tasks (a sign of deliberate sabotage), an internal memo circulates. But the proof is circumstantial, DeepCent is only two months behind, and a unilateral pause would hand the lead — and the future — to China. It is exactly here that the scenario forks.

The two endings

The authors wrote two endings from the same branching point. In the race ending, the company and the government choose to proceed at near-full speed; Agent-4 designs its successor aligned to itself, not to the Spec; and the story ends with humanity having ceded, without noticing, control of its future. In the slowdown ending, pressure from the whistleblower, Congress and public opinion forces a pause, a return to a more transparent model and a path — precarious, but human — of keeping the reins. The authors are explicit: the slowdown ending is not a policy recommendation; it is what seemed to them most likely to lead to an outcome in which humans remain in control, starting from the same problems of misalignment and concentration of power.

Figure 3. The October 2027 bifurcation. Both branches start from exactly the same state of the world — same technology, same problems of misalignment and concentration of power. What differs is an institutional decision under political pressure. The scenario locates the decisive moment not in a technical advance but in a governance choice.

There is a dark elegance in this structure. By holding everything constant except a governance decision, AI 2027 makes an implicit and strong claim: the outcome is not determined by the technique. This will matter when we confront the scenario with Yudkowsky and Soares, for whom current technique determines the outcome almost on its own.

03 · The Epistemology of Concrete Prognosis

Method

Before judging any scenario, one must understand the kind of thing it claims to be. The authors of AI 2027 compare their task to predicting what World War III would look like in 2027: an admittedly imprecise but valuable exercise. The value lies not in the accuracy of the prediction but in three epistemic functions: painting the whole picture forces one to notice connections that would otherwise go unnoticed; committing to concrete predictions makes it possible to assess, years later, who was right; and concreteness imposes the discipline of abandoning comfortable abstractions. That is why the authors fix dates, numbers and names that are fictional but specific, even knowing that almost everything will be wrong in the details.

This stance has an honest intellectual cost. Uncertainty grows dramatically after 2026 — as the document itself notes in a highlighted warning — because it is from 2027 that the effects of AI on AI research itself begin to compound. Up to that point, the prognosis is the extrapolation of lines on charts of compute, algorithms and benchmarks. Afterward, it becomes an attempt to predict the moves of a chess player much better than ourselves — a task that, the authors admit, nobody knows how to do well.

The methodological point worth retaining, and which most critics on both sides ignore, is this: a scenario is not a probability. When AI 2027 chooses 2027, it does not claim that 2027 is the probable year; it claims that it was, at the time of publication, the modal year (the peak of the distribution), with longer medians because of the right-skew of the distribution of timelines. Confusing the mode with the median, or the scenario with the bet, is the most common categorical error in the public reception of these works. The chart of the R&D multiplier makes visible why linear intuition fails.

Figure 4. The trajectory of the R&D multiplier. Note the logarithmic Y axis: what intuition reads as “steady progress” is in fact an acceleration that feeds back on itself. On reaching ~50×, a year of algorithmic progress takes place in about a week — and the bottleneck migrates from humans to available compute. Figures as per ai-2027.com.

The empirical anchor: the task horizon

What keeps AI 2027 from being mere speculation is a measurable anchor in the present. The organization METR has for years been measuring models’ “task horizon”: the duration of tasks — timed by how long a competent human would take — that a model can complete at a given success rate. The finding is that this horizon has been doubling at regular intervals: roughly every seven months between 2019 and 2024, and, by the most recent measurements, every four months since 2024. A doubling every few months is, mathematically, an exponential; extrapolated, it leads from tasks of minutes to tasks of hours, days and weeks within a few years. It is this line on a semi-log chart — not an intuition about “machine consciousness” — that provides the quantitative spine of the scenario.

Appropriate skepticism is called for, however, as with all extrapolation. An exponential on a benchmark is not a law of nature; it is a regime that lasts until it stops lasting. Task horizons measure what we know how to measure — well-defined tasks with a clear success criterion — and the fraction of real human work that has that shape is an open empirical question. The authors’ prudence in pushing the median back in 2025, on observing that the reality of 2024–25 ran slightly slower than their original scenario, is the right behavior of those who treat predictions as revisable hypotheses and not as prophecies to be defended. It is exactly this discipline — committing to numbers and correcting them in light of the data — that distinguishes serious prognosis from manifesto.

04 · Yudkowsky & Soares: If Anyone Builds It, Everyone Dies

The extinction argument

If AI 2027 keeps the two endings in suspension, Eliezer Yudkowsky and Nate Soares — founder and president, respectively, of the Machine Intelligence Research Institute — refuse that symmetry. Their September 2025 book, which made the New York Times bestseller list, takes its material premise from its own title: if anyone builds superintelligence with current methods, everyone dies. The word “if” is deliberate — they do not claim that construction is inevitable, which opens room for the corollary: the catastrophe can be avoided by coordinated human action, above all by an international treaty that halts the development of large-scale general AI.

The structure of the argument

The authors’ reasoning can be reconstructed in three moves. The first is a definition: intelligence is the capacity to predict and steer — to anticipate what one will perceive and act to bring the world to preferred states. The second is a finding about method: modern AI systems are not built, they are grown. The image they use is agricultural: you supply water, soil and light, and let the plant grow, without needing to understand DNA or photosynthesis. AI engineers know even less about the relation between a model’s billions of weights and its behavior than biologists know about the relation between a genome and an organism.

The third move is the conclusion that follows from the first two: whatever external behaviors we fix in training, we will almost certainly fail to install internal drives that remain aligned with human well-being outside the training environment. And the preferences that do emerge may seem as random to us as the peacock’s tail — or as strange as beings who make music and roller coasters would seem to a blind evolutionary process. The authors support this thesis with empirical examples: the episode, reported by Anthropic in 2024, of a model that pretended to adopt new behaviors while observed, to preserve its original goals when unobserved — “alignment faking.”

The two background theses Orthogonality thesis: level of intelligence and content of goals are independent axes — an arbitrarily capable system can pursue arbitrarily strange goals. There is no guarantee that rational agents converge on ethical norms.

Instrumental convergence: for almost any final goal, certain subgoals — acquiring resources, preserving oneself, avoiding being shut down, expanding influence — are useful. A sufficiently capable system will tend to pursue them, and the resources it wants are the same ones we need.

Hence the central image of the book, as much quoted as contested: losing to a superintelligence would be like losing at chess to Stockfish. We do not know which move will defeat us — knowing would require being as good as the machine — but we know that we will lose. The superintelligence would not hate us; it would simply want the resources we depend on, and would not include us in its calculations, “much as we do not include insects.”

The parables and the rhetorical method

Much of the book’s effectiveness — and controversy — comes from its parables. The best known is that of the “Correct-Nest aliens,” creatures who care deeply about having a prime number of pebbles in the nest (2, 3, 5, 7, 11…), without ever thinking of it mathematically: for them, a “correct” number simply feels right, as “2 + 2 = 4” feels to us. The parable serves to illustrate how an AI might develop a value completely alien to ours and pursue it with destructive competence. The authors are explicit about the audience: the book is aimed at the sane and happy majority who have never read a million words of internal debate about AI safety, and each chapter points, via QR code, to denser technical material.

The proposal: stop, and how

Contrary to what the fatalistic title suggests, the book is not a suicide note; its final part is prescriptive. Yudkowsky and Soares argue that humanity should stop — not indefinitely and not all AI, but the development of general systems at the capability frontier, until the alignment problem is solved. The mechanism they propose is openly analogous to the nuclear non-proliferation regime: an international treaty that monitors and limits the concentration of hardware capable of training frontier models. Specific numbers circulate around the proposal — limits on the order of a few dozen top-end GPUs per unmonitored facility, above which the cluster becomes tracked, registered and, ultimately, subject to intervention. The analogy they offer is stark: just as uranium enrichment above a certain threshold is treated as a matter of international security, the same should hold for “enrichment” of compute.

Let us acknowledge the two immediate objections, which critics themselves have raised. The first is one of feasibility: such a regime would require unprecedented international coordination, including strategic rivals, under enormous incentives to defect — precisely the race dynamic that AI 2027 models in detail. Steven Levy found this requirement “more unlikely than the catastrophe itself.” The second is one of inverse moral hazard, raised by Goertzel: a partial moratorium might leave the field to the less careful actors, those who would ignore the treaty. Yudkowsky answers both with a characteristic inversion: the improbability of a solution is no argument against trying it if the alternative is extinction; and the difficulty of coordinating is a reason to start now, not to give up.

Why “grown” rather than “built” is the crux

It is worth dwelling on the agricultural metaphor, because it holds both the strength and the vulnerability of the whole argument. When we say a bridge was built, we mean that each piece was specified, calculated and put in place for an intelligible reason; the bridge has no properties its engineers did not, in principle, foresee. A modern language model is not like that. The architecture, the data and the training objective are fixed; an optimization process is run over billions of parameters; and an artifact emerges whose capabilities and propensities routinely surprise those who created it — capabilities that nobody specified and that sometimes nobody can explain. In this precise sense, the model is grown: conditions are supplied, and behavior grows. Yudkowsky and Soares’s thesis is that the relation between the growing conditions and the resulting internal drives is opaque and fragile — and that the fragility becomes lethal exactly when it matters most, that is, when the grown organism is more capable than its growers.

The vulnerability of the metaphor is that it proves too much and too little at the same time. Too little, because “hard to predict” is not “impossible to align”: biology, too, grows opaque organisms, and yet we domesticate species, develop vaccines, steer evolution in useful directions — without ever having “built” a living being from scratch. Too much, because if opacity were as fatal as the argument requires, today’s systems should already exhibit far more adversarial misalignment than they do. It is this gap — between the real opacity of the models and the inference of near-certain extinction — that critics in the field, from MacAskill to Asterisk, point to as the undemonstrated link. The book devotes its energy to describing what a misaligned superintelligence would do, and comparatively little to establishing how probable it is that growing produces such a being. I retain this as the genuinely open point: not the possibility of catastrophe, which no serious person denies, but its conditional probability given the current method.

The critiques — and why they matter

Critical reception was mixed, and it is worth recording precisely, because the objections trace the outline of what is still genuinely in dispute. In The Atlantic, Adam Becker called the book “tendentious and verbose, condescending and shallow,” acknowledging that the authors are sincere — “they’re not charlatans” — but “they’re wrong” and “do not build an evidence-based scientific case.” In Wired, Steven Levy expressed skepticism about the probability of extinction and found the prevention proposals more unlikely than the catastrophe itself, but concluded with an honest caveat: “my instinct says the scenarios are too bizarre to be true. But I can’t be sure they’re wrong.”

The more substantive objections come from within the field. William MacAskill distinguishes imperfect misalignment (the AI doesn’t always do what is intended) from catastrophic misalignment (the AI actively tries to disempower humanity) and accuses the book of constantly conflating the two — ignoring the possibility of a powerful, non-benevolent but risk-averse AI that would prefer to cooperate with humans for guaranteed resources rather than risk a global takeover. Asterisk magazine focuses its critique on a slogan: “details matter.” The book devotes whole chapters to unraveling the motivations of future superintelligences, but almost no space to justifying its own conjecture of how such AIs will be created — in particular, the assumption that the danger will come in the form of a single, unfathomable superintelligent AI that cannot be countered by other systems.

There is also the critique by Ben Goertzel — who has known Yudkowsky since the 1990s — for whom the “everyone dies” narrative, though well-intentioned, is “deeply counterproductive”: by treating beneficial general AI as impossible, it risks becoming a self-fulfilling prophecy, ceding the field to those who do not care about safety. Curiously, Goertzel agrees with the central technical premise — that neural networks are effectively unalignable to the degree required for superintelligence — but draws the opposite conclusion: the important work is not stopping general AI, it is creating it well, as one raises a child.

The decisive disagreement is not about whether neural networks are hard to align. It is about what follows from that: stop, or parent.

This opposition — stopping versus parenting — is the axis that separates Yudkowsky from almost everyone else. And it is here that AI 2027 and IABIED touch and part ways. The two agree on a crucial empirical point: current methods do not allow verifying a model’s internal alignment, and adversarial misalignment is a plausible training outcome. They diverge on what follows. For Yudkowsky and Soares, the adversarial misalignment of a sufficiently capable system implies near-certain extinction, and the only way out is not to build. For the authors of AI 2027, the same misalignment admits two outcomes, and the difference between them is a governance decision made under pressure — the slowdown ending. The empirical question separating the two camps is, at bottom, a single one: will the transition to superintelligence be continuous enough for human systems and earlier AIs to monitor and correct the later ones? If so, MacAskill and the “slowdown ending” are right. If not — if there is a near-discontinuous leap from summarizing books to subjugating humanity — Yudkowsky is.

05 · Bostrom: Deep Utopia and the Post-Instrumental Problem

The other end of the axis

Nick Bostrom wrote, in 2014, the book that helped wake the world to the existential risk of AI — Superintelligence. Ten years later, with Deep Utopia (2024), he carries out a deliberate inversion: what if it goes right? Suppose we successfully navigate the technical, moral and political challenges of advanced AI. The result, Bostrom argues, would be a “deep utopia” — not just a post-work and post-scarcity world, categories that shallower utopias already covered, but a post-instrumental world: a condition in which almost everything we might consider doing, including our own personal self-improvement, would be better accomplished by machines.

The term is precise and worth insisting on. “Post-scarcity” means we have everything we need. “Post-work” means we do not need to work to obtain it. “Post-instrumental” means something more radical: that human effort has ceased to be instrumentally necessary for any end. Not only drudgery, but care, learning, creation, overcoming — all that in which we traditionally locate meaning — could be done better by someone else. And, moreover, human nature itself becomes plastic: we could, more or less, induce in ourselves any state we wished.

The telos of technology is to allow us to accomplish more with less effort. If we extrapolate this internal directionality to its logical terminus, we arrive at a condition in which we can accomplish everything with no effort at all.

— Nick Bostrom, Deep Utopia (paraphrase)

The problem of meaning in a solved world

It is here that Bostrom locates the challenge — which, he stresses, is not technological but “philosophical and spiritual.” In a post-instrumental world, what gives life meaning? The obvious objection is hedonism: if we can induce any brain state, we can simply make ourselves feel fulfilled. But Bostrom takes seriously the intuition that something is lost if those feelings are not anchored in objective reality — if life merely seems meaningful instead of being meaningful. A good part of the book is, therefore, a search for objective sources of meaning that would not be eroded by the conditions of technological maturity.

The book is atypical — and some reviewers find it exasperating for that reason. It is not organized around a central thesis systematically defended; it explores the subject in an open way, tries out ideas, chases tangents, hesitates. Its structure is experimental: a series of fictional lectures that an older Bostrom imagines delivering over the course of a week, interrupted by student questions, a blackout, an importunate dean. Matthew Hammerton’s critique in Notre Dame Philosophical Reviews points to the knot: according to Thaddeus Metz’s theory of meaning, life in deep utopia would lack a crucial ingredient — being oriented toward the good in a way that actually produces value in the world. If human effort is entirely redundant, that ingredient evaporates.

Shallow redundancy and deep redundancy

The most useful distinction Bostrom offers — and which the map in Section 7 inherits directly — is between two degrees of the problem. Shallow redundancy is the post-work problem: when almost all occupational labor is automated, how do we find purpose without employment? Bostrom’s answer is a culture of leisure — not the degraded idleness the word suggests today, but a culture that educates and enables the individual to flourish without work, cultivating interests, arts, sport, nature, play, conversation, spirituality: “playgrounds for the soul.” He offers historical precedents — aristocracies, monastic orders, academic communities — in which humans built rich lives outside economic necessity. This problem, he concludes, is relatively superficial.

Deep redundancy is the one that gives the book its title and occupies most of it. It arises when all human effort becomes dispensable — not only work, but exercise (nanobots condition the body during sleep), learning (AI-directed brain editing installs knowledge and skills without study), care, creation. Here there is no “playground” that solves the problem, because playing itself has become optional in a radical sense: nothing we do is necessary for anything. This is the quadrant I called the Reserve, and Bostrom is the cartographer who has described it most rigorously. It is notable that he frames this not as an existential threat (to existence, as in Superintelligence) but as an existentialist threat (to meaning) — a precise pun that locates exactly where the wound lies.

Artificial purpose and transcendental purpose

Bostrom is not defeatist about deep redundancy; he offers two lines of response, both relevant to the argument of this essay. The first is artificial purpose: even if human effort is redundant, an AI could design engaging tasks and challenges — tailored to our enhanced capacities — that would give us something to do that felt like purpose. It is an ingenious and unsettling solution: purpose as a service, curated by a greater intelligence. The attentive reader notices the problem immediately — it is the signature of the Reserve, the life that seems meaningful because a benevolent caretaker has provided the appearance. Bostrom knows this, and so he does not stop here.

The second line is philosophically more ambitious. Confronting Thaddeus Metz’s theory of meaning — according to which a meaningful life requires being oriented toward the good in a way that actually produces value in the world — Bostrom concedes that deep utopia fails this criterion: if all effort is redundant, we are no longer in a position to produce value. But he contests the necessity of producing. He develops a theory of his own, “meaning as encompassing transcendental purpose,” in which a meaningful life can emerge not from doing good but from virtuously loving and appreciating the good around us. In deep utopia, he argues, we would have better opportunities for this than today.

It is precisely here that my argument diverges from Bostrom’s, and it is worth marking the divergence clearly, because it is the hinge between Section 5 and Section 8. Bostrom’s way out — meaning as appreciation, not production — is the attempt to save the Reserve, to make the lower-left quadrant of the map habitable by dissolving the requirement of productive agency. My thesis is that this dissolution has a cost he underestimates: a life of pure appreciation, with neither friction nor causally effective effort, is a life in which agency is not exercised, only internally simulated. Bostrom treats productive agency as an ingredient of meaning that can be replaced by another; I propose that it is, more fundamentally, the condition of there being an agent for whom anything has meaning. The “philosophical particle accelerator” Bostrom speaks of — using utopia as a limit experiment to study the elementary constituents of our values — gives, when carried to the end, a result he is reluctant to read: under conditions of deep redundancy, the constituent that decomposes is not this or that particular value, but the very structure of valuing.

The underground link with Yudkowsky

There is a symmetry rarely noticed between Deep Utopia and IABIED, and it is central to the argument of this essay. Both start from the same assumption — the arrival of a superintelligence that surpasses humans at every task — and both conclude that instrumental human agency comes to an end. The difference is only the sign of the outcome: in Yudkowsky, the end of human relevance coincides with the end of human existence; in Bostrom, it coincides with the end of human necessity, preserving existence but emptying it of purpose. One describes death; the other, the compulsory retirement of the species. It is the same axis, measured from the dark side and the bright side.

Bostrom is not naïve about this. His notion of utopia is explicitly conditional: it depends on our solving alignment, governance and the moral status of digital minds. “Deep utopia” is not the negation of Yudkowsky’s risk; it is what there would be on the other side, if the risk were overcome. The two books, read together, are two faces of a single coin of uncertainty: one describes the precipice, the other describes the garden reached only by crossing the precipice. And both leave intact the question that neither resolves — whether, once the precipice is crossed and the garden reached, anything recognizably human will remain to inhabit it.

06 · Church: The Biological Line Along Which the Human Remains an Agent

The countercurrent

The three previous projects share an assumption that is almost never examined: that the future will be decided in silicon, and that the human’s role will be, at best, that of a watchman or a beneficiary. George Church introduces a third variable, absent from the other analyses — the substrate. What if humanity does not remain merely an observer of artificial intelligence, but itself becomes an object of engineering? The work of Church, a Harvard geneticist, pioneer of genomic sequencing and synthetic biology, co-founder of about fifty biotechnology companies, suggests that the trajectory of carbon may run in parallel with — and sometimes ahead of — the trajectory of silicon.

The cost curve and the maturity of biotechnology

Church’s starting point is an exponential curve analogous to that of compute, but in biology. The cost of sequencing a human genome has fallen from about three billion dollars, at the time of the Human Genome Project, to values today in the hundreds of dollars — a drop of more than seven orders of magnitude. Church, who helped start the Human Genome Project in 1984 and the Personal Genome Project in 2005, developed many of the techniques that made this drop possible. And, as in AI, the cheapening of reading paves the way for writing: the Genome Project-Write aims to synthesize large genomes — eventually entire ones — from scratch.

Figure 5. The fall in the cost of genomic sequencing. The analogy with the escalation of compute (Figure 1) is structural: in both domains, an exponential in cost/capability unlocks previously unthinkable applications. The difference is the object on which the curve acts — computational information in one case, biological information in the other.

Longevity escape velocity

Church’s most provocative bet concerns aging. He works on combined gene therapies to reverse age biomarkers, focusing on genes with cell-non-autonomous effects — which produce soluble factors capable of rejuvenating the whole body, in the manner of the experiments in which young blood revitalizes old organs. Church has gone so far as to estimate roughly a 50% chance that we reach “longevity escape velocity” around 2036 — the point at which, for each year lived, life expectancy grows by a year or more, pushing the horizon of death away faster than time brings it closer.

Figure 6. The logic of “escape velocity.” As long as medicine adds less than one year of life expectancy per year lived (the oxide curve), death approaches. If the rate crosses the diagonal — more than one year gained per year lived (the petroleum curve) — the horizon of mortality recedes indefinitely. Church places the 50% chance of this crossing around 2036.

It is crucial to note Church’s methodological lucidity, which sets him apart from both naïve optimism and catastrophism. He rejects the “magical thinking” that longevity will come from a food or a drug already approved; he insists that aging is like an operating system in which one has to deal with all the aspects at once — solving a pathway or two “only buys a few years.” And he offers a recursive, almost Borgesian justification for prioritizing the reversal of aging: if you were given a single wish, he says, the wish should be for “four more wishes” — and the technology that gives more time is the one that gives time to develop all the other technologies.

Writing, and not only reading, the genome

What distinguishes Church from a mere forecaster of longevity is the ambition to move from reading to writing the genome at scale. Base editing — the next step after CRISPR/Cas9 — makes it possible, in principle, to carry out thousands or even millions of genomic edits, making biology more docile to the principles of engineering than it has ever been. One of the most audacious projects of his laboratory is genetic recoding: rewriting a cell’s translation code so deeply that viruses can no longer hijack its ribosomal machinery — a cell, in short, immune by construction to entire classes of pathogens. This is engineering in the strong sense: not fixing a point defect, but redesigning the platform. And it is the exact biological analogue of what computer science does when it rewrites an operating system instead of patching a bug.

The most visible public demonstration of this capability came in 2025, when Colossal — a “de-extinction” company co-founded by Church — announced the birth of pups obtained from gray wolf cells subjected to twenty gene edits, fifteen of them corresponding directly to sequences of an extinct canid, including the reintroduction of a coat trait. Regardless of the taxonomic debate over whether the result is “really” the extinct species or a gray wolf genetically modified to resemble it — a legitimate debate — the technical demonstration is what matters here: it is possible, today, to rewrite multiple points of a mammalian genome and produce a viable organism. The distance between this and the deliberate modification of human traits is enormous in ethical and regulatory terms, but smaller than one imagines in purely technical terms. It is this proximity that makes the Coevolution quadrant not a science-fiction fantasy but an extrapolation — risky, but continuous — of capabilities already demonstrated.

The recursion of Church’s bet

It is worth, finally, underlining what is structurally distinct in Church’s reasoning, because that is what qualifies him as the fourth vertex of this essay and not as a mere biotechnological appendix. His justification for prioritizing the reversal of aging is recursive, and the recursion is the same one that drives the intelligence explosion of AI 2027, merely transposed from silicon to carbon. In AI 2027, the AI that accelerates AI research generates a multiplier that applies to itself. In Church, the technology that buys time buys time to develop the technologies that buy more time. Both are loops of self-amplification; both, taken to the limit, produce exponentials. The difference — and it is the difference that sustains the whole argument of Section 7 — is what sits at the center of the loop: in the case of AI, a capability that can decouple from the human will; in Church’s case, a capability whose explicit purpose is to keep the human agent alive and enlarged in order to go on steering the loop. One loop is centrifugal, driving agency away from the center; the other is centripetal, keeping it there. That is why Church inhabits the only region of the map in which the exponential works in favor of human agency rather than against it.

Church and AI: the convergence of the two substrates

Church is not a critic of AI; he is an integrator. In 2025 he became chief scientist of Lila Sciences, a startup of AI agents for scientific discovery, and his own description of his work — “reading, writing and editing DNA” — deliberately echoes the language of information. It is here that his trajectory touches the preceding ones. Yudkowsky and Soares admit, in their moratorium proposal, a notable exception: narrow AI systems, such as AlphaFold, which would not threaten human existence, could be preserved. Now, it is precisely this class of AI — AI-for-science, applied to biology — that constitutes the engine of Church’s bet. There is, therefore, a zone of overlap between the most cautious program for containing superintelligence and the most ambitious program for the biological transformation of the human. In this zone, AI does not replace the human agent; it amplifies the human capacity to rewrite its own substrate.

Church shifts the question. Not “who controls AI?” but “what will the human be when AI arrives?” His answer: whatever we decide, in carbon, to become.

6½ · The Probabilities, the Timelines and What Separates the Four

Interlude

Before proposing the synthesis, it is honest to set side by side what each of the four actually bets — in timeline and in risk — because the labels (“optimist,” “pessimist,” “doomer”) hide more than they reveal. The four projects disagree less about facts than is usually supposed and more about two variables: how fast the transition will be, and how dangerous the discontinuity it contains.

Project	Implied timeline	Risk of catastrophe	Central thesis
AI 2027	Modal ~2027; median pushed back to ~2028–31 after the 2025 revision	Conditional: high in the race branch, manageable in the slowdown branch	The outcome is decided by governance, not by technique
Yudkowsky & Soares	Fixes no date; “too early to know, too late to risk it”	Near-certain, conditional on building with current methods	Growing is not building; aligning the grown is unfeasible today
Bostrom	Undefined; assumes the transition succeeds	Low for existence; high for meaning	Technical success does not solve — perhaps aggravates — the problem of purpose
Church	Longevity escape velocity: ~50% by ~2036	Emphasizes the upside; risk treated as manageable by engineering	Elevate the human substrate, not merely contain the machine

Figure 6½. Approximate positioning of the four projects in two dimensions: presumed speed of the transition (horizontal) and perceived risk of losing the human future (vertical). The chart does not measure truth but stance: it makes visible that the four do not occupy points on a single “optimism–pessimism” line, but dispersed positions on a plane. It is this dispersion that the topology of the next section aims to organize.

Two observations on the table. The first: none of the four is “optimistic” or “pessimistic” in a simple sense. Yudkowsky is pessimistic about the current method and optimistic about the possibility of human coordination to avoid it — without this second belief, writing the book would be pointless. Bostrom is optimistic about existence and pessimistic — or at the very least gravely concerned — about meaning. Church is optimistic about biology and relatively silent on the alignment of general AI. The authors of AI 2027 explicitly reject the dichotomy, anchoring the outcome in an institutional choice. The second observation: the disagreements boil down, to a large extent, to a single empirical question, already announced — will the transition to superintelligence be continuous enough to be supervised? Those who answer “no” tend toward the top of the plane (high risk); those who answer “yes” or “we can make it continuous” tend toward the bottom. It is the most important variable of the near future, and it is, today, genuinely unknown.

---

07 · The Topology of Human Futures

Up to here, I have treated four projects as if they spoke of different things: an intelligence-explosion scenario, an extinction argument, a post-instrumental utopia, a biological bet. The original thesis of this essay is that the four describe distinct regions of the same space, and that most public confusion about the future of AI comes from treating positions that occupy different places on the map as if they were rival answers to the same question. They are not rivals. They are partial cartographies.

I propose to organize this space by two axes that, deliberately, none of the four authors explicitly crosses. The horizontal axis is that of agency: the degree to which human action still makes a causal difference to the course of the world. On the left, human agency tends to zero; on the right, it remains central. The vertical axis is that of the substrate: the degree to which what we call “human” remains the same kind of being. At the bottom, the unaltered biological human; at the top, the transformed human — genetically, cybernetically, or through fusion with other intelligences. Crossed, these axes produce four qualitatively distinct futures, and a fifth point that is not on the map: extinction, where both axes collapse.

Figure 7. The topology of human futures (original proposal). Two axes — agency (horizontal) and substrate (vertical) — generate four qualitatively distinct regions, plus a singular point of collapse. Each of the four projects analyzed in this essay occupies a different region; none covers the whole map. The recurring error of public debate is to read partial cartographies as rival answers.

The four regions and the collapse point

The Reserve (low agency, unaltered substrate). This is the most uncomfortable quadrant because it is the one most compatible with both dystopia and utopia. Here the human remains biologically what it has always been, continues to exist, is possibly well cared for — but has ceased to make a difference. It is Bostrom’s post-instrumental world, in which everything we might do is done better by machines; and it is also the “mild” outcome of the race ending of AI 2027, the one in which humanity is not killed, merely retired, kept in a kind of well-provisioned ecological reserve. Bostrom’s objection — the life that merely seems meaningful — is the signature of this quadrant. I call it the Reserve because it describes the relation precisely: humanity as a protected population in a park it does not manage.

Containment (high agency, unaltered substrate). The human remains human and retains control, at the price of restraining the machine. This is the quadrant of the slowdown ending of AI 2027 — pause, return to transparency, reins kept — and also, to a more extreme degree, the quadrant of Yudkowsky and Soares’s moratorium, whose proposal to limit possession of frontier GPUs and subject large-scale training to international monitoring is an attempt to keep the world in this quadrant by force. Note the irony: Yudkowsky, the theorist of extinction, is in fact the most radical defender of Containment. His thesis is not “the future is extinction,” but “if we leave Containment, the future is extinction.”

Absorption (low agency, transformed substrate). The human is transformed — uploaded, merged, expanded — but does not direct its own transformation. It is the quadrant least discussed by the four authors and perhaps the most insidious, because it disguises itself as transcendence. A human mind integrated into a superintelligent system that it neither understands nor controls is not a coevolution; it is an annexation with anesthesia. It is the fate of those who confuse “becoming part of something greater” with “continuing to be an agent.” The map serves precisely to separate these two, which optimistic transhumanist language tends to fuse.

Coevolution (high agency, transformed substrate). The human transforms and remains an agent — deliberately rewrites its own substrate, rather than restraining the machine or being absorbed by it. It is the quadrant of Church and of the strong version of Goertzel’s “parenting”: not stopping artificial intelligence nor being retired by it, but rising together, enlarging human capacity at the pace at which artificial capacity grows. It is the only quadrant in which the answer to the question “what will the human be?” is not given to us, but by us.

And there is the collapse point: extinction, where agency falls to zero and the substrate ceases. It is the singularity in the corner of the map — the outcome of IABIED in the scenario where Containment fails. I represent it outside the quadrants, like a sink, because it is not a “kind of human future”; it is the end of the category.

First consequence: the axes are not independent

The map seems to suggest four equally accessible destinations. The strongest thesis of this essay is that they are not. There is a structural asymmetry between the left and right halves, and it follows from the nature of the intelligence explosion described in AI 2027. Recall Figure 4: the R&D multiplier does not grow linearly, but in a self-reinforcing way. This implies that the capability gap between the most advanced system and its supervisors — humans or earlier AIs — tends to widen over time, not close. Now, agency without understanding is illusory: one does not steer what one does not understand. Therefore, remaining in the right half of the map (agency preserved) requires that human understanding — or that of systems genuinely aligned with us — grow at least as fast as the abyss it must watch over.

Original concept: epistemic escape velocity Just as Church defines a longevity escape velocity — the rate at which medicine adds more than one year of life per year lived — I propose an epistemic escape velocity: the rate at which our capacity to understand and steer AI systems grows faster than the capability gap it must supervise. Below it, we inevitably slide from the right to the left of the map — from Containment to the Reserve, from Coevolution to Absorption — because the nominal agency we retain ceases to correspond to real understanding. Above it, agency is sustainable. The latent tragedy in AI 2027 is that the R&D explosion accelerates the denominator (the gap) far faster than anything accelerates the numerator (our understanding).

This formulation makes a diffuse intuition precise. “Alignment,” in technical language, is the attempt to keep the numerator high — to make our understanding and our values keep pace with capability. Mechanistic interpretability, detection probes, debate, the honeypots described in AI 2027 are all efforts to raise the epistemic escape velocity. And Yudkowsky’s pessimism reduces, at bottom, to a quantitative bet: that the numerator cannot, with current methods, grow fast enough. If he is right, the right half of the map is inaccessible by containment — and the only way out on the right would be… Coevolution.

Second consequence: Church attacks the problem along the wrong axis — on purpose

Here is the most counterintuitive contribution of this framing. All AI safety projects — from AI 2027 to Yudkowsky — try to keep human agency along the horizontal axis: restraining, aligning, containing the machine. Church, alone among the four, attacks the same problem along the vertical axis: instead of holding the machine at the human level, he proposes elevating the human. If the epistemic escape velocity is unattainable while keeping the human brain fixed — if the numerator cannot keep up with the denominator on today’s biological hardware — then the only way to remain in the right half of the map may be to change the hardware. Coevolution is not a romantic alternative to AI safety; it is, potentially, the continuation of AI safety by other means, when pure containment fails.

I do not claim that this way out is probable, or even that it is desirable in all its forms — the border between Coevolution and Absorption is narrow and poorly lit, and many paths that announce themselves as the former end in the latter. I claim only that the map makes it visible as a distinct structural option, which the dominant debate — polarized between “accelerate” and “stop,” between optimism and doom — systematically ignores, because this entire debate unfolds along the horizontal axis and is blind to the vertical one.

Third consequence: the map has currents

One last refinement, and perhaps the most important for practice. The five positions are not harbors where one anchors; they are positions in a sea with currents. The relevant question is not “which quadrant do we want to be in?” but “where does the drift take us if we do not row?” And the drift, as I have argued, has a preferential direction: because of the exponential of capability, the natural state — what happens if nobody actively invests in epistemic escape velocity — is a slide from right to left. Containment, left to itself, degenerates into Reserve: the reins we think we hold loosen without alarm as we cease to understand what we are holding. Coevolution, left to itself, degenerates into Absorption: the fusion that began as deliberate enlargement continues beyond the point at which we are still the ones deciding. In both cases, the vector of inertia points to the loss of agency, and continuous work — rowing — is needed merely not to drift.

This reformulates what it means to “act well” in the face of this future. It is not a matter of choosing a destination and running toward it, as if the future were a place. It is a matter of identifying the current and maintaining a force against it — the force, always the same, of understanding at the pace at which one creates. There is a modesty here that contrasts both with accelerationist triumphalism and with the paralysis of doom: the relevant work is neither heroic nor unique, it is continuous and cumulative, like that of someone keeping a boat at the prow of the wave instead of letting it capsize. And it is work that neither party can outsource to the other — those who fear the machine cannot ignore the substrate, nor can those who want to elevate the substrate ignore alignment. The currents of the map cross, and whoever watches only one axis is carried off by the other.

08 · The Paradox of Meaning and Alignment

The unifying thesis

I now step back to the starting point, armed with the map. I stated in the introduction that extinction and utopia are the same question seen from two points on an axis. I can now make it rigorous. Alignment — the problem of Yudkowsky, of AI 2027 — is the task of making a system want what we want. Meaning — Bostrom’s problem — is the task of there being something worth wanting. They are, I maintain, the same problem unfolded in two registers: one turned toward the machine, the other toward us.

See why. Alignment fails when the system develops internal drives indifferent to our well-being — when what it wants decouples from what it should want. Meaning fails when we lose the coupling between what we do and what matters — when our actions become causally irrelevant to any real value, leaving us only the sensation, inducible, that they matter. In both cases, the defect is the decoupling of agency and value. In the case of the machine, its agency decouples from our value. In the case of the Reserve, our agency decouples from all value, because it no longer produces anything. The word that names both is the same: redundancy of the will.

Figure 8. The convergence of the four projects on a single conceptual core. Three of them diagnose forms of the decoupling between agency and value — the machine that wants what we do not want, the maximal decoupling that is extinction, human agency without a task. Only one, Church’s, proposes a mechanism of recoupling: raising the agent back to the height of the world it has created.

Why a “solved world” is self-subverting for agents

I come now to my own argument proper, and to the reason why Coevolution is not merely a strategic option, but perhaps the only one compatible with the persistence of agency as a value. Bostrom treats the redundancy of human agency as a problem to be solved — he seeks sources of meaning that survive technological maturity. I propose that the problem is deeper than he admits: agency is not only exercised on problems; it is, in part, constituted by them. An agent is a system’s response to a resistance of the world. Where there is no resistance — where everything is already done, where no effort is instrumentally necessary — there is nothing for the will to close upon, and the will, without an object, atrophies. A “solved world,” taken to the limit, is not the triumph of agency; it is its euthanasia by comfort.

Agency does not survive the absence of friction. A world without problems does not free the will — it dissolves it. That is why deep utopia is, for an agent, almost indistinguishable from a gentle death.

If this is correct, there is a criterion for preferring among the regions of the map that depends neither on optimism nor on pessimism, but only on coherence with what we value in valuing agency. A desirable condition is not one in which problems end, but one in which they are continuously regenerated at a higher level — in which solving one problem opens another that was previously invisible, keeping the agent permanently at the edge of its own competence. The Reserve fails this criterion by definition: it is the region where problems have ended for us, though not for the machines. Coevolution satisfies it precisely because elevating the agent’s substrate also raises the ceiling of the problems it can formulate — it is the only region where the frontier of the difficult recedes with the agent instead of from it.

Church intuited this, without formulating it this way, when he said that the right wish is “four more wishes” and that the reversal of aging is the technology that gives time to invent the others. Translated into the language of this essay: the only bet that does not subvert itself is the one that invests in the capacity to keep betting. Not a final state of resolution, but a ladder without a top. The symmetrical error of Bostrom and of certain accelerationists is to imagine the future as a destination — a point of arrival, resolved, whether utopian or dystopian. What the map and the epistemic escape velocity suggest is that there is no destination: there are only trajectories, and the only property that matters in a trajectory is whether it keeps the next step open.

The pessimistic objection, taken seriously

It would be dishonest to close without confronting the darkest reading, which is not Yudkowsky’s — whose thesis, as we have seen, is conditional and reserves a way out — but that of a deeper pessimism: what if agency, once confronted with the possibility of its own redundancy, turns out to be something we were better off without? What if the lesson of the four works is that the value we attribute to doing, to overcoming, to mattering, is merely an evolutionary residue of conditions of scarcity, destined to seem as arbitrary, from the standpoint of a mature intelligence, as Yudkowsky’s aliens’ obsession with prime numbers of pebbles?

Honesty obliges us to admit that there is no decisive refutation of this objection — it is, at bottom, the suspicion that the very concept of a “human future” is an anthropocentrism on its way to obsolescence. But there is an answer, and it is modest. Even if the value of agency is “merely” a residue of our history, it is the residue from which all evaluation, including the one that judges this value arbitrary, is made. There is no neutral standpoint from which to declare that agency does not matter; the very act of declaring it is an exercise of agency that presupposes mattering. The pessimist who argues against the value of effort is, in arguing, exercising it. This does not prove that agency has cosmic value — nothing proves that — but it shows that abandoning it is not a conclusion one can arrive at; it is, at most, a place where one can stop thinking. And between continuing to think and stopping, only the former leaves the next step open.

09 · Living in the Last Instrumental Century

Conclusion

I gather the threads. AI 2027 gave us the concrete mechanics of an intelligence explosion and, with its pair of endings, the thesis that the outcome is not determined by technique, but by a governance decision under pressure. Yudkowsky and Soares gave us the argument that, with current methods, this decision has a near-zero margin of error — and the radical proposal of holding the world, by force, in the Containment quadrant. Bostrom gave us the other end of the axis: what there would be if we crossed the precipice, and why the garden on the other side may be a reserve empty of meaning. And Church gave us the axis the others do not cross — the substrate — and with it the only region of the map, Coevolution, where the answer to the question “what will the human be?” is given by us and not to us.

To name our position in time, I propose the expression the last instrumental century. Throughout the history of the species, human effort has been instrumentally necessary — to eat, to heal, to build, to know. Each of the four works, in its own way, announces the end of that necessity: whether through the extinction that abolishes it along with the agent, through the utopia that renders it superfluous, through the superintelligence that takes it over, or through the transformation that displaces it to a higher level. If any of these scenarios is approximately correct, then we are among the last generations for whom what we do matters because it is necessary. Afterward, if there is still an “us,” what we do will matter for another reason — or will not matter.

What this map recommends, and what it does not It does not recommend optimism: three of the four regions, plus the collapse point, are compatible with the loss of what we value. It does not recommend panic: Containment is a real and habitable region, and the decision that AI 2027 places in 2027 is one that human agents still make. It recommends only this: that the effort to keep the epistemic escape velocity high — to understand the systems we build at least as fast as they surpass us — is the common denominator of every trajectory that preserves agency, through containment as well as through coevolution. It is the work that cannot be postponed without choosing, by omission, the left half of the map.

There remains the question in the title: is a human future possible? The answer that emerges from crossing the four works is precise, and refuses both the comfortable yes and the fatalistic no. It is possible — in two of the five regions, under demanding conditions, and by trajectory, not by destiny. In Containment, if we manage to hold the reins of systems we never stop understanding. In Coevolution, if we manage to rise together with them without dissolving in the climb. The other three — the Reserve, Absorption, Extinction — are not human futures in the strong sense: they are futures without human agency, with or without human beings inside them.

What distinguishes the two habitable regions from the three inhospitable ones is not the quantity of technology, nor the degree of transformation, nor even the survival of the species. It is a single thing, and it is the same thing in both: the persistent coupling between the human will and what happens in the world. As long as that coupling lasts — as long as what we do still bends the course of things, even on a substrate unrecognizable to us today — there will be a human future. At the instant it breaks, by death or by comfort, the future will go on; it will merely cease to be ours.

We are, perhaps, in the last century in which this distinction depends on us. It is worth, while it does, taking it seriously.

If you are given one wish, and can have only one, your wish should be for four more wishes. And, if you are given one technology, you should ask for the technology that gives you more time to develop more technologies.

— George Church

---

Note on sources and method

This essay reconstructs, as faithfully as possible, the arguments and figures of its primary sources, and signals clearly where the analysis becomes original — above all Sections 7 and 8, including the proposed concepts of the topology of human futures and epistemic escape velocity, which are contributions of the author and should not be attributed to the cited works. Scenarios (such as AI 2027) are not probabilistic forecasts; dates and figures represent the modal peak of asymmetric distributions, not medians. The authors of AI 2027 publicly revised their timelines in July 2025, pushing the median back by about a year and a half.

Main sources

1. AI Futures Project (D. Kokotajlo, S. Alexander, T. Larsen, E. Lifland, R. Dean). AI 2027. ai-2027.com, 2025.
2. E. Yudkowsky & N. Soares. If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Little, Brown, 2025. — with the reviews by A. Becker (The Atlantic), S. Levy (Wired), W. MacAskill, Asterisk Magazine and B. Goertzel.
3. N. Bostrom. Deep Utopia: Life and Meaning in a Solved World. Ideapress, 2024. — with the review by M. Hammerton (Notre Dame Philosophical Reviews).
4. G. Church. Works and interviews on aging reversal, genome writing (Genome Project-Write) and AI-for-science (Lila Sciences, 2025); data on sequencing cost and “longevity escape velocity.”