Surely this analysis has been done before in the bowels of LessWrong. I gave up searching because it was too irritating trying to disambiguate the terms survival and hazard in the technical mathematical sense that I needed, against the more colloquial sense that they are used in AI safety discourse. Feel free to point me to prior work in the comments.
AI usage: much! So many diagrams and calculations were assisted by AI. There are likely to be some residual AI text chunks in this prose but it should be mostly mine by now.
Figure 1
I recently had a discussion where the contested point was whether getting more compute was, ceterus paribus a “good” or “bad” thing for AI safety research. What follows is my attempt to formalize Goodness and Badness for AI Dooms in terms of hazard models, which seems to me a nice way of making our assumptions and disagreements explicit, and scenario-independent besides. Slaughterbots? Matrix-style VR capture? Engineered plagues? Meatballs falling from the sky? Doesn’t matter. If we disagree about what ASI disasters or utopias look like, but agree that compute is a forcing term, then we can still talk about things like pauses, slowdowns, and accelerations without sweating the details.
The model setup is as follows. I assume two things can happen — an AI catastrophe (“doom”), or a breakthrough that guarantees alignment (“deliverance”). Both become more likely as more compute accumulates in the world, but they draw on different, potentially overlapping, pools of compute. The question becomes: given a trajectory of total compute and a policy for splitting it between capability and safety, what is the probability that doom arrives before deliverance?
The machinery I use is survival analysis of competing risks — the branch of statistics built for this kind of “which stochastic event comes first?” problem. I am always faintly annoyed when people don’t use this formalism, which is simple and elegant for reasoning about the timing of stochastic events, and really well developed by the insurance industry and just generally the obviously correct baseline against which any other model should need to prove itself.
A hazard rate is the instantaneous probability per unit time that an event fires right now, given that it hasn’t fired yet.
A survival function\(S(t)\) is the probability that nothing has happened yet by time \(t\). It starts at 1 and decays toward 0 as hazard accumulates.
Competing risks means two or more events are racing to fire first. We observe whichever wins; the loser is never realized.
If you are coming from LessWrong, some terms might be handy: the safety fraction \(\alpha\) is a compute-denominated cousin of the “alignment tax”, which usually means the capability cost of aligning one system rather than a fraction of the world’s compute, and “P (doom)” is here derived from a competing-risks integral.
I distinguish between the rate at which compute is performed and the aggregate of computation that has been done: throughput versus spend.
Let \(c(t)\) denote the compute throughput — the FLOP/s of AI-related computation happening in the world at time \(t \geq 0\). This is capacity: hardware, data centres, and investment. It is non-negative and right-continuous. I make no distinction between training and inference — \(c(t)\) is unstructured compute, all of it. The more of it that is running, the more things are happening; I don’t model the internal structure of what kinds of workloads are running for this level of analysis.
The compute spend — total FLOPs performed by time \(t\) — is
\[
\mathcal{C}(t) = \int_0^t c(u)\, du.
\]
A fraction \(\alpha(t) \in [0,1]\) of the throughput goes to safety and the rest to capability:
Capability throughput\(c_c(t) = (1-\alpha(t))\, c(t)\): compute aimed at expanding what AI systems can do.
Safety throughput\(c_s(t) = \alpha(t)\, c(t)\): compute focused on ensuring AI systems behave safely.
The spend in each pool is then
\[
\mathcal{C}_c(t) = \int_0^t (1-\alpha(u))\, c(u)\, du, \qquad \mathcal{C}_s(t) = \int_0^t \alpha(u)\, c(u)\, du,
\]
with \(\mathcal{C}_c(t) + \mathcal{C}_s(t) = \mathcal{C}(t)\) at all times. To calibrate this to the actually-existing world, where a lot of compute has already been spent, we would start the clock mid-history with legacy spends \(\mathcal{C}_c(0) = \mathcal{C}_{c,0}\) and \(\mathcal{C}_s(0) = \mathcal{C}_{s,0}\) added to the integrals above. For simplicity I do not do that yet, but I do return to it later.
The spend/throughput distinction seems important because we, or at least I, imagine that capabilities persist. We can’t necessarily “un-train” a frontier model by switching off the data centre; the weights, the papers, the algorithmic insights are already in the world, at least while DeepSeek et al are out there reverse-engineering the western labs and adding their own tricks. Safety progress persists too, ideally — proved theorems, verified architectures, and alignment techniques don’t evaporate (unless these also languish in Anthropic’s file drawers). So the timing of doom should depend on the capability spend \(\mathcal{C}_c(t)\), not the instantaneous throughput \(c_c(t)\). Throughput sets how fast the spend grows; the spend sets the hazard. Doubling \(c(t)\) doesn’t double the hazard at time \(t\), but it does reach any given spend sooner. Empirically we know (Sweeney and Sterman 2000) that people have really bad intuitions about such stock-and-flow systems.
The function \(\alpha(\cdot)\) is the allocation control, which a civilization sets by doing whatever governance is necessary (tax incentives, setting research agendas, etc.). The trajectory \(c(t)\) is the pacing control: exogenous in the base model, but it is what compute governance acts on — licensing or reporting thresholds for large training runs, chip export controls, caps on data-centre build-out, nuking compute-heavy facilities. So the model has two controls, allocation and pacing, and the pause is an intervention on the second rather than a third control. There is a lot of texture in those choices, and some deep governance problems, which I will skate over here while hoping that my colleagues in the Credible Theory of Change department have this one in the bag.
2 Two competing events
I model doom and deliverance as the first arrivals of inhomogeneous point processes — random events whose chance of firing at any moment depends on how much compute has been spent so far:
Doom (\(T_d\))
An X-risk catastrophe happens. This is an irreversible absorbing state, because everyone is dead and rendered into computronium or paperclips or whatever.
Deliverance (\(T_s\))
We achieve guaranteed alignment — a state after which X-risk from AI is effectively zero, and some kind of end-game high score is unlocked.
Each event has a latent arrival time — \(T_d\) and \(T_s\) are the times at which doom and deliverance would fire if nothing else intervened. But the race ends at \(T^* = \min(T_d, T_s)\): one of them fires first, or neither fires within any horizon we care about. When I write “\(P(\text{doom})\)” below, I mean the outcome “doom fires first” — i.e. \(P(T_d < T_s)\) — not the marginal probability that the doom process would eventually fire in isolation. These coincide only if no deliverance is possible.
Each event has a hazard rate (instantaneous arrival intensity given that neither event has yet occurred — i.e. while we are still in limbo) that depends on the spend in its respective pool:
where \(g, h : [0, \infty) \to [0, \infty)\) are monotonically non-decreasing, with \(g(0) = 0\) and \(h(0) = 0\), or near enough in the sigmoidal case below. Monotonicity says that more capability spend never lowers the catastrophe hazard, and more safety spend never lowers the alignment-breakthrough hazard.
Monotonicity is agnostic about the shape of the response — linear, concave, convex, sigmoidal. That shape, for \(g\) and for \(h\), is where I think the non-trivial disagreements in AI safety discourse live.
Code
x = np.linspace(0, 10, 300)scenarios = [ ("Linear", g_lin(x), h_lin(x)), ("Convex-doom (g convex, h concave)", g_pess(x), h_pess(x)), ("Concave-doom (g concave, h convex)", g_opt(x), h_opt(x)), ("Overhang (sigmoidal)", g_sig(x), h_sig(x)),]fig = go.Figure()for name, gv, hv in scenarios: vis = name.startswith("Convex-doom") fig.add_trace(go.Scatter(x=x, y=gv, name="g(x) doom", line=dict(color=C_DOOM, width=3), visible=vis, hovertemplate="%{y:.2f}")) fig.add_trace(go.Scatter(x=x, y=hv, name="h(x) safety", line=dict(color=C_SALV, width=3), visible=vis, hovertemplate="%{y:.2f}"))buttons = []for i, (name, _, _) inenumerate(scenarios): vis = [False] * (2*len(scenarios)) vis[2*i] =True; vis[2*i+1] =True buttons.append(dict(label=name, method="update", args=[{"visible": vis}]))fig.update_layout(**LAYOUT, updatemenus=[dict(type="buttons", direction="down", x=1.0, xanchor="left", y=1.0, buttons=buttons, bgcolor="white")], xaxis_title="Compute spend 𝒞", yaxis_title="Hazard rate", height=420,)# The buttons occupy the top-right corner, where the legend would otherwise sit;# every scenario's curves rise from the origin, so the top-left is empty.fig.update_layout(legend=dict(x=0.02, y=0.98, xanchor="left", yanchor="top"))fig.show()
Figure 2: Response functions g (doom hazard, red) and h (safety hazard, blue) as functions of spend under four shape assumptions. Use the buttons to switch scenarios.
3 Three outcomes of the race
The probability that the race remains in limbo at time \(t\) — because neither event has occurred — is the overall survival function, the survival of the minimum:
In words: the probability of still being in limbo decays exponentially as the hazard accumulates. Here \(\Lambda_d(t) = \int_0^t g(\mathcal{C}_c(u))\, du\) and \(\Lambda_s(t) = \int_0^t h(\mathcal{C}_s(u))\, du\) are cumulative hazard functions. Note the nested structure: each cumulative hazard is an integral over time of a function that itself contains an integral over time.
In the model \(\mathcal{C}(t)\) grows without bound, and so does the spend in each pool as long as \(\alpha\) stays away from 0 and 1, so whichever of \(g\), \(h\) is not identically zero is eventually bounded below by a positive constant, \(\Lambda_d(t) + \Lambda_s(t) \to \infty\), and \(P(\text{limbo}) = 0\): the race always resolves eventually. But “eventually” can be a very long time. Over any finite horizon there is residual probability mass on “neither yet” — the green curve \(S(t)\) in the plots below — and a world where \(S(t)\) stays large at human-relevant timescales is one where we just muddle through indefinitely, which is arguably closer to the standard baseline expectation than either doom or deliverance.
At each instant \(t\), while we’re still in limbo, the probability that some event fires within \([t, t+dt)\) is \((\lambda_d(t) + \lambda_s(t))\, dt\), and the conditional probability that the firing event is doom rather than deliverance is
which decomposes into, “probability we’re still waiting at time \(t\)” × “probability something happens right now” × “probability that the something is doom rather than deliverance.” The middle two factors, \([\lambda_d(t) + \lambda_s(t)]\, S(t)\), are the probability density of the resolution time \(T^*\), so the integral is an expectation:
Doom probability is the conditional ratio \(\pi_d\) averaged over the moment the race resolves. For constant \(\alpha\), both hazards are functions of the total spend \(\mathcal{C}(t)\) alone, so we can equally write \(P(\text{doom}) = \mathbb{E}[\pi_d(\mathcal{C}(T^*))]\), an average over the spend at which the race resolves. Everything below is a corollary of this identity. The split \(\alpha\) sets the shape of \(\pi_d\) as a function of spend. The distribution of the resolution spend \(\mathcal{C}(T^*)\) is set by the growth rate of \(c(t)\), the overall scale of the hazards, any pause, and also by \(\alpha\), since the split changes the total hazard at each spend too. So pacing can only move the distribution of \(\mathcal{C}(T^*)\) that the expectation is taken over; allocation moves both that distribution and the \(\pi_d\) being averaged.
Figure 3: Resolution of the race with superlinear doom risk (g(𝒞) = 0.005𝒞²) and linear safety progress (h(𝒞) = 0.015𝒞). Compute throughput c(t) = e^{0.1t}, α = 0.5. The green curve (right axis) is S(t), the probability of limbo — neither doom nor deliverance has yet occurred. The shaded areas (left axis) show doom density λ_d(t)S(t) (red) and deliverance density λ_s(t)S(t) (blue). Their integrals give P(doom) and P(deliverance); the residual 1 − P(doom) − P(deliverance) is P(limbo) over this time window.
4 The constant-hazard-ratio case
Consider an ultra-simple sanity check. If the ratio \(\pi_d(t) \equiv \pi_d\) is constant over time, then \(P(\text{doom}) = \pi_d\) regardless of the compute trajectory — a standard competing-risks identity. That happens when \(g\) and \(h\) are linear and \(\alpha\) is constant: if \(g(x) = ax\) and \(h(x) = bx\), then \(\mathcal{C}_c(t) = (1-\alpha)\mathcal{C}(t)\) and \(\mathcal{C}_s(t) = \alpha\mathcal{C}(t)\), so
and the \(\mathcal{C}(t)\) cancels. The probability of a doom outcome \(P(\text{doom})\) depends only on \(\alpha\) and the ratio \(a/b\), not on the throughput \(c(t)\) or how fast it grows. This is the regime in which the speed of progress doesn’t matter — only the split between capability and safety compute determines our fate. If you’ve ever heard someone say “it doesn’t matter how fast AI progresses, only whether we invest enough in safety” — this is the (very specific) model in which that’s true. It is the implicit model whenever we reason about a safety budget as a fraction without asking where on the curve we are.
It’s also not very plausible, in my opinion. Let’s get more realistically complicated.
5 Sweeping the split
With non-linear response functions this invariance breaks down: \(P(\text{doom})\) depends on the full trajectory \(c(t)\), because the time spent at each level of spend determines how much hazard accumulates there. The plot below sweeps \(\alpha\) under exponential growth \(c(t) = e^{rt}\) at three growth rates.
Code
alphas = np.linspace(0.01, 0.99, 80)t_sim = np.linspace(0, 100, 3000)def compute_pdoom(g_fn, h_fn, alpha_val, r, Cc0=0.0, Cs0=0.0, leaky=False):"""P(doom) by integration over a trajectory, from legacy spends Cc0, Cs0. leaky=True feeds g the total spend instead of the capability spend.""" c_rate = np.exp(r * t_sim) Cc = Cc0 + cumulative_trapezoid((1- alpha_val) * c_rate, t_sim, initial=0) Cs = Cs0 + cumulative_trapezoid(alpha_val * c_rate, t_sim, initial=0) ld = g_fn(Cc + Cs if leaky else Cc) ls = h_fn(Cs) cumhaz = cumulative_trapezoid(ld + ls, t_sim, initial=0) S = np.exp(-cumhaz) doom_density = ld * Sreturn np.trapezoid(doom_density, t_sim)curves = [ ("Linear", g_lin, h_lin, C_NEUT), ("Convex-doom", g_pess, h_pess, C_DOOM), ("Concave-doom", g_opt, h_opt, C_SALV), ("Overhang", g_sig, h_sig, C_RUIN),]fig = go.Figure()for name, gf, hf, col in curves:for r, dash, suffix in [(0.3, "solid", "fast r=0.3"), (0.05, "dash", "slow r=0.05"), (0.0, "dashdot", "buildout stop r=0")]: pr = np.array([compute_pdoom(gf, hf, a, r) for a in alphas]) fig.add_trace(go.Scatter(x=alphas, y=pr, name=f"{name} ({suffix})", line=dict(color=col, width=2.5, dash=dash), hovertemplate="α=%{x:.2f} P(doom)=%{y:.3f}"))fig.add_hline(y=0.5, line_dash="dot", line_color="#bbb", annotation_text="P(doom) = ½", annotation_position="bottom right")fig.update_layout(**LAYOUT, xaxis_title="Safety fraction α", yaxis_title="P(doom)", height=480,)fig.show()
Figure 4: Probability of doom as a function of safety fraction α, computed by integration over an exponential compute trajectory. Solid lines: fast growth (r = 0.3); dashed: slow growth (r = 0.05); dash-dot: no growth (r = 0), throughput held at its starting value from the outset. For the linear case (grey), the curves coincide — growth rate doesn’t matter. For non-linear responses, faster growth shifts the doom probability because it changes how much time the system spends in different hazard regimes.
Code
# Quantities quoted in the prose below, so they cannot drift from the plots.def resolution_stats(g_fn, h_fn, alpha_val, r, scale=1.0):"""Median resolution spend 𝒞(T*) and time T* for one trajectory.""" c_rate = np.exp(r * t_sim) C = cumulative_trapezoid(c_rate, t_sim, initial=0) Cc = cumulative_trapezoid((1- alpha_val) * c_rate, t_sim, initial=0) Cs = cumulative_trapezoid(alpha_val * c_rate, t_sim, initial=0) lam = scale * (g_fn(Cc) + h_fn(Cs)) F =1- np.exp(-cumulative_trapezoid(lam, t_sim, initial=0)) i =min(np.searchsorted(F, 0.5), len(t_sim) -1)return C[i], t_sim[i]C_med, t_med = resolution_stats(g_pess, h_pess, 0.5, 0.1)C_med_slow, t_med_slow = resolution_stats(g_pess, h_pess, 0.5, 0.1, scale=0.01)C_med_lin, _ = resolution_stats(g_lin, h_lin, 0.5, 0.05)ov_slow_half = compute_pdoom(g_sig, h_sig, 0.5, 0.05)ov_slow_09 = compute_pdoom(g_sig, h_sig, 0.9, 0.05)_ov_fast = np.array([compute_pdoom(g_sig, h_sig, a, 0.3) for a in alphas])_ov_slow = np.array([compute_pdoom(g_sig, h_sig, a, 0.05) for a in alphas])ov_cross = alphas[np.where(np.diff(np.sign(_ov_fast - _ov_slow)) !=0)[0][-1]]ov_alpha_star = X0_SAFE / (X0_SAFE + X0_DOOM) # safety's switch flips first iff alpha exceeds this
NoteScale, and muddling through
The race ends at \(T^* = \min(T_d, T_s)\), at total spend \(\mathcal{C}(T^*)\), and the shapes of \(g\) and \(h\) matter only over the spends below that, the functions never being evaluated outside that (random) range. Multiplying \(g\) and \(h\) by a common constant leaves their shapes the same but sets how much compute is spent before the race resolves: larger hazards end it in expectation sooner, at a smaller spend. The convex and concave responses used here, \(x^2\) and \(\sqrt{x}\), cross at \(x = 1\), so calling the doom hazard “convex” says something about the race only if \(\mathcal{C}(T^*) \gg 1\) with high probability. I chose the constants so that the median resolution spend lands at around 11 units, past the crossover. That is monopoly money; an empirically calibrated model would use \(\mathcal{C}(T^*)\) in GW-years or FLOPs and scale every unit accordingly.
The same constant sets how long the muddling through lasts. Shrink the coefficients by a hundred and the convex-doom race takes about 3 times as long, but resolves at a spend about 8 times larger, 82 units rather than 11. Near the origin the convex and concave curves are hard to tell apart; far out, the asymptotic shape dominates, and \(\pi_d\) hovers near 1 in the convex-doom world. So while muddling through, our fate is decided by the tails of the response functions, which are the part we can observe least, and every trajectory looks like muddling from inside at first.
6 Some scenarios
Each choice of \(g\) and \(h\) is a world, and in each world the sweep above tells us which of the two controls, allocation \(\alpha\) and pacing \(c(t)\), does anything. The four shape assumptions in the plot are toy versions of familiar worldviews.
The linear scenario. Both \(g\) and \(h\) are linear, the constant-hazard-ratio case: \(\pi_d\) is the same at every spend, so the average over \(\mathcal{C}(T^*)\) is trivial, speed is irrelevant and only \(\alpha\) matters.
Convex-doom. Convex \(g\), concave \(h\), “fast takeoff, hard alignment.” Every unit of capability spend adds more doom hazard than the last, and every unit of safety spend adds less deliverance hazard than the last. While both spends are small the safety curve can be the higher of the two, but the doom curve keeps steepening, overtakes it at some spend and pulls away, and from there \(\pi_d\) climbs towards 1. Run the race far enough and doom wins no matter how we split the compute, so the game is getting to deliverance before the overtaking point. Faster growth means more compute is spent before the race is done, so the race is more likely to end past that point; in the sweep that is the solid fast-growth line for this scenario, placed just above the dashed slow one, and the gap is widest when the overtaking point falls between where the slow race ends and where the fast one does. So “fast takeoff, hard alignment” is the claim that doom’s curve overtakes safety’s before the race is over.
Concave-doom. Concave \(g\), convex \(h\), “slow takeoff, tractable alignment”, the optimistic mirror image of the previous one. The doom hazard saturates and the deliverance hazard is superlinear once we invest enough, so \(\pi_d\) falls with spend and faster growth helps. This generates “we just need to invest enough in alignment” scenarios where the alignment tax is finite and worth paying.
Compute overhang. Sigmoidal \(g\) and \(h\) with different inflection points, “the capability overhang.” Each hazard here is more like a switch than a slope: near off until its spend crosses a threshold, then on. Doom’s switch is at a certain capability spend and safety’s at a certain safety spend, and the two need not coincide. Once the first flips, one hazard is running and the other is still off. If doom’s flipped first, \(\pi_d \approx 1\) for that stretch, the dangerous window; if safety’s did, there is a deliverance window (yay). With fixed \(g\), \(h\), the split decides which flips first: in the plots doom’s threshold is at 3 units of capability spend and safety’s at 7 units of safety spend, so with spends \((1-\alpha)\mathcal{C}\) and \(\alpha\mathcal{C}\), safety’s flips first only when \(\alpha\) exceeds 0.7. The growth rate decides how long the stretch lasts. Under slow growth it is long, the first hazard has the race almost to itself, and \(P(\text{doom})\) is a cliff in \(\alpha\) rather than a slope: 0.96 at \(\alpha = 0.5\), 0.04 at \(\alpha = 0.9\). Under fast growth the second switch flips before the race has resolved, both hazards run together, and the cliff softens into a slope. So speed helps at low \(\alpha\), where doom flips first and faster growth shortens the dangerous window, and hurts at high \(\alpha\), where safety flips first and faster growth shortens its head start. In the sweep that reversal is where the fast and slow lines cross, at \(\alpha \approx\) 0.73, which is also where the cliff is, since both are set by which switch flips first. The low-\(\alpha\) case is my version of the compute-overhang argument against pausing: slowing down while doom’s switch is on and safety’s is off only prolongs the time spent at \(\pi_d \approx 1\). NB that same argument is lethal in the convex-doom scenario, where no safety switch is coming and faster growth hurts at every \(\alpha\), and from inside the dangerous window the two look the same, doom hazard climbing and safety hazard flat, until safety’s switch flips or doesn’t.
The pause. A “compute pause” here means a stop to buildout: from some \(t_0\) the throughput \(c(t)\) holds at its current value instead of growing. The dash-dot curves in the sweep are the limiting case \(t_0 = 0\), a world whose throughput never grew; a pause imposed after compute has already been spent is a legacy question, and I come to it below. Under a buildout stop the spends keep growing, linearly rather than exponentially, so the hazards keep growing too and the race still resolves eventually, at a smaller spend than it otherwise would. A pause therefore helps where \(\pi_d\) rises with spend and hurts where it falls: the implicit assumption behind calls for a compute pause is the convex-doom world, and the overhang argument against pausing is the low-\(\alpha\) overhang world. A pause does more than this only if we spend it raising \(\alpha\).
7 What if safety compute is also capabilities compute?
In the base model the doom hazard depends only on capability spend \(\mathcal{C}_c\) and the deliverance hazard only on safety spend \(\mathcal{C}_s\). That quarantine is itself optimistic. Suppose safety research requires running large models, probing their behaviour, red-teaming, and training oversight systems, all of which also advance capabilities as a side effect; at the limit, all compute advances capabilities regardless of intent, because “capabilities are fungible”. The leaky model makes the doom hazard a function of total spend,
so that \(\alpha\) cannot lower the doom hazard at all, only raise the competing one. In the linear case, the ratio is \(\pi_d = a/(a + b\alpha)\), against the separable model’s \(a(1-\alpha)/(a(1-\alpha) + b\alpha)\). With \(a = b\), the separable model reaches even odds at \(\alpha = 0.5\) and the leaky one only at \(\alpha = 1\); even there, every step toward deliverance still drags doom closer.
Code
fig = go.Figure()leak_gap = {} # leaky minus separable P(doom) at α = 0.9, quoted below_i09 = np.argmin(np.abs(alphas -0.9))for g_fn, h_fn, name, col in [ (g_lin, h_lin, "Linear", C_NEUT), (g_pess, h_pess, "Convex-doom", C_DOOM), (g_opt, h_opt, "Concave-doom", C_SALV),]: r =0.1 pd_sep = np.array([compute_pdoom(g_fn, h_fn, a, r) for a in alphas]) pd_leak = np.array([compute_pdoom(g_fn, h_fn, a, r, leaky=True) for a in alphas]) leak_gap[name] = pd_leak[_i09] - pd_sep[_i09] fig.add_trace(go.Scatter(x=alphas, y=pd_sep, name=f"{name} (separable)", line=dict(color=col, width=2, dash="dash"), hovertemplate="α=%{x:.2f} P(doom)=%{y:.3f}")) fig.add_trace(go.Scatter(x=alphas, y=pd_leak, name=f"{name} (leaky)", line=dict(color=col, width=3), hovertemplate="α=%{x:.2f} P(doom)=%{y:.3f}"))fig.add_hline(y=0.5, line_dash="dot", line_color="#bbb", annotation_text="P(doom) = ½", annotation_position="bottom right")fig.update_layout(**LAYOUT, xaxis_title="Safety fraction α", yaxis_title="P(doom)", height=480,)fig.show()
Figure 5: P(doom) vs safety fraction α, comparing the separable model (dashed) with the leaky model (solid), where the doom hazard depends on total spend. The gap between them is the cost of the leak. Compute throughput c(t) = e^{0.1t}.
The gap between the dashed and solid curves is the cost of the leak. As expected, the gap grows with \(\alpha\). At \(\alpha = 0.1\) the curves nearly coincide; at \(\alpha = 0.9\) they differ by 0.49 in the linear world and 0.36 in the convex-doom world, where the superlinear doom hazard is driven by total spend. In the concave-doom world the leak hardly matters, because deliverance wins the race regardless. Which is to say, the leak matters most to the people who want to spend the most on safety, and the case that “safety research needs frontier models” is also the case that safety spend counts toward the doom hazard. The leak also makes a pause the only intervention that touches the doom hazard at all, since \(\alpha\) cannot, though whether that moves \(P(\text{doom})\) is again a question of shape.
8 Starting in the middle
The sweeps above all start from zero compute, which is pedagogically convenient but not realistic because there is clearly a bunch of compute out there already (I say, asking Claude to deploy 8 H100s to check my spelling). The legacy spend that brought us here was made at, presumably, a negligible safety fraction, so it is nearly all capability spend, and its hazard \(g(\mathcal{C}_{c,0})\) has been running for a while already. It is reasonable to apply the split only to new compute; it can dilute the legacy spend but not remove it.
In the linear world the conditional doom probability becomes
which starts at the legacy ratio, nearly all doom, and drifts toward the constant \(a(1-\alpha)/(a(1-\alpha) + b\alpha)\) as new compute swamps the old. How fast it drifts depends on \(c(t)\), so the growth rate now matters even in the linear world: at a fixed split, faster growth lowers \(P(\text{doom})\), because we spend less time at the legacy ratio and accumulate less doom hazard while there.
Code
fig = go.Figure()for Cc0, col in [(0, C_NEUT), (5, C_RUIN), (20, C_DOOM)]:for r, dash, suffix in [(0.3, "solid", "fast r=0.3"), (0.05, "dash", "slow r=0.05")]: pr = np.array([compute_pdoom(g_lin, h_lin, a, r, Cc0=Cc0) for a in alphas])if Cc0 ==20and r ==0.05: legacy_hi, legacy_lo = pr.max(), pr.min() # quoted below fig.add_trace(go.Scatter(x=alphas, y=pr, name=f"𝒞_c,0 = {Cc0} ({suffix})", line=dict(color=col, width=2.5, dash=dash), hovertemplate="α=%{x:.2f} P(doom)=%{y:.3f}"))fig.add_hline(y=0.5, line_dash="dot", line_color="#bbb", annotation_text="P(doom) = ½", annotation_position="bottom right")fig.update_layout(**LAYOUT, xaxis_title="Safety fraction α", yaxis_title="P(doom)", height=480,)fig.show()
Figure 6: P(doom) vs safety fraction α in the linear world, starting from a legacy capability spend 𝒞_{c,0} of 0, 5 or 20 units and no legacy safety spend. Solid: fast growth (r = 0.3); dashed: slow growth (r = 0.05). With no legacy spend the two growth rates coincide; with one, faster growth dilutes it sooner and the curves separate.
I chose the values 5 and 20 relative to the model’s scale: starting from zero, the linear race resolves at a spend of roughly 12 units, so the red curves describe a world that has already spent more on capability than the whole race would otherwise take to resolve. At that ratio of old to new, the whole sweep of \(\alpha\) under slow growth moves \(P(\text{doom})\) only from 1.00 to 0.92, from near-certain to slightly less certain. Whether the split matters is a question about the ratio of legacy spend to \(\mathcal{C}(T^*)\), and I don’t know that ratio for the world, only that it is not zero. Legacy spend also pulls the speed verdict the opposite way from convexity: legacy makes faster growth good, because new spend at the current, better split swamps it sooner, and a convex \(g\) makes it bad, because the race resolves further out along the curve. Which wins is a question about the same two numbers, the legacy ratio and the overtaking point.
9 What did we just learn?
Whether more compute is good or bad for safety is not, I argue, a well-posed question, but a more explicit model of how doom and deliverance would actually happen gives us some that are:
What are the local shapes of \(g\) and \(h\) over the spends the race passes through before it resolves?
How much compute leaks between the two pools, in each direction?
How large are the legacy spends relative to \(\mathcal{C}(T^*)\)?
This is, I think, the actual content of the disagreement I started with. The people who say more compute is good believe something like “the race will be over before doom’s curve overtakes safety’s”, or that “the safety leak is small”; the people who say it is bad believe the reverse.
I care about that, because each of those quantities is at least in principle estimable, and they give us a way to quantify a “P(doom)” over some interventions. Evals that track how dangerous capabilities scale with training compute are the nearest thing we have to an estimate of the local shape of \(g\). Whether safety techniques transfer across model scales is an estimate of the shape of \(h\). Whether interpretability results get folded into the next capabilities run is an estimate of the leak. None of these are easy to measure, mind you, but they are the kind of thing we could be wrong about in a checkable way—cruxes—and it is not obvious to me that the discourse has disclosed as many such cruxes as it should.
The other thing the model gives us is a sharper vocabulary for policy interventions. “Pause” and “invest in safety” are not rival strategies; they act on different inputs to the hazard functions, and we can more easily understand how they might complement or substitute for one another in the production function of continued existence by spelling them out. Anyone who advocates one intervention in all worlds is implicitly claiming to know which world we are in, and I would like to see that claim made explicit.
10 What to model next
This framework is minimal and stylized. Some important aspects it ignores:
Multiple actors. There is no game theory in this model — just a single planner choosing \(\alpha\). In practice, the allocation is the outcome of many actors with misaligned incentives.
More complicated hazards than pure time integration. Above I assume \[
\mathcal{C}(t) = \int_0^t c(u)\, du.
\] which doesn’t seem crazy to me, but in fact we could imagine that the doom process involves both the compute that went before and the compute currently being done. \[
\mathcal{C}(t) = \gamma \int_0^t c(u)\, du + \delta c(t).
\] Here, \(\gamma\) and \(\delta\) are parameters that determine the relative importance of past versus current compute in driving the doom hazard. I would chase this version to make the model properly robust.
The possibility of partial doom and incremental deliverance. Both events are modelled as discrete point arrivals — single moments at which the state transitions irreversibly. This is reasonable for doom (a single catastrophe), but strange for deliverance. Real alignment progress is incremental: better interpretability, verified properties, scalable oversight, each partially reducing risk. A more plausible model would replace the deliverance point process with one whose arrivals down-modulate the doom hazard rate — each safety milestone reduces \(g\) rather than ending the race outright. We could also model incremental doom — each catastrophe raises the baseline risk, although that feels less natural. If we’re worried about bad-but-not-doom events, we’d probably move to some continuous badness index, like “dollar value of harm” or “number of lives lost”, rather than a binary doom/deliverance outcome.
Optimal control. We haven’t solved for the optimal \(\alpha(\cdot)\). This is a dynamic optimal control problem, because \(\alpha(t)\) affects the spends \(\mathcal{C}_c(u)\) and \(\mathcal{C}_s(u)\) at all future times \(u > t\). That sounds fun, but probably not worth investigating because even if the model were true we wouldn’t know the response functions well enough to solve it, and even if we knew how to solve it, I cannot imagine us coordinating to implement that solution.
Granular allocation of compute to many different teams or ideas with different safety/capability profiles, rather than a single aggregate \(\alpha\). People have made the case to me that this matters. I think we might be able to produce a more granular model by allocating compute to buckets via some kind of stick-breaking process, then taking the max hazard? Definitely out of scope for this post, but maybe worth exploring in the future.
Granular allocation of compute to “training” or “inference”. Deliberately not explored here because I personally don’t think that distinction will survive for long in even the status quo world, let alone the world where continual inference-time learning is incentivised by training bans. But it would not be such a weird distinction to make in a more detailed model if that kind of thing floats your boat.
