I've noticed two clusters of thought in how people expect a global AI pause to be shaped:
An incrementally-shaped global pause is both more likely and more tractable than the single-shot global pause. This is largely because the third-party ecosystem needs practice building trust, technical competency, and general experience doing lower-scale assessments before a global pause can be competently pulled off.
I also think that the concept of a single-shot global pause is an unhelpful red herring that people often implicitly use to discredit the possibility of a global pause in general.
If a single-shot global pause is to happen, I think this would require substantial government buy-in, but it’s hard to build that sort of buy-in without precedent of similar things succeeding in the past. And if we just hack together a single-shot global pause without any precedent of similar things succeeding in the past, the chance that the pause falls apart due to defection, incompetence, and mistrust will be high.
The fundamental building block of an AI pause is that party A (a company or a country) can make a claim about sacrificing capability advancement for safety, and that party B can believe that claim. This is a public good, both between US AGI companies, and between countries, that is currently very scarce. Building more of this public good would generally make a pause much more likely to happen.
I don't just mean trust in the sense of "party B doesn't think party A is blatantly lying," but also in the broader sense that party A isn't omitting critical information or misinterpreting evidence.
Take for example the multiple cases where AGI companies have, in 2026, made statements to the effect that "we paced some of our capabilities for some safety!" These claims have been underspecified, impossible to verify from the outside, and have often been silently disbelieved by much of the industry. This air of distrust has made it much harder for companies to engage in tit-for-tat strategies where companies can become safer/slower as other companies become safer/slower.
Currently, a lot of people at the frontier AGI companies would prefer a state where AI progress is slower across the entire industry, but they have no way of trusting each other to actually uphold their commitments. We need to enter a different state where, instead of general distrust, parties can sacrifice capabilities for safety in a way that other parties actually believe.
Even small engagements between third parties and AGI companies can meaningfully improve trust between companies. Imagine if the companies that made claims about sacrificing capabilities for safety in 2026 actually had third parties verify those claims to each other. This would have made those sacrifices much more impactful towards incentivizing other companies to follow suit.
You might say "But verifying those claims in 2026 would have been very difficult and would have cost too many resources of the third parties and the AGI companies! We don't have the science or hardware to verify these claims now!" I don't believe this at all.
Let's assume that companies aren’t actively deceiving third-party evaluators by e.g. selectively surfacing logs, coaching employees to present a false story, or secretly running compute on undisclosed systems. It seems very easy to verify a claim to the effect that "Our company stopped a specific class of training run during a specific time interval." You just need a handful of people from a third party to go in, conduct some interviews, inspect some logs, and cross-check their findings. Assuming good process transparency , the public and other AGI companies would very likely believe the claims verified by the third party. Also, process transparency can protect both against minor deception and inadequate access, as the third party can directly tell the public that they weren’t able to arrive at proper conclusions.
The fact that more such engagements didn’t happen in 2026 seems like a major missed opportunity to me.
If third parties get lots of experience assessing safety-relevant claims, then they can build a knowledge base over time that makes the pause more robust. There are likely a number of technical and organizational pitfalls about verifying pauses that are hard to avoid without prior experience doing lower-stakes projects of a similar kind.
Contrast this with a vision where most of the setup of the pause is figured out without practicing subsets of it first. Maybe lots of people write lots of memos and papers about how to make the pause good. Without grounding in reality, companies and countries will be hard to convince that the pause will actually succeed and that other parties will uphold their commitments.
I’ve heard many people express the sentiment that we need to not pause/pace right now and save our pause/pacing activities for later when they’re more useful. This is based on a model where pacing right now makes it less likely that we will be able to pace in the future. I think this model is exactly backwards for the reasons I’ve laid out, like establishing technical competency, trust, and relationships necessary for successful pausing.
Instead, we should view pacing right now as enabling more involved pacing activities (including a pause) later.
There is a real worry that, if we spend our political capital now and cause AGI companies to slow down for no good reason, then this might lead to pauses being seen as useless, and therefore less likely to happen, going forward. I think we should keep this in mind, but I don't expect it to outweigh the benefit of enabling better pausing later. Part of why I think the downside is relatively small is that there appears to be no such rhetoric surrounding temporary slowdowns AGI companies have announced in 2026 (although again, few people took those seriously due to mistrust, so this provides limited evidence).
More importantly, the risk from backlash is mostly from slowdowns that seem arbitrary and unproductive, but I expect that we're at a point where companies could use breathing room to conduct large projects to make their systems safer, and third parties could help verify that companies are actually investing in safety during slowdowns.
Concretely, the following kinds of third-party assessments seem most valuable for setting up the right conditions for a global pause:
I’d guess that we don’t need any special hardware expertise to verify a US-only pause, we just need a few dozen experienced FTEs at a handful of third-party organizations that have done high-access assessments with AGI companies. Plausibly even a single third-party org with enough access and around a dozen FTEs would have enough capacity to verify a US-only pause as long as the AGI company employees were cooperative.
For a global pause, I’d guess that we will need substantial investment into hardware research, because trust and understanding are much lower between the US and China than between companies within the US. If tomorrow, the US government entrusted some competent third party to fly a dozen of their staff to China and verify that a Chinese company paused their training runs, I think that the combination of language barriers, cultural barriers, lack of organizational context, lack of existing relationships, time zone differences, strained international relations, and general increased latency would make it much harder to verify things than in a US-only context.
It’s plausible that AI R&D will be fully automated by the end of 2027. As a result we might have under a year to make a successful pause happen.
I think we need to incrementally shape a pause as soon as possible, and we need to very quickly increase the ambition of third-party engagements if we want the technical competency and trust in third parties to reach the appropriate level before it’s too late.
An example timeline of what it might look like to incrementally shape a global pause given ASI in mid-2028 by default:
I don’t have detailed takes on how exactly hardware verification scales up in 2027 in the scenario to meet the moment in 2028 to verify a global pause (because I don’t know much about hardware verification). Presumably, a number of smaller pilots happen in 2027 with multiple AGI companies participating, which leads to lessons learned for verifying a global pause.
Third parties need to very quickly scale up the ambition of the collaborations they are doing with AGI companies. If they fall notably behind the rough timeline I laid out (as measured by the scale of third-party engagements done by AGI companies), I would be pretty pessimistic about a pause being doable without substantial government buy-in, and even if it did happen I would be pessimistic about it succeeding.
A better end goal than "global pause" may be continued incremental development of coordination for exogenously controlling AI capability growth. In terms of your figure, the staircase should keep going even after we reach global pause (unless we are considering global stop). This continuation should include many pauses with increasing length and severity. A pause is economically costly and accumulates potential energy for future AI development. There are many mechanisms to reduce this cost & accumulation, which are certainly important to pursue, but none are exhaustive. Unfettered AI development is (obviously) quite risky, in part because human systems will be too slow to adequately respond. So nations can balance these risks over time by cycling between slow and fast capabilities growth. During slow downs or pauses, human systems (e.g. tech. diffusion, governance, safety research) get a chance to respond. Nations could coordinate by agreeing on measures for both when to start and when to end slowdowns or pauses. These conditions can be developed incrementally with practice, and enable negotiation between actors with different risk assessments. Current lab efforts towards pacing the frontier seems to be along these lines; although opaque, ill-defined, hard to verify, and ad hoc.
Even if you do it incrementally I still don't see how a global pause will ever come into effect. This seems to me to be the key sticking point: 1. The USA decides it wants to pause. The Chinese say: "Sure, we are benevolent responsible leaders of the global system and agree that superintelligent ASI is an existential risk. So for this Pause. Are you willing to let us catch up entirely to your level of AI capabilities? Or instead, do you come down to our levels of capability" [1] I find it very hard to imagine that the USA is willing to forego it's technological lead given realist great power competition concerns. I also do not think any coercive measures such as greater export controls for China, or threats of military intervention will change China's mind on this. They will instead see this as an act of warfare, and retaliate forcefully. They consider the USA as trying to curb its technological development in general . And recently they have started to forcefully retaliate against such actions. In short. I believe the USA really has to truly benevolently let the Chinese catch up to its AI frontier capabilities for a global pause to be feasible.
Note this is exactly what the Chinese have told the USA concerning nuclear weapons. The Chinese are still building out their nuclear arsenal at a rapid pace as of today.
The calculus for China is going to change as the USA approaches ASI and China realises it no longer has any hopes of catching up. China is several months behind, an eternity in the AI race. If Chinese intelligence confirmed today that America is on the verge of ASI, the CCP would understand the race is lost and shift to damage control mode. For China, a pause that solidifies the current USA lead would still be preferable to a non-pause that allows America to cross the ASI threshold unchallenged (because, I reiterate, at this point China would be too far behind to catch up).
The risk, in fact, would be the US government rejecting Chinese offers for a pause because "we're not going to quit while we're ahead". In fact, this happens in AI 2027: "Given China’s fear of losing the race, it has a natural interest in an arms control treaty, but overtures to U.S. diplomats lead nowhere".
I see. I think that's one plausible way things could go. But I don't know if the CCP sees it that way or will ever see it that way. I currently personally doubt it because I don't think they consider digital technologies as real technologies (this is in their own words), and that they consider to own the means of production on earth, i.e., the heights of their industrial might, and that they may think that as long as they control those they aren't under much of any threat. From this worldview it does follow that as soon as industrial output in the USA starts expanding rapidly the CCP would change its position. But China itself controls sorta all upstream supply chains on earth, so, it's basically impossible for the USA to scale industry without asking China's permission. This style of thinking is also why they are only really bullish on AI for 'improving real production' and so forth and so on, and are working really hard on diffusing AI in all industrial sectors. @Daniel Kokotajlo I'm really curious what you think about these things. I don't see an understanding of supply chains in this community (and how inflexible they are within time periods of let's say 25 years).



正在加载评论…