I'd add to the examples you list at the start. They'll tend on the Plan S / inform the public side since that's where I find myself. (Thanks for exploring the A vs S spectrum in your piece.)
Control AI's
https://www.narrowpath.co/ framing was well-reviewed (in both senses, number who've fed back, and level of quality perceived.) https://controlai.org/dip followed, and https://asi-prevention.com offers insight on middle powers. Strongly suspect these ideas are a good candidate for making concrete and gaming.
You linked EY's old Time article but more detail and related ideas reside in MIRI's papers:
Lastly, the Baruch plan initiative recently suggestes a particular approach/branding in the context of the current Trump presidency: https://www.cbpai.org/
The problem with the projections of the AI futures project , and most futurists, is that they assume that the fundamental computing paradigm under AI will remain stable.
There is no reason to believe this is the case. The current paradigm of AI compute is extremely, extremely inefficient.
We are essentially ignoring a wide swath of less conventional computing processes that perform the same computational functions at efficiencies and scales 6-10 orders of magnitude better , because we are simulating with great effort computational processes that physics basically gives us for free if we harness analog processes.
Those technologies are being developed, but take years to scale. They will start to gain traction around 2030-35. At that point, it may be possible to run current SOTA level models(2-3TP) in single user scale on about 20watts inside 10cm3 of infrastructure, if current lab examples can be scaled.
The idea of AI controllability is much more tractable if significant development requires massive capex and energy expenditure… but there is no physical constraint that makes that a stable assumption.
There is a foreseeable possibility of technologies like thermal wells and other physics based computing paradigms enabling inference so efficient and cheap that it becomes less expensive to simulate compute than to build it.
Here's one suggestion for future work: learn how things are actually made. Talk to people designing robots. Talk to semiconductor engineers. Talk to construction engineers. Etc
Not just "talk" - prod them to provide timelines and address bottlenecks. Not "I vaguely remember the robotics guy being guilty about replacing human labor". Ask when.
As of now, you are good at "bits" and totally ignorant of "atoms". Therefore your economic forecasts are laughable, "100% automation of all human labor by 2031", I can't even. This will not happen.
What are the scenarios if there is no broad deployment? It would seem that there would be a much smaller economic growth, and likely much slower AI progress, as AI progress depends on money, which can likely only be generated by investors getting something in return. You mention that you are not sure whether broad deployment is desireable or not, but you do not mention this potential benefits of minimal deployment. For example, Mythos being not deployed is a signal to AI companies that they may not be able to reap the economic benefits of more powerful models.
On the verification question: the interpretability-as-lie-detector hope just got a concrete measurement worth pricing in. Anthropic's July 6 global-workspace paper (J-lens) puts the reportable, inspectable part of a model at under a tenth of its activity, and shows it's approximate and steerable: ablate the evaluation-awareness representations, and concealed behavior surfaces. For an agreement that has to detect defection, an audit that reads the reportable layer and can be steered is the gap a defector plays in.
I'm not convinced that superintelligent AI is inherently dangerous unless aligned with human values. The preponderance of evidence over thousands of year shows us that it is humans who are inherently dangerous.
> To achieve victory, a covert project must stay undetected and make fast enough progress, which we think is less than 10% in this particular scenario. This means that the covert project strategy would not be worth taking in the first place.
Slowing America could be a goal in itself. Agreeing to Plan A and continuing a covert project is a win for China no matter the outcome because it will slow American progress. Therefore, this scenario seems much more likely than 10%.
Thinking and reading more, is this another way to think about it, in a more pragmatic vein?
Plan A: The US spends (some of) its AI lead to buy the world several years of alignment research, knowing China will cheat at the margins and eventually leave, and betting that this is still better than the alternative.
This "plan," if executed, would represent an unprecedented intrusion of the government into all aspects of the economy and R&D workflows. In effect, this would exchange the risk of ASI for turning over real-time universal monitoring of all compute to the US and Chinese governments. To me anyway, that is as or more scary than the risks associated with ASI.
I'd add to the examples you list at the start. They'll tend on the Plan S / inform the public side since that's where I find myself. (Thanks for exploring the A vs S spectrum in your piece.)
Control AI's
https://www.narrowpath.co/ framing was well-reviewed (in both senses, number who've fed back, and level of quality perceived.) https://controlai.org/dip followed, and https://asi-prevention.com offers insight on middle powers. Strongly suspect these ideas are a good candidate for making concrete and gaming.
You linked EY's old Time article but more detail and related ideas reside in MIRI's papers:
* https://intelligence.org/wp-content/uploads/2025/05/AI-Governance-to-Avoid-Extinction.pdf
* https://arxiv.org/abs/2412.08653
* https://intelligence.org/wp-content/uploads/2024/11/Mechanisms-to-Verify-International-Agreements-About-AI-Development-27-Nov-24.pdf
Lastly, the Baruch plan initiative recently suggestes a particular approach/branding in the context of the current Trump presidency: https://www.cbpai.org/
The problem with the projections of the AI futures project , and most futurists, is that they assume that the fundamental computing paradigm under AI will remain stable.
There is no reason to believe this is the case. The current paradigm of AI compute is extremely, extremely inefficient.
We are essentially ignoring a wide swath of less conventional computing processes that perform the same computational functions at efficiencies and scales 6-10 orders of magnitude better , because we are simulating with great effort computational processes that physics basically gives us for free if we harness analog processes.
Those technologies are being developed, but take years to scale. They will start to gain traction around 2030-35. At that point, it may be possible to run current SOTA level models(2-3TP) in single user scale on about 20watts inside 10cm3 of infrastructure, if current lab examples can be scaled.
The idea of AI controllability is much more tractable if significant development requires massive capex and energy expenditure… but there is no physical constraint that makes that a stable assumption.
There is a foreseeable possibility of technologies like thermal wells and other physics based computing paradigms enabling inference so efficient and cheap that it becomes less expensive to simulate compute than to build it.
Thanks for all you’ve doing, Daniel. AMDG
Thank you so much for everything you do! I will be doing all my projects on these scenarios in film school this fall. This is an incredible resource.
Here's one suggestion for future work: learn how things are actually made. Talk to people designing robots. Talk to semiconductor engineers. Talk to construction engineers. Etc
Not just "talk" - prod them to provide timelines and address bottlenecks. Not "I vaguely remember the robotics guy being guilty about replacing human labor". Ask when.
As of now, you are good at "bits" and totally ignorant of "atoms". Therefore your economic forecasts are laughable, "100% automation of all human labor by 2031", I can't even. This will not happen.
What are the scenarios if there is no broad deployment? It would seem that there would be a much smaller economic growth, and likely much slower AI progress, as AI progress depends on money, which can likely only be generated by investors getting something in return. You mention that you are not sure whether broad deployment is desireable or not, but you do not mention this potential benefits of minimal deployment. For example, Mythos being not deployed is a signal to AI companies that they may not be able to reap the economic benefits of more powerful models.
On the verification question: the interpretability-as-lie-detector hope just got a concrete measurement worth pricing in. Anthropic's July 6 global-workspace paper (J-lens) puts the reportable, inspectable part of a model at under a tenth of its activity, and shows it's approximate and steerable: ablate the evaluation-awareness representations, and concealed behavior surfaces. For an agreement that has to detect defection, an audit that reads the reportable layer and can be steered is the gap a defector plays in.
I have this "naive" idear of how to fix it.. i have version of it from different perspectives:
it is wrok in progress but have a read:
https://x.com/mrselfmadetoo/status/2025361115442512148
a shorter version: https://x.com/mrselfmadetoo/status/2064793628233580720
even shorter https://x.com/mrselfmadetoo/status/2045966406630478323
or a video on youtube: https://www.youtube.com/watch?v=SsBiGBK-BO0 published by safeaiforever.org
is it dead at conception ?
I'm not convinced that superintelligent AI is inherently dangerous unless aligned with human values. The preponderance of evidence over thousands of year shows us that it is humans who are inherently dangerous.
> Covert Project Further Research
Yes, definitely think more here needs to be done. The current scenario doesn't give me much confidence :
https://ai-2040.com/?choices=plan-a-root#branchpoint-china-attempts-a-covert-agi-project
> To achieve victory, a covert project must stay undetected and make fast enough progress, which we think is less than 10% in this particular scenario. This means that the covert project strategy would not be worth taking in the first place.
Slowing America could be a goal in itself. Agreeing to Plan A and continuing a covert project is a win for China no matter the outcome because it will slow American progress. Therefore, this scenario seems much more likely than 10%.
Thinking and reading more, is this another way to think about it, in a more pragmatic vein?
Plan A: The US spends (some of) its AI lead to buy the world several years of alignment research, knowing China will cheat at the margins and eventually leave, and betting that this is still better than the alternative.
This "plan," if executed, would represent an unprecedented intrusion of the government into all aspects of the economy and R&D workflows. In effect, this would exchange the risk of ASI for turning over real-time universal monitoring of all compute to the US and Chinese governments. To me anyway, that is as or more scary than the risks associated with ASI.