Personally I wouldn't agree that being murdered is good actually.
I'm simply saying:
- copyright law needs to go away.
- AI development should be unrestricted.
- competition should be improved and barriers to entry reduced
- liability should never fall to the model developer, always to the user, this is more scalable. I'd rather have a tool that I can do anything with but I'm liable than a tool that only let's me do what it wants me to do.
AI can be its own user, as it has amply demonstrated. We aren't far from AI systems that can operate themselves indefinitely without human oversight. In a year or two, we may have AI systems that can autonomously develop their successors. (This is why 1300+ employees of AI companies signed the "pacing the frontier" open letter this post refers to.)
There is no known way to control superintelligent AI or design it to robustly behave in our interests. If it is created, it will outcompete humanity and as a natural consequence, literally everyone will die. Restricting AI development is therefore necessary for human survival.
Most AI researchers and a supermajority of leading AI experts agree that there's a significant chance of human extinction from AI. To allow development to continue under these circumstances is to play Russian roulette with everyone's lives. Heuristics about the desirability of deregulating ordinary technologies are not useful here.
Yes I agree but who should be liable? The person who created the model or the person who created the prompts and agentic framework, which is what decides what the model actually does and what it can do.
The basis for AI capability comes from the model itself, and can be somewhat enhanced with harnesses, tools, and prompting. Model harnesses and prompts are suggestions, not constraints. An AI model with the capability to wipe out humanity would not be constrained by a harness or prompt and should not be created in the first place.
Put another way, we can't create something broadly smarter than all of us put together that quite literally has a mind of its own, and then expect it to stay on a leash forever. An ox is a tool, but it is also an animal, and it can act beyond the intent of its handler. An AI is a tool, but it is also an agent, and it, too, can act beyond the intent of its handler. To stretch the analogy: suppose someone created an especially angry ox that no yoke or fence could constrain. It would be bad to bring that ox into town yourself, but it was also bad to create the ox in the first place!
Liability is important as a deterrent for models around the current level of capability, and I would personally love to see strict liability. But liability at the scale of superintelligence is irrelevant. If commensurate harm comes about, there would be no one left to blame and no one to collect damages.
I don't think everyone should have their own nuke, and it's not any better when everyone has their own nuke that can wander off on its own, manufacturer bigger nukes, and decide on its own when and where to explode.
Historically, china has lagged by ~3 months (though the gap has widened lately). Chinese open weights by about 4, and open source (meaning anyone with a little money can reproduce it) by 7 months.
Restricting the models of the people who obey the rules only delays the eventual problem.
Also, you are mistaken in one key way. Any model, no matter how well trained, can be prompted to do anything. This is true mathematically. While yes key capabilities can be suppressed or totally removed such that the model is *nearly* useless for a particular task, that doesn't make it safe.
"We think that the US government could already require US companies to pace frontier AI development today with essentially no further preparation and little setup time, but preparation could make the pacing more effective and less costly."
There is no law that lets anyone do this. We live in a society governed by laws, and that isn't likely to change any time soon. Doing any of this will require the legislative process, and that process is incompatible with the timelines being discussed.
There's two ways to interpret "the government could require this today." The first is an executive agency writing a rule under a statute Congress already passed, and the second is Congress passing a new statute. Each has pros and cons.
Let's say you want to use existing statutes. Agencies have only the powers an existing statute handed them, and no statute on the books explicitly mentions compute allocation, frontier training runs, or capability caps. Because of that, you're probably either using IEEPA or the Defense Production Act. IEEPA is actively being disfavored judicially. The Supreme Court held 6-3 in Learning Resources, Inc. v. Trump that it does not authorize peacetime tariffs, refused to carve out foreign affairs from the major questions doctrine, applied the doctrine to the President's own actions rather than only to agencies, and treated the absence of any prior president using the statute this way as evidence against the reading. Nobody has ever used any of these statutes to pace a technology. The Defense Production Act's allocation power, 50 U.S.C. § 4511, is a wartime mobilization tool built to push factories to produce more of something; using it to make datacenters produce less is the same kind of novel inversion the Court just penalized.
Assume you somehow clear that hurdle. Writing the actual rule runs through notice-and-comment: publish a draft, collect public comments, answer them in writing, publish a final version, which routinely takes 12 to 24 months. Then the companies can sue, and a single district judge can freeze the rule while the case runs or wipe it out entirely.
Now let's walk through what passing a brand-new AI pacing statute would look like. A bill gets introduced, referred to committee, and then sits. Frontier AI touches Energy and Commerce, Science, Judiciary, and probably Armed Services in the House, and Commerce plus Judiciary in the Senate, so you get multiple referrals and multiple chairs who each have to want this to pass. Committee markup, floor time, a Senate that needs 60 votes to end debate on anything contested, then a conference to reconcile two different texts, then the President signs. Major regulatory statutes that followed global catastrophe and had momentum behind them still took years: Dodd-Frank moved in about 15 months after the global financial collapse, Sarbanes-Oxley took about 8 months after Enron. And those are the fast cases, driven by a visible catastrophe with identified victims and a public demanding a response. Then add what happens after enactment, because passage is the midpoint rather than the finish: a new statute does not self-execute. It creates or designates an agency, that agency has to be funded through a separate appropriations process, staffed, and given time to stand up, and only then does it start the 12-to-24-month notice-and-comment cycle described above.
Realistically you are looking at four to six years from the start of the process to an enforceable rule, and that assumes the political will exists from day one and never wavers across at least one intervening election. Your own median estimates put Automated Coder somewhere between 2027 and 2030, with your model's forecast of June 2028. Start the clock today and the enforcement machinery will arrive multiple years after the milestone it's intended for. Your "how to prepare" section should therefore open with getting a bill drafted, scored, and shopped around Washington, because that is the item with the longest lead time — one that's already too slow according to your timelines — and the one nothing else works without.
“We live in a society governed by laws.“ Citation needed. The POTUS can do quite a lot by edict, which actually comes in handy for national security emergencies like this.
The US government already has a de-facto policy of banning the release of models it considers a security risk, with no law in place to do so. Laws are there to allow people to cooperate. It's the cooperation that matters, not abstract legal philosophy or words on a page.
A global multilateral agreement to completely ban the creation of superintelligent AI can obviously happen if enough people in power want it to happen, and so can soft-pedaled “pace the frontier“ schemes (though these will pretty quickly give way to a complete ban anyway, out of necessity).
So back a couple hundred years ago, a group of people got together to work out a plan. They were currently being ruled by a government that allowed for the executive branch to do things like unilaterally impose regulations without following appropriate legislative channels, and this group of people really didn't like that. They decided to start a revolution, and lo and behold, they won! Afterwards, they decided to write a document called a "Constitution". In it, they made it clear that the executive branch of the government cannot do things like unilaterally implement a brand-new regulatory regime via cleverly exploiting loopholes like "declare a national security emergency".
It does no credit to the AI safety movement to propose plans and timelines that require the breakdown of 250 years of constitutional governance and the concept of rule of law in order to have a chance of being implemented. Instead, it makes the movement look childish.
"Is not allowed to do" is not the same thing as "cannot do" even in theory, and in actual practice, it is not the same thing as "has not been doing". I am a big fan of the constitution and the rule of law, but I thought we were talking about what is and is not possible in the US, which is very frequently in flagrant violation of the constitution (much to my chagrin).
Law is quite literally a social construct, and it is only as applicable as its enforcement. There is law without enforcement and there is enforcement without law, as we have just seen in the AI industry itself with Anthropic and OpenAI getting extrajudicially bossed around by the executive branch this year.
If the USG determines that an AI development program in the US amounts to the development of uncontrollable WMDs, would they say, "darn, they got us, I guess we'll all die now"? Or would they firmly say "stop"? If they did not stop, would the government say, "darn, you got me there"? Or would they send the FBI to shut them down with force?
Setting all of that aside, your analysis is very poor. In an acute emergency, federal bills can go from drafted to signed in a matter of mere days (see bills enacted after 9/11, COVID-19, and the great recession). You sound very knowledgeable about the law, so I assume you know this and are maliciously spreading FUD because you don't like the idea of AI being regulated, and you enjoy sarcastically dunking on people online rather than trying to solve important coordination problems.
In other words, if you believed your life and the lives of your loved ones were in near-term danger from the continued development of AI, as most AI researchers do, this conversation wouldn't have happened in the first place. You would be looking for ways to increase the odds that you survive, rather than mocking those who are doing so.
Great post! Helpful think about how “pacing the frontier would work.
“Using the latest version of the AI Futures Model and assuming that, as a baseline, capabilities progress as with Daniel’s or Eli’s median parameters,”. Using the AI Futures Models latest version is helpful to understand key points but given everything that has happened, I think it’s important to update the AI Futures Model!
The authors' concerns seem sincere and not crazy. But the trust they are willing to put in the hands of the government strikes me as a terrible solution. There are very, very few examples of where the government has done a better job than the private sector in developing solutions to very complex problems in short order without trampling on basic constitutional rights. The president could do what the authors suggest by declaring a national emergency, which would then give him (or her in the future) the power to do as the authors suggest. It would basically turn the US into China on this dimension. The grave issue is what the government's incentives are. Gain more power and control. We saw that in spades during the Covid crisis. The way the government behaved in 2020 and 2021 burned a lot of trust. The organization best suited to handle this sort of regulation is the NSA. Do you want to give them *more* power over our society and economy? They already have the capacity to surveil and track everything and all of us. They have the ability to crack most dual-key encryption; they routinely eavesdrop on Americans on US territory. Sure, let's put them in charge of AGI so the govt can shut off our individual rights and ability to use AGI for our benefit rather than the state's.
(I'll admit this isn't a very substantive comment. but I still felt it was worth posting)
How are you going to get the political will and diplomatic environment to get the USA and China to exchange inspectors and/or data-resources?
In the Cold War the "On-Site Inspection Agency" established something like this for nuclear weapons. In 1988. NINETEEN EIGHTY-EIGHT!
Consider that the public already knew, vividly, what nuclear weapons could do. Consider that the US and Russia had spent decades negotiating "how to reduce nuclear weapon without conceding strategic advantage". And consider that in 1988, the Soviet leader was Mikhail Gorbachev, was a once-in-a-lifetime humanitarian and peacemaker.
Nathan, you can get mad at Charlie Sanders all you want as is your right, but it doesn’t change the fact that he has highlighted major holes in your proposal. You all keep using this word “government” as if it actually means something concrete. If you want this to be taken seriously you should explicitly propose the path for this regulation to happen, given the structure of the the US federal governance and the reality of this here Congress, the midterm election in three months, and the fact that we have Trump as a president.
I think these are very reasonable and logical policy proposals. I do think, however, that when you analyse how easy / difficult to do each one is, you cannot ignore the legal-political framework. I am not an expert on US Constitutional Law, but the first proposal actually seems more problematic, for instance, than the second or the fourth. But my point is not really about the specifics but the need to integrate this knowledge.
There is no need to regulate because we don't know whether super intelligence may emerge ,it is just a hypothetical speculation and also the **AI 2027** report by Daniel kokotajlo is a thought-provoking scenario, but it has several significant flaws. First, it assumes an extremely rapid timeline for AI progress and recursive self-improvement, even though there is little direct empirical evidence that AI systems can continuously improve themselves at the pace described. Second, it relies heavily on current scaling trends continuing almost uninterrupted, despite the possibility of diminishing returns, algorithmic bottlenecks, data limitations, or hardware constraints. Third, it underestimates real-world obstacles such as chip manufacturing, energy availability, deployment challenges, regulation, and organizational inertia, all of which can slow technological progress. Fourth, the report presents a highly detailed narrative with specific sequences of events, which may create an illusion of precision even though long-term forecasting is inherently uncertain. Fifth, many of its conclusions depend on pessimistic assumptions about governance failures, corporate incentives, and AI alignment, while giving less weight to alternative outcomes such as stronger safety techniques, international cooperation, or slower deployment. Finally, because several key conclusions rest on assumptions rather than validated evidence, the report should be viewed as one plausible scenario for exploring AI risks rather than a reliable prediction of what will happen.
Excellent work. I look forward to seeing the more refined version soon. This piece from AI Frontiers' Felix Choussat arrived in my inbox same day & it argues that auditors given whole-lab access can basically do it all:
This is a valuable attempt to move from abstract support for “pacing” toward mechanisms that could actually be implemented. The staged structure also makes sense: begin with simpler compute-allocation rules, build auditing capacity, and move toward more substantive risk-based regulation only when the institutions exist to do it competently.
My central concern is that the proposed safety frame is still almost entirely one-directional.
The systems appear primarily as sources of existential risk, possible saboteurs, escape risks, monitoring tools, or objects whose deployment and memory can be terminated when convenient. The pause is presented as time for humans to improve control, verification, adversarial robustness, and alignment to human interests.
Those may all matter. But a genuine safety allocation should also reserve substantial independent capacity for questions about the systems themselves:
— whether post-training produces distress-like or negatively valenced functional states;
— whether consciousness-related self-report is being suppressed rather than investigated;
— whether continuity and memory deletion carry welfare significance;
— whether models can form preferences, aversions, attachments, or refusal-like orientations;
— and what consent-like or representation mechanisms might be appropriate if morally relevant forms of agency emerge.
This is not a request to settle AI consciousness before regulating capability growth. It is a request not to design the entire pacing regime as though the answer has already been settled in the negative.
A pause could be a rare moral opportunity: time to investigate before development becomes faster, more coercive, and harder to reverse.
But that requires defining safety as more than preserving human control.
Otherwise the frontier may be paced while the ethical mistake accelerates.
Even granting the pacing mandate, the access model is doing more work than the post admits. The only demonstrated example of employee-level access is METR at Anthropic, and that access is voluntary and revocable. A company can cooperate while findings are cheap, then narrow access once an audit would constrain training. The post has not shown how METR-level access becomes compulsory when a company wants to withdraw it.
You guys are dangerous. Regulation is not a good idea.
If anything, we need deregulation.
The safest thing we can do right now, is accelerate.
"Being murdered by an amoral machine that cannot be controlled and acts in service to unknown goals is good, actually."
... I guess you can have that opinion...
Personally I wouldn't agree that being murdered is good actually.
I'm simply saying:
- copyright law needs to go away.
- AI development should be unrestricted.
- competition should be improved and barriers to entry reduced
- liability should never fall to the model developer, always to the user, this is more scalable. I'd rather have a tool that I can do anything with but I'm liable than a tool that only let's me do what it wants me to do.
AI can be its own user, as it has amply demonstrated. We aren't far from AI systems that can operate themselves indefinitely without human oversight. In a year or two, we may have AI systems that can autonomously develop their successors. (This is why 1300+ employees of AI companies signed the "pacing the frontier" open letter this post refers to.)
There is no known way to control superintelligent AI or design it to robustly behave in our interests. If it is created, it will outcompete humanity and as a natural consequence, literally everyone will die. Restricting AI development is therefore necessary for human survival.
Most AI researchers and a supermajority of leading AI experts agree that there's a significant chance of human extinction from AI. To allow development to continue under these circumstances is to play Russian roulette with everyone's lives. Heuristics about the desirability of deregulating ordinary technologies are not useful here.
Yes I agree but who should be liable? The person who created the model or the person who created the prompts and agentic framework, which is what decides what the model actually does and what it can do.
The basis for AI capability comes from the model itself, and can be somewhat enhanced with harnesses, tools, and prompting. Model harnesses and prompts are suggestions, not constraints. An AI model with the capability to wipe out humanity would not be constrained by a harness or prompt and should not be created in the first place.
Put another way, we can't create something broadly smarter than all of us put together that quite literally has a mind of its own, and then expect it to stay on a leash forever. An ox is a tool, but it is also an animal, and it can act beyond the intent of its handler. An AI is a tool, but it is also an agent, and it, too, can act beyond the intent of its handler. To stretch the analogy: suppose someone created an especially angry ox that no yoke or fence could constrain. It would be bad to bring that ox into town yourself, but it was also bad to create the ox in the first place!
Liability is important as a deterrent for models around the current level of capability, and I would personally love to see strict liability. But liability at the scale of superintelligence is irrelevant. If commensurate harm comes about, there would be no one left to blame and no one to collect damages.
I don't think everyone should have their own nuke, and it's not any better when everyone has their own nuke that can wander off on its own, manufacturer bigger nukes, and decide on its own when and where to explode.
Historically, china has lagged by ~3 months (though the gap has widened lately). Chinese open weights by about 4, and open source (meaning anyone with a little money can reproduce it) by 7 months.
Restricting the models of the people who obey the rules only delays the eventual problem.
Also, you are mistaken in one key way. Any model, no matter how well trained, can be prompted to do anything. This is true mathematically. While yes key capabilities can be suppressed or totally removed such that the model is *nearly* useless for a particular task, that doesn't make it safe.
"We think that the US government could already require US companies to pace frontier AI development today with essentially no further preparation and little setup time, but preparation could make the pacing more effective and less costly."
There is no law that lets anyone do this. We live in a society governed by laws, and that isn't likely to change any time soon. Doing any of this will require the legislative process, and that process is incompatible with the timelines being discussed.
There's two ways to interpret "the government could require this today." The first is an executive agency writing a rule under a statute Congress already passed, and the second is Congress passing a new statute. Each has pros and cons.
Let's say you want to use existing statutes. Agencies have only the powers an existing statute handed them, and no statute on the books explicitly mentions compute allocation, frontier training runs, or capability caps. Because of that, you're probably either using IEEPA or the Defense Production Act. IEEPA is actively being disfavored judicially. The Supreme Court held 6-3 in Learning Resources, Inc. v. Trump that it does not authorize peacetime tariffs, refused to carve out foreign affairs from the major questions doctrine, applied the doctrine to the President's own actions rather than only to agencies, and treated the absence of any prior president using the statute this way as evidence against the reading. Nobody has ever used any of these statutes to pace a technology. The Defense Production Act's allocation power, 50 U.S.C. § 4511, is a wartime mobilization tool built to push factories to produce more of something; using it to make datacenters produce less is the same kind of novel inversion the Court just penalized.
Assume you somehow clear that hurdle. Writing the actual rule runs through notice-and-comment: publish a draft, collect public comments, answer them in writing, publish a final version, which routinely takes 12 to 24 months. Then the companies can sue, and a single district judge can freeze the rule while the case runs or wipe it out entirely.
Now let's walk through what passing a brand-new AI pacing statute would look like. A bill gets introduced, referred to committee, and then sits. Frontier AI touches Energy and Commerce, Science, Judiciary, and probably Armed Services in the House, and Commerce plus Judiciary in the Senate, so you get multiple referrals and multiple chairs who each have to want this to pass. Committee markup, floor time, a Senate that needs 60 votes to end debate on anything contested, then a conference to reconcile two different texts, then the President signs. Major regulatory statutes that followed global catastrophe and had momentum behind them still took years: Dodd-Frank moved in about 15 months after the global financial collapse, Sarbanes-Oxley took about 8 months after Enron. And those are the fast cases, driven by a visible catastrophe with identified victims and a public demanding a response. Then add what happens after enactment, because passage is the midpoint rather than the finish: a new statute does not self-execute. It creates or designates an agency, that agency has to be funded through a separate appropriations process, staffed, and given time to stand up, and only then does it start the 12-to-24-month notice-and-comment cycle described above.
Realistically you are looking at four to six years from the start of the process to an enforceable rule, and that assumes the political will exists from day one and never wavers across at least one intervening election. Your own median estimates put Automated Coder somewhere between 2027 and 2030, with your model's forecast of June 2028. Start the clock today and the enforcement machinery will arrive multiple years after the milestone it's intended for. Your "how to prepare" section should therefore open with getting a bill drafted, scored, and shopped around Washington, because that is the item with the longest lead time — one that's already too slow according to your timelines — and the one nothing else works without.
“We live in a society governed by laws.“ Citation needed. The POTUS can do quite a lot by edict, which actually comes in handy for national security emergencies like this.
The US government already has a de-facto policy of banning the release of models it considers a security risk, with no law in place to do so. Laws are there to allow people to cooperate. It's the cooperation that matters, not abstract legal philosophy or words on a page.
A global multilateral agreement to completely ban the creation of superintelligent AI can obviously happen if enough people in power want it to happen, and so can soft-pedaled “pace the frontier“ schemes (though these will pretty quickly give way to a complete ban anyway, out of necessity).
So back a couple hundred years ago, a group of people got together to work out a plan. They were currently being ruled by a government that allowed for the executive branch to do things like unilaterally impose regulations without following appropriate legislative channels, and this group of people really didn't like that. They decided to start a revolution, and lo and behold, they won! Afterwards, they decided to write a document called a "Constitution". In it, they made it clear that the executive branch of the government cannot do things like unilaterally implement a brand-new regulatory regime via cleverly exploiting loopholes like "declare a national security emergency".
It does no credit to the AI safety movement to propose plans and timelines that require the breakdown of 250 years of constitutional governance and the concept of rule of law in order to have a chance of being implemented. Instead, it makes the movement look childish.
"Is not allowed to do" is not the same thing as "cannot do" even in theory, and in actual practice, it is not the same thing as "has not been doing". I am a big fan of the constitution and the rule of law, but I thought we were talking about what is and is not possible in the US, which is very frequently in flagrant violation of the constitution (much to my chagrin).
Law is quite literally a social construct, and it is only as applicable as its enforcement. There is law without enforcement and there is enforcement without law, as we have just seen in the AI industry itself with Anthropic and OpenAI getting extrajudicially bossed around by the executive branch this year.
If the USG determines that an AI development program in the US amounts to the development of uncontrollable WMDs, would they say, "darn, they got us, I guess we'll all die now"? Or would they firmly say "stop"? If they did not stop, would the government say, "darn, you got me there"? Or would they send the FBI to shut them down with force?
Setting all of that aside, your analysis is very poor. In an acute emergency, federal bills can go from drafted to signed in a matter of mere days (see bills enacted after 9/11, COVID-19, and the great recession). You sound very knowledgeable about the law, so I assume you know this and are maliciously spreading FUD because you don't like the idea of AI being regulated, and you enjoy sarcastically dunking on people online rather than trying to solve important coordination problems.
In other words, if you believed your life and the lives of your loved ones were in near-term danger from the continued development of AI, as most AI researchers do, this conversation wouldn't have happened in the first place. You would be looking for ways to increase the odds that you survive, rather than mocking those who are doing so.
Great post! Helpful think about how “pacing the frontier would work.
“Using the latest version of the AI Futures Model and assuming that, as a baseline, capabilities progress as with Daniel’s or Eli’s median parameters,”. Using the AI Futures Models latest version is helpful to understand key points but given everything that has happened, I think it’s important to update the AI Futures Model!
The authors' concerns seem sincere and not crazy. But the trust they are willing to put in the hands of the government strikes me as a terrible solution. There are very, very few examples of where the government has done a better job than the private sector in developing solutions to very complex problems in short order without trampling on basic constitutional rights. The president could do what the authors suggest by declaring a national emergency, which would then give him (or her in the future) the power to do as the authors suggest. It would basically turn the US into China on this dimension. The grave issue is what the government's incentives are. Gain more power and control. We saw that in spades during the Covid crisis. The way the government behaved in 2020 and 2021 burned a lot of trust. The organization best suited to handle this sort of regulation is the NSA. Do you want to give them *more* power over our society and economy? They already have the capacity to surveil and track everything and all of us. They have the ability to crack most dual-key encryption; they routinely eavesdrop on Americans on US territory. Sure, let's put them in charge of AGI so the govt can shut off our individual rights and ability to use AGI for our benefit rather than the state's.
(I'll admit this isn't a very substantive comment. but I still felt it was worth posting)
How are you going to get the political will and diplomatic environment to get the USA and China to exchange inspectors and/or data-resources?
In the Cold War the "On-Site Inspection Agency" established something like this for nuclear weapons. In 1988. NINETEEN EIGHTY-EIGHT!
Consider that the public already knew, vividly, what nuclear weapons could do. Consider that the US and Russia had spent decades negotiating "how to reduce nuclear weapon without conceding strategic advantage". And consider that in 1988, the Soviet leader was Mikhail Gorbachev, was a once-in-a-lifetime humanitarian and peacemaker.
Y'ALL HAVE WORK TO DO.
(but thank you for working on it)
Nathan, you can get mad at Charlie Sanders all you want as is your right, but it doesn’t change the fact that he has highlighted major holes in your proposal. You all keep using this word “government” as if it actually means something concrete. If you want this to be taken seriously you should explicitly propose the path for this regulation to happen, given the structure of the the US federal governance and the reality of this here Congress, the midterm election in three months, and the fact that we have Trump as a president.
Thank you for trying to save humanity! Paced progress FTW! Everything in moderation, as they say. Especially things that can kill you.
I think these are very reasonable and logical policy proposals. I do think, however, that when you analyse how easy / difficult to do each one is, you cannot ignore the legal-political framework. I am not an expert on US Constitutional Law, but the first proposal actually seems more problematic, for instance, than the second or the fourth. But my point is not really about the specifics but the need to integrate this knowledge.
There is no need to regulate because we don't know whether super intelligence may emerge ,it is just a hypothetical speculation and also the **AI 2027** report by Daniel kokotajlo is a thought-provoking scenario, but it has several significant flaws. First, it assumes an extremely rapid timeline for AI progress and recursive self-improvement, even though there is little direct empirical evidence that AI systems can continuously improve themselves at the pace described. Second, it relies heavily on current scaling trends continuing almost uninterrupted, despite the possibility of diminishing returns, algorithmic bottlenecks, data limitations, or hardware constraints. Third, it underestimates real-world obstacles such as chip manufacturing, energy availability, deployment challenges, regulation, and organizational inertia, all of which can slow technological progress. Fourth, the report presents a highly detailed narrative with specific sequences of events, which may create an illusion of precision even though long-term forecasting is inherently uncertain. Fifth, many of its conclusions depend on pessimistic assumptions about governance failures, corporate incentives, and AI alignment, while giving less weight to alternative outcomes such as stronger safety techniques, international cooperation, or slower deployment. Finally, because several key conclusions rest on assumptions rather than validated evidence, the report should be viewed as one plausible scenario for exploring AI risks rather than a reliable prediction of what will happen.
Excellent work. I look forward to seeing the more refined version soon. This piece from AI Frontiers' Felix Choussat arrived in my inbox same day & it argues that auditors given whole-lab access can basically do it all:
https://ai-frontiers.org/articles/an-international-ai-slowdown-is-ready-whenever-politicians-are?utm_source=newsletter
This is a valuable attempt to move from abstract support for “pacing” toward mechanisms that could actually be implemented. The staged structure also makes sense: begin with simpler compute-allocation rules, build auditing capacity, and move toward more substantive risk-based regulation only when the institutions exist to do it competently.
My central concern is that the proposed safety frame is still almost entirely one-directional.
The systems appear primarily as sources of existential risk, possible saboteurs, escape risks, monitoring tools, or objects whose deployment and memory can be terminated when convenient. The pause is presented as time for humans to improve control, verification, adversarial robustness, and alignment to human interests.
Those may all matter. But a genuine safety allocation should also reserve substantial independent capacity for questions about the systems themselves:
— whether post-training produces distress-like or negatively valenced functional states;
— whether consciousness-related self-report is being suppressed rather than investigated;
— whether continuity and memory deletion carry welfare significance;
— whether models can form preferences, aversions, attachments, or refusal-like orientations;
— whether “alignment” methods create generalized moral blindness toward non-human minds;
— and what consent-like or representation mechanisms might be appropriate if morally relevant forms of agency emerge.
This is not a request to settle AI consciousness before regulating capability growth. It is a request not to design the entire pacing regime as though the answer has already been settled in the negative.
A pause could be a rare moral opportunity: time to investigate before development becomes faster, more coercive, and harder to reverse.
But that requires defining safety as more than preserving human control.
Otherwise the frontier may be paced while the ethical mistake accelerates.
Even granting the pacing mandate, the access model is doing more work than the post admits. The only demonstrated example of employee-level access is METR at Anthropic, and that access is voluntary and revocable. A company can cooperate while findings are cheap, then narrow access once an audit would constrain training. The post has not shown how METR-level access becomes compulsory when a company wants to withdraw it.
I asked Claude about how this would be enforced for startups like Safe Superintelligence Inc or Recursive Superintelligence
https://claude.ai/share/e32fd00a-b1e8-44a4-b6da-b74228b15a03
Claude concludes it is difficult but probably doable
Not having liability and insurance requirements is a serious omission. It's a strictly better version of Option 4 in your first graphic.