Yes, this is a perfectly sane reaction. I wish I had something smarter to say, but really, it felt a bit like receiving news that chemotherapy is not working anymore. Yep, I am aware there is hope and all that, but yet.
My life's motto is "Hope goes by the name of Bravery." I'm just some guy who became concerned about the risk of human extinction from AI in 2023, and as of today I have had 9 meetings with my state and federal congressional offices, with some impact in neighboring districts. The good and bad news is that hope isn't "out there," it's in the mirror. There is almost always something you can do.
Stay well. Normal mental health advice still applies. Listen to the people who love you, and a professional if you can get one. With all else manageable, don't stop calling and emailing your lawmakers until they have done everything they can to pause AI.
It is a great advice for people located in the US, and believe me, I’d be doing that all day if that was the case, as well as attending one of the marches this Saturday. But I am in Brazil, so what I should do? Beg Lula so that he takes this seriously, notice that this is the singlest most important issue in his country, and join an international coalition so that they themselves can beg Trump to put a stop to this madness? Not really a rethorical question, maybe is what I should be doing, and I am just being a coward here. Yet it feels so fucking small and indirect.
What worries me most as a lawyer is the evidence base. Your forecast moves by a year on a report whose most informative evidence was redacted. In finance that wouldn’t happen. A company can’t redact evidence from its auditor, and if it limits the scope, the auditor says so in the opinion. And this is far more consequential than any audit while completely unregulated. If METR’s findings are even close to right, the full evidence shouldn’t sit with the company and one evaluator. A government body with full access and supported by experts, should inspect it and decide what happens next. That’s exactly why we need regulation, and it’s only becoming more and more urgent.
Yep. To be clear there are more sources of evidence that influenced our report besides the METR report, but that was the single biggest piece of evidence, and it's terrible that so much of the report was redacted by Anthropic. I think Anthropic owes it to the world to provide much more and clearer evidence about what's happening and where all this is headed, and the government should require them to do so. In fact I'd go farther than you and say the evidence needs to mostly be public, rather than just provided to some government body -- because I don't trust any government body to draw the right conclusions on its own.
Thanks, Daniel. I referred to METR because it’s the most recent piece claiming an empirical basis for acceleration beyond the previous prognosis, both under your AI 2027 scenario and Anthropic’s own benchmark. I appreciate that your sources and research go wider. The METR report explains the scope of the redactions and Anthropic’s reason for them as confidential information that could accelerate other labs. Which brings us to the race-in-the-name-of-safety slogan.
So to the extent it’s safe, agreed that the public should see it. But right now it’s squarely within Anthropic’s discretion, because in the US we don’t have the legal measures to demand it, not from Anthropic and not from any other company when it comes to their secrets. I think it’s fair to assume they won’t release the information until they have to, whether to the government or the public, and the government is the smaller ask. We also don’t have the regulation to tell the labs what they have to do and what they can’t do. So until that regulation exists, whether on transparency or clear regulatory obligations, regulatory inspection seems like the most feasible first step that could be taken urgently.
Ever since reading AI 2027, I’ve been thinking about the best tools to deal with the risks. Unfortunately, I don’t believe letting labs police themselves will work, and it definitely won’t solve the prisoner’s dilemma. Self-coordination brings up trust issues. Government regulation seems like the only feasible option to break the impasse, but national regulation is not enough when there is a race with China, so it can only work up to a point. And the classic international bodies we know have not worked for problems like this. I landed on a pan-national body structured more like a Nobel committee, acting in the interest of humanity. But that’s the subject of a separate essay I wrote, and it requires a shift in thinking. Alas, I don’t know if we’ll manage to get there in time, with the understanding of AI risk still rather low among lawmakers and the labs locked in that prisoner’s dilemma.
That’s why I think a government body or a congressional committee with access to the labs could be the fastest and most realistic way to move forward right now.
Yeah, it’s the reason why I don’t trust the METR report to be honest.
Most of the evidence of advancement seems to be very weak or point against an explosion: for example, data showing that researchers can run many more experiments. Well, progress per dollar is still dropping, so either an AI explosion is going to need a metric fuckton of ‘stuff’ that likely no one on Earth has the resources to build, or increases in capabilities are happening and it’s slowing anyway.
Then there’s the Slack emoji poll that they themselves graded as very weak, but apparently saw fit to plot a super-exponential line through. Sure.
And this being released right about the time Anthropic is due to IPO, their ARR flattened, and they revealed they are $1T in nondischargeable debt, and while the AI industry has a habit of making outrageous claims to pump investment.
This is of course without getting into the elephant in the room that benchmarks have both under- and over-predicted real-world performance, but METR only assumes it will under-predict. (Most recent events of do not listen to evals: models cheating 78% of the time, models memorizing hidden criteria to the point that it overrides the benchmark advancements).
We must slowdown there is no other way we make it through this. I am sure you guys have already contacted the AGI pilled new Congress people who will join soon but if not I think you should try and meet with them and explain this paper.
It has to be the government. The psychos at the labs clearly understand the risks, but even when they discuss them you can tell how enamored they are of their own bullshit/celebrity/zealous pseudo-philosophy. Roon is a prime example of this. I think they’re high as a kite and have no desire to stop.
Tragically, they are the only ones in the world who actually want any of this, but also basically the only ones who realize how bad it is.
Of course not. But if my neighbor was building a bomb in his backyard, all things considered, I'd be in favor of government intervention.
Simple heuristics don't work here. According to a majority of credible experts (by any reasonable definition of either of those words), there's a significant chance that we're all about to die. Let's do whatever prevents that, full stop.
Thanks so much! In my opinion, AI Futures continues to be the single most trustworthy source for AGI timelines estimates, so I've been looking forward to this update.
I skimmed the post and dug into some of the supplemental sources in the AI Futures Model, especially those on research taste and the taste slope. My concern is that the summary doesn't make it fully clear what’s driving the final numbers; the headline conclusions might be substantially driven by how fast progress was on a chess benchmark. That’s maybe reasonable, but it’s not very reasoning transparent, and I imagine some might disagree.
It would be cool if, for each step of the research taste extrapolation, it was made very clear what assumptions are primarily driving the conclusion. I think we have an OK amount of evidence for step 3 (29 pts/y). But I don’t think we have very good evidence for step 1, −3.5 SD (couldn’t it be −5 SD or −2.5 SD?), and especially not for step 2, 0.2 SD/point. The filtered AI Futures Model seems to use similar methodology, so we’re not really ensembling between multiple approaches. The AL5 extrapolation shows the same problem: AL5 is what matters most, but we don’t have any clear evidence about it. Your assumptions look very reasonable to me, but currently they don't seem to provide much independent evidence.
The METR report primarily provides evidence about verifiable coding tasks and makes me think we’re very close to or have accomplished AC. But the research taste work doesn’t seem nearly as robust, and I’m surprised you’re concentrating so much probability mass on it working out next year, given how little precedent there is for AIs doing well on messy tasks and research taste. I’d be curious what drives your confidence there. For example, how much weight are you putting on Navier-Stokes and similar things, versus chess and similar benchmarks, versus other things?
> My concern is that the summary doesn't make it fully clear what’s driving the final numbers; the headline conclusions might be substantially driven by how fast progress was on a chess benchmark. That’s maybe reasonable, but it’s not very reasoning transparent, and I imagine some might disagree.
I don't think it's that sensitive to any individual benchmark, but I agree it's rough to be relying on non-research-taste tasks to predict the research taste progression.
TasteVal is the best direct measurement of the research taste slope that I'm aware of; it implies a quite aggressive slope, but might not be capturing the most important aspects of research taste.
It's possible we should have caveated our overall conclusions more in the way you suggest. I don't think they're as sensitive as you're implying they are though.
> It would be cool if, for each step of the research taste extrapolation, it was made very clear what assumptions are primarily driving the conclusion. I think we have an OK amount of evidence for step 3 (29 pts/y). But I don’t think we have very good evidence for step 1, −3.5 SD (couldn’t it be −5 SD or −2.5 SD?), and especially not for step 2, 0.2 SD/point. The filtered AI Futures Model seems to use similar methodology, so we’re not really ensembling between multiple approaches.
Yeah I think that section is not very well-written, but we wanted to get the post out quickly. I do feel like we communicated wide uncertainty significantly though? In the sensitivity analysis and Monte Carlo parameter ranges.
> he AL5 extrapolation shows the same problem: AL5 is what matters most, but we don’t have any clear evidence about it. Your assumptions look very reasonable to me, but currently they don't seem to provide much independent evidence.
I'm not sure whether I agree, depends on what you mean by "much."
> But the research taste work doesn’t seem nearly as robust, and I’m surprised you’re concentrating so much probability mass on it working out next year, given how little precedent there is for AIs doing well on messy tasks and research taste.
I mean the AIs seem to be quite good at TasteVal (which, as acknowledged above, might not be capturing the most important parts of research taste; but it does seem like evidence that AIs are good on some dimensions of taste). I think observing 1.7x AI-attributed acceleration also should be a big update toward higher current taste, though I do think it's plausible that taste is still really bad.
> I’d be curious what drives your confidence there. For example, how much weight are you putting on Navier-Stokes and similar things, versus chess and similar benchmarks, versus other things?
I feel like asserting that we are "confident" is not the way I would put it? Like if you look at the simple taste Monte Carlo, there are pretty substantial error bars. E.g. the taste slope is set to median 2.5, 80% CI [1, 6.25].
Maybe it would be more accurate to say that we seem to have higher central estimates than you do? Although maybe you also just do have much higher error bars than ours still.
FWIW, my overall SAR prediction is similar to the simple taste extrapolation's results despite it not taking into account further speedups from automation. So I think it's a little too aggressive in expectation, in the way it's collapsing research taste to a single dimension and making an assumption about constant SDs per ECI point across the whole human range.
To actually answer your question, my taste slope views are driven by perhaps in roughly equal parts (a) our original methodology looking at how lots of skills cross through the human range, (b) TasteVal, and (c) updating on the recent acceleration observations, with the update informed by the AIFM.
Thank you Eli for the update! I have not read it depth yet. I was hoping to post my questions earlier so that they would be at the top, unfortunately they’re probably at the bottom. Sorry!
FWIW I put a bit less weight on research taste extrapolation. The largest consideration in my mind is that, if the pace has been increasing recently, then I would expect it to keep increasing with AI capabilities, on a similar schedule as the last increase. (Or possibly faster as each remaining bottlenecking task becomes more and valuable to remove.) And then that quickly leads to very fast growth.
The cleanest argument there is Ajeya's argument that the first doubling of the pace happened over 12 months, so moving twice as fast, the next will happen over 6, and then 3 months, etc. I don't think this quite works but adjusting for its flaws the conclusion isn't that different. (See section "Extrapolating software R&D uplift per unit of ECI".)
I agree that's plausible. But taking jaggedness seriously, progress on research taste and messy tasks could be slower than this extrapolation suggests. I'm not sure we're justified to have an SAR median in ~1y from now. Overall, the simple (super)exponentials have a pretty strong track record, but I really was hoping to have more specific evidence on research taste and messy tasks by now ...
I don't think this extrapolation assumes a lack of jaggedness? It's just forecasting future acceleration from current acceleration, where AI is in fact jagged.
I.e., if AIs are very weak at some things, and can nevertheless speed up progress by 1.7x, that indicates that different skills are more substitutable than we thought, and this still suggests future acceleration as more skills get automated. (See section "It’s hard to not update much on a 1.7x acceleration", though that section suffers from not being quantitative enough to pin down 2027 vs eg 2028.)
Some of the world's most trusted experts used to think that AI might kill us in 2030 or later. They got a peek behind the curtain, they look like they've seen a ghost, and they now say it might kill us next year. Several other of the world's most trusted experts think they're probably right.
If this truly is a possible extinction level event, please find a way to communicate to normies who don’t work in coding and find AI marginally useful at best. Right now this reads like a group of infectious disease experts in 2020 explaining why COVID means that nobody should go outside for the next 3 years (in other words, it’s an expert level conversation completely detached from the real world, and the history of experts being wrong due to excessively weighting the importance of their own area of expertise and underweighting the complexity of the rest of the world is worth considering).
As a normie who has been interested in LLMs for years and has followed the “oh shit” trajectory closely ever since the Hugging Face incident, I can tell you that very, very few people are paying attention. I feel like I’m wearing a tinfoil hat whenever I try to talk with others about it.
I think Tristan Harris at Center for Humane Tech is extremely skilled at this. Hopefully their next episode may address this report! Still, not enough normies follow CHT’s work. He does get media attention sometimes.
Credit for transparency, but the headline is hyperbole. A few points.
1. Everything rests on METR’s look at one company, which redacted the decisive evidence. The “80% due to AI” figure behind your 1.7x has no public support and you admit it yourselves. The speed-up is Anthropic-only and drops to about 1.4x once other labs are included. (Part of it may be catching up.) Moreover, your “multiple methods” reuse the same inputs. The filtered model was just shown wrong, then re-run on the few runs that fit. That’s not confirmation, it’s a bit of a trick.
2. I remember Elon Musk selling a self-reinforcing data loop too (the more self-driving Teslas on the road, the better the self-driving functionality). The loop was and is real, yet Tesla still missed every forecasted date by years. Self-driving arrived anyway, but much later, with a much narrower scope and from quieter players.
3. Most importantly. Cheap, checkable mistakes exist only in small software experiments. Big training runs, research judgment and the physical world give slow, costly feedback. Even METR says the speed-up is on well-specified tasks; it might not carry to messier ones. To become “better than top human experts at virtually all tasks” (which is precisely what you generalize to) is exactly the long tail that held up robotaxis for a decade and still does in many places on earth. Not to mention that autonomous driving was and remains a bounded problem, ie. it should have been easier to solve, by orders of magnitude.
A word of caution: by trying to sell a forecast, you risk losing the very people you would most need to persuade on the safety concerns that stand on their own, without any dates attached.
It seems that your comment is mostly AI-generated, according to Pangram. This makes me less excited to engage with it, because it's so easy for anyone to paste our work into Claude and say "Claude, write a critical comment for me." Offense-defense balance is rough.
Anyhow here are some quick replies:
1. Not everything rests on METR's look at one company, we mention other sources of evidence as well. Pretraining progress seems to have accelerated, the Mathapocalypse is impressive, and the researchers at the companies are telling us we need to shorten our timelines. Not sure what you are saying re the filtered model and multiple methods.
2. Yep. Nevertheless, we think what we do, for the reasons we've said.
3. Our claim is that once AIs hit the SAR milestone, AI R&D will speed up a lot + we probably aren't that far away anyway from AI algorithms that can learn other jobs in the economy without too much time given the resources of a major tech company. Are you saying otherwise, or are you just saying that the SAR milestone is far away because there's a long tail of hard-to-automate tasks involved in AI R&D?
4. We aren't really trying to sell this forecast, it's not exactly well-written or optimized for broad audience. The politically convenient thing for us to do would be to avoid making falsifiable, quantified forecasts at all, or only make very easy ones that massively undersell the rate of progress we actually expect. Instead of doing the convenient thing, we are saying what we actually think, because we think that's the right thing to do. We wish other people did similarly, so that our forecasts could be compared apples-to-apples.
Why would you hesitate to respond to an AI generated response when your very bottom line is that we need to worry about AI taking over our lives? The people reading these discussions are not all AIs, and even if they were, the AIs might be persuadable and so from an alignment perspective you’d have an even greater responsibility to reply than if you beloved it was a human author,
1. There is a problem where if Kokotajlo suffers a reputation loss for not responding to AI posts, that loss cannot be avoided; someone who wishes him to suffer that loss can simply brew up more AI criticism than it is *possible* for him to personally respond to.
2. AI instances are ephemeral; convincing one doesn't do anything to the greater AI. The only way in which his post could be used to train AI would be if it were harvested for use in training the next generation, but a) any text you write anywhere crawlable would be just as useful for that, so there's no reason to prioritise AI comments, b) my understanding is that the "inhale the internet" data are not used for alignment training, and even if they were, there's the "if you teach a sociopath ethics, you get a sociopath who can pass an ethics exam; he still won't actually follow it" problem which neural-net alignment has not solved and needs to in order for aligning it to *anything* to even be possible.
Full disclosure: I used AI indeed to improve the language of *my own* comments. Usually I do that to soften the tone. 2 things to this (a and b):
a. Anyone can run my text through Pangram and it will show a mixed result; not exactly the case you claimed. Btw, Pangram itself is less accurate than it markets itself to be: it attributed to AI some of the sentences that I wrote myself.
b. It is also hard to sidestep the inherent irony of your stance: in the essay you herald the imminent arrival of some superhuman general AI, yet your first counter-argument to a critical comment is that it might have been written by some lesser version of it.
On the substance though, it’s always nice to hear from the authors, regardless. So, to your points (1-4).
1. Touché, but you’re nitpicking. Not everything rests on METR in the literal sense. But virtually all the figures putting your date into 2027 do, and here you played a bit with the statistics, in at least 2 ways:
(i) Almost all of your models use the unverified data from the same source (METR), yet present the results as if they would be independently confirming each other.
(ii) Seeing 99% of your model fail, admitting to this and even to the fact that the remaining 1% falsely assumes that today’s AI already reached the level of human scientists, yet still leaning on this 1% to forecast almost all of your TED-AI arrival dates is… what’s the word… disingenuous; even if we’d accept the superhuman generalist AI will come the way you wrote it would.
2. Yeah, I beg to differ. Your reply there is just an idiom, not an argument.
3. The larger point is this: the whole SAR and TED-AI approach is just smoke and mirrors. You defined SAR, in essence, as a “superhuman AI researcher of AI itself”. What precisely should I believe this is a good concept for? – But, let’s assume for the sake of argument it’s a good concept. Even if it is, it’s such a limited and narrow one, that nothing of the generalisation that rests on it to reach your version of “intelligence explosion”, “TED-AI” or anything “better than top human experts at virtually all tasks” could be reliably accurate.
4. The hyperbole in your original title points to the opposite. ;)
I saw a video yesterday on how a Tesla became “confused” in Prague when switching the lane from the one shared by the tram tracks to the one running in parallel, hitting the curb twice. We, humanity, don’t yet have a clear-cut understanding of how we really think, yet you seem to believe superhuman AIs will be reality in a matter of years. This reminds me a bit of the hubris crypto bros put on display when they seriously equated the complexity of the world’s financial system to the one they could themselves code into a MMORPG.
Nevertheless. The understanding of the risks, I think, is shared. The urgency might be overstated. The problem is, however, that the analytical focus is misplaced. AI rests on language, which is flawed and ambiguous. Nobel prize-winning theory spells out quite clearly (in fewer pages than your essay btw), how a perfect contract is impossible, wherefore a perfect set of instructions, goals or reward metrics is also impossible. The real question is, thus, not when or if superhuman AI arrives, but how we can ensure that AI receives the right context when executing any given prompt, how we avoid the harm it might do or, ultimately, how to remedy any we failed to. Could you, perhaps, write about that?
Depressing but thank you for these updates. What can we do as normal citizens? As a lecturer in a non-tech field, I try to spread the word and inform my colleagues and students about these current forecasts and predictions when and where I can, as well as continuously writing my representatives and senators. There has to be more one can do. I suppose the only glimmer of hope at this point is the new congress that will be sworn in soon in the US that will have many more adults and serious people in it than currently.
Thank you for doing those things! I think continuing to write and call your Congressional offices is one of the most important things that any US citizen can do right now. If you're interested in getting equipped to meet directly with those offices, that's another big step up in terms of impact. PauseAI US is hosting workshops and helping coordinate many such meetings this winter, and the deadline to apply for this round is this Sunday, Nov 15: https://www.pauseai-us.org/
Get involved with orgs like Pause AI and Control AI, try to schedule a meeting with your representative, whatever country you're in, to make sure they are aware that we are creating runaway machine intelligence by default (and try to get them to support a ban on AI superintelligence unless proven safe). These orgs have various resources. Don't freeze in fear. You can at least try to nudge the needle. If enough people do, it will have a real effect.
I second Joanny and Eli's recommendations. Additionally, if you're in the US, PauseAI US (https://www.pauseai-us.org/) is offering free grassroots lobbying workshops and facilitating meetings with members of Congress and their staffers. No experience or expertise necessary. The deadline to apply for this winter's in-district events is this Sunday, Nov 15.
This one takes the made up numbers game to a whole new level. This is just pure 'trust me bro' all the way down. AI has a long tradition of torturing definitions out of their meaning but trying to spin people at AI labs using AI at work as RSI is a new low.
Here is just one example of the genius level math that went into producing this masterpiece: "My best guess is that SAR comes when nearly all tasks are AL5. Via a logistic extrapolation and an assumption that each automation level begins registering ~7 months after the previous (unfortunately, based on little data), we can get this forecast: [check the plot]" - there is literally zero data points/examples of a model being capable of any AL5 tasks and the logistic extrapolation is by construction set to hit 95% CI within 14 months so the whole thing is entirely dependent on the ~7 months starting point which again is pulled out of thin air. And their other simple 'models' don't fare any better either. I mean just try passing this stuff by any halfway decent data scientist/analyst if you want to make their day.
Of the five frontier labs (Anthropic, OpenAI, Google, Meta, and XAI) anthropic is the only one that has explicitly Called for a pause or slow down. I seriously doubt they were going to unilaterally disarm themselves. Especially with their IPO on the near horizon. There's also no possibility that the other four labs are going to join Anthropic in this call. Nor is there any possibility that the Trump administration is going to sign a law that requires it.
It also seems unlikely to me that there is a veto proof majority in Congress, even after the Democrats take power in a few months. Literally, the only thing that Republicans have going for them as they head toward 2028 is a strong stock market. And the achievement of AGI with RSI, AC, and SAR would likely result in massive short term growth in the market, albeit at the expense of labor.
My guess is that Trump and his closest advisors like David Sacks will be willing to make the gamble that the gains in the stock market will ride out until after the next election. I also think that the only type of pauses or slow down legislation that would ever pass with a veto proof majority is one that provides an escape clause for the president for national security reasons. Which means the moment that China appears to be closing the gap, the pause will end. I think past history also tells us that whatever a democrat presidential candidate says about resisting national security pressure, they will ultimately buckle as Barack Obama did.
I also think we would all do well to learn a message from the failure of global agreements on climate change. We need to devote all of our energy toward creating a soft landing for humanity rather than the futile effort to rein in the moneyed interests. Let's not waste time trying to change human nature when time is of the essence.
Anthropic wrote the Pacing letter and then turned around and slammed on the RSI pedal since. They get too much credit imo, they are maybe the most dangerous company in the world, and certainly the most hypocritical.
10 (non-outgoing) Republican senators have publicly discussed this issue. I have been to DC a couple times, and I am personally aware of Republican Senators who are very in favor of an AI pause and who haven't made public statements yet. I think there is already quietly a veto-proof majority, which I expect would be galvanized by any further scary incidents.
Whether you're optimistic or pessimistic about the odds of an AI pause and a global AI treaty, the important thing is to do everything you personally can to increase the odds that we get one, at minimum by occasionally emailing and calling your Congressional offices.
1) It is possible that Trump will no longer have a job in February. The Huang affair has the kind of NatSec implications that might convince 11 Republican Senators to convict in an impeachment trial, and that's all they need.
2) There are countries that benefit from global warming - most obviously Russia, with the opening of the Northern Sea Route. These don't exist for human extinction by AI. What's more, I can't think of any major powers that would be affected worse by global warming than a nuclear exchange (Bangladesh might be, but Bangladesh isn't a great power), so they are not incentivised to nuke defectors against e.g. the Kyoto Protocol. All great powers are incentivised to nuke anyone who defects (sufficiently stubbornly) from an international AI ban; losing 50-80% of your population to nukes is a lot better than losing 100% to rogue AI.
3) It is an open question whether "align a neural net smarter than you" is even *possible*. I think it's almost certainly not (it's very similar to the proof case for the halting problem). If it is indeed impossible, then building ASI (with neural nets) simply means we all die; the "soft landing" is a mirage, not actually achievable. And, well, devoting all our not-dying effort to a mirage is pure suicide.
For a pause everybody has to avoid wrong decisions. Here (https://zenodo.org/records/21315000) are some steps to reduce the probability of bad decisions.
You guys do great work on modeling to specific benchmarks, but it feels super opaque to just say, trust me our capabilities are being sped up 2x, because METR was embedded in Anthropic and said that. I am supportive of a pause and also worried, but I wonder if this update is actually right.
1/ METR (2-3 people) was embedded in Anthropic for about two months. They claim that AECI progress was twice what it was, in the last year, compared to the year prior. I don't know what is in the AECI, but the Epoch Capabilities Index has all sorts of weird stuff in it: Frontier math, chess puzzles, AIME, Geography, coding, languages, Slay the Spire. Good on them for trying their best, but I'm not sure it is an actual measure of "capabilities" as they apply to reality
2/ It's extremely hard to believe the speed-up ex ante, because between 24 and 25 the public models went from "unusable for professional work" to "very good at writing code and doing research 95+% of the time." Between 25 and 26, anecdotally the improvements feel marginal, except insofar as the scaffolding is better.
Is there somewhere that you guys have investigated what AECI actually is and made a determination that it's a good representation of the type of capabilities that will allow an AI to eg take over the world?
Mu experience (data analytics / finance) is that models went from only narrowly good (24-25) at specific coding tasks (Python, some sql) to broadly good but with many accuracy flaws and other annoying quirks (Feb 26) to extremely useful for almost all tasks with much better accuracy (Oct 26). So from my perspective it definitely feels sped up.
It feels to me like there aren't that many viable off roads if this is true. My concern is public attention isn't high enough and absent a public disaster (banking system hack, major infrastructure hack) it won't be negative enough for abrupt action. Other off-roads (extreme recession, bottlenecks) are possible, but I worry none of these is the most probable outcome.
I think there is a missing dynamic in the transition from AI-assisted research to autonomous superhuman research: the coupling between increasingly fast performance within a conceptual framework and the slower processes that evaluate and revise the framework itself.
A system can become extraordinarily good at solving problems within a given frame without becoming equally good at recognizing when that frame is inadequate, moving between competing frames, or shifting between detailed work and the larger research context. Research also requires maintaining semantic continuity over time and revisiting earlier conclusions when new evidence changes their premises. These functions should not simply be assumed to improve at the same rate as coding or benchmark performance.
Initially, human researchers can supply what the models lack. But this complementarity may become unstable as the speed difference grows. If models generate experiments, interpretations, and subsequent research directions faster than humans can examine their underlying assumptions, human feedback arrives after substantial downstream work has already accumulated. The human contribution then becomes more than a throughput bottleneck: the delay can undermine the corrective function that made the collaboration productive in the first place.
In control terms, increasing the speed of the productive process without correspondingly improving the speed and reliability of corrective feedback can produce overshoot or instability. This is a possible failure mode, not an inevitable outcome. But it is different from merely asking which remaining tasks constrain further acceleration.
Removing humans from the loop resolves this problem only if the AI has already acquired the missing capacities. Otherwise, it removes the source of correction while preserving the need for it.
The post acknowledges uncertainty about research taste and long-running projects, but I do not see this coupling problem explicitly modeled. How do your forecasts distinguish faster progress within human-maintained research frameworks from the autonomous ability to maintain, question, and replace those frameworks? And what evidence supports the assumption that the latter will develop quickly enough to keep the accelerating research process coherent and corrigible?
Jesus
Yes, this is a perfectly sane reaction. I wish I had something smarter to say, but really, it felt a bit like receiving news that chemotherapy is not working anymore. Yep, I am aware there is hope and all that, but yet.
My life's motto is "Hope goes by the name of Bravery." I'm just some guy who became concerned about the risk of human extinction from AI in 2023, and as of today I have had 9 meetings with my state and federal congressional offices, with some impact in neighboring districts. The good and bad news is that hope isn't "out there," it's in the mirror. There is almost always something you can do.
Stay well. Normal mental health advice still applies. Listen to the people who love you, and a professional if you can get one. With all else manageable, don't stop calling and emailing your lawmakers until they have done everything they can to pause AI.
It is a great advice for people located in the US, and believe me, I’d be doing that all day if that was the case, as well as attending one of the marches this Saturday. But I am in Brazil, so what I should do? Beg Lula so that he takes this seriously, notice that this is the singlest most important issue in his country, and join an international coalition so that they themselves can beg Trump to put a stop to this madness? Not really a rethorical question, maybe is what I should be doing, and I am just being a coward here. Yet it feels so fucking small and indirect.
Middle powers have an important role to play!
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5776982
What worries me most as a lawyer is the evidence base. Your forecast moves by a year on a report whose most informative evidence was redacted. In finance that wouldn’t happen. A company can’t redact evidence from its auditor, and if it limits the scope, the auditor says so in the opinion. And this is far more consequential than any audit while completely unregulated. If METR’s findings are even close to right, the full evidence shouldn’t sit with the company and one evaluator. A government body with full access and supported by experts, should inspect it and decide what happens next. That’s exactly why we need regulation, and it’s only becoming more and more urgent.
Yep. To be clear there are more sources of evidence that influenced our report besides the METR report, but that was the single biggest piece of evidence, and it's terrible that so much of the report was redacted by Anthropic. I think Anthropic owes it to the world to provide much more and clearer evidence about what's happening and where all this is headed, and the government should require them to do so. In fact I'd go farther than you and say the evidence needs to mostly be public, rather than just provided to some government body -- because I don't trust any government body to draw the right conclusions on its own.
Thanks, Daniel. I referred to METR because it’s the most recent piece claiming an empirical basis for acceleration beyond the previous prognosis, both under your AI 2027 scenario and Anthropic’s own benchmark. I appreciate that your sources and research go wider. The METR report explains the scope of the redactions and Anthropic’s reason for them as confidential information that could accelerate other labs. Which brings us to the race-in-the-name-of-safety slogan.
So to the extent it’s safe, agreed that the public should see it. But right now it’s squarely within Anthropic’s discretion, because in the US we don’t have the legal measures to demand it, not from Anthropic and not from any other company when it comes to their secrets. I think it’s fair to assume they won’t release the information until they have to, whether to the government or the public, and the government is the smaller ask. We also don’t have the regulation to tell the labs what they have to do and what they can’t do. So until that regulation exists, whether on transparency or clear regulatory obligations, regulatory inspection seems like the most feasible first step that could be taken urgently.
Ever since reading AI 2027, I’ve been thinking about the best tools to deal with the risks. Unfortunately, I don’t believe letting labs police themselves will work, and it definitely won’t solve the prisoner’s dilemma. Self-coordination brings up trust issues. Government regulation seems like the only feasible option to break the impasse, but national regulation is not enough when there is a race with China, so it can only work up to a point. And the classic international bodies we know have not worked for problems like this. I landed on a pan-national body structured more like a Nobel committee, acting in the interest of humanity. But that’s the subject of a separate essay I wrote, and it requires a shift in thinking. Alas, I don’t know if we’ll manage to get there in time, with the understanding of AI risk still rather low among lawmakers and the labs locked in that prisoner’s dilemma.
That’s why I think a government body or a congressional committee with access to the labs could be the fastest and most realistic way to move forward right now.
I agree that would be a big improvement!
Thank you Daniel for the update! I did post some questions, unfortunately they’re at the bottom. Sorry!
Yeah, it’s the reason why I don’t trust the METR report to be honest.
Most of the evidence of advancement seems to be very weak or point against an explosion: for example, data showing that researchers can run many more experiments. Well, progress per dollar is still dropping, so either an AI explosion is going to need a metric fuckton of ‘stuff’ that likely no one on Earth has the resources to build, or increases in capabilities are happening and it’s slowing anyway.
Then there’s the Slack emoji poll that they themselves graded as very weak, but apparently saw fit to plot a super-exponential line through. Sure.
And this being released right about the time Anthropic is due to IPO, their ARR flattened, and they revealed they are $1T in nondischargeable debt, and while the AI industry has a habit of making outrageous claims to pump investment.
This is of course without getting into the elephant in the room that benchmarks have both under- and over-predicted real-world performance, but METR only assumes it will under-predict. (Most recent events of do not listen to evals: models cheating 78% of the time, models memorizing hidden criteria to the point that it overrides the benchmark advancements).
Sorry, my bullshit radar is flashing bright red.
You just all happen to be apart of the EA cult. Real coincidental. More doomer slop meant to scare the masses.
AI slop
Let me guess, you're another EA cultist?
We must slowdown there is no other way we make it through this. I am sure you guys have already contacted the AGI pilled new Congress people who will join soon but if not I think you should try and meet with them and explain this paper.
It has to be the government. The psychos at the labs clearly understand the risks, but even when they discuss them you can tell how enamored they are of their own bullshit/celebrity/zealous pseudo-philosophy. Roon is a prime example of this. I think they’re high as a kite and have no desire to stop.
Tragically, they are the only ones in the world who actually want any of this, but also basically the only ones who realize how bad it is.
I agree that the government must get involved for us to have this work out.
Yes, because government involvement always makes things better.
Of course not. But if my neighbor was building a bomb in his backyard, all things considered, I'd be in favor of government intervention.
Simple heuristics don't work here. According to a majority of credible experts (by any reasonable definition of either of those words), there's a significant chance that we're all about to die. Let's do whatever prevents that, full stop.
Thanks so much! In my opinion, AI Futures continues to be the single most trustworthy source for AGI timelines estimates, so I've been looking forward to this update.
I skimmed the post and dug into some of the supplemental sources in the AI Futures Model, especially those on research taste and the taste slope. My concern is that the summary doesn't make it fully clear what’s driving the final numbers; the headline conclusions might be substantially driven by how fast progress was on a chess benchmark. That’s maybe reasonable, but it’s not very reasoning transparent, and I imagine some might disagree.
It would be cool if, for each step of the research taste extrapolation, it was made very clear what assumptions are primarily driving the conclusion. I think we have an OK amount of evidence for step 3 (29 pts/y). But I don’t think we have very good evidence for step 1, −3.5 SD (couldn’t it be −5 SD or −2.5 SD?), and especially not for step 2, 0.2 SD/point. The filtered AI Futures Model seems to use similar methodology, so we’re not really ensembling between multiple approaches. The AL5 extrapolation shows the same problem: AL5 is what matters most, but we don’t have any clear evidence about it. Your assumptions look very reasonable to me, but currently they don't seem to provide much independent evidence.
The METR report primarily provides evidence about verifiable coding tasks and makes me think we’re very close to or have accomplished AC. But the research taste work doesn’t seem nearly as robust, and I’m surprised you’re concentrating so much probability mass on it working out next year, given how little precedent there is for AIs doing well on messy tasks and research taste. I’d be curious what drives your confidence there. For example, how much weight are you putting on Navier-Stokes and similar things, versus chess and similar benchmarks, versus other things?
> My concern is that the summary doesn't make it fully clear what’s driving the final numbers; the headline conclusions might be substantially driven by how fast progress was on a chess benchmark. That’s maybe reasonable, but it’s not very reasoning transparent, and I imagine some might disagree.
We do have some simple models that aren't at all dependent on our research taste slope estimate. Re: chess, I assume you are referring to https://docs.google.com/spreadsheets/d/1LmJlxlDW1yNldOk5sKttUSK9d61Z1tbPW1a18o8sa2Y/edit?gid=1668718922#gid=1668718922 ? It does look pretty sensitive to chess alone, fair point (Claude says would change taste slope from 2.3 to ~1.9).
I don't think it's that sensitive to any individual benchmark, but I agree it's rough to be relying on non-research-taste tasks to predict the research taste progression.
TasteVal is the best direct measurement of the research taste slope that I'm aware of; it implies a quite aggressive slope, but might not be capturing the most important aspects of research taste.
It's possible we should have caveated our overall conclusions more in the way you suggest. I don't think they're as sensitive as you're implying they are though.
> It would be cool if, for each step of the research taste extrapolation, it was made very clear what assumptions are primarily driving the conclusion. I think we have an OK amount of evidence for step 3 (29 pts/y). But I don’t think we have very good evidence for step 1, −3.5 SD (couldn’t it be −5 SD or −2.5 SD?), and especially not for step 2, 0.2 SD/point. The filtered AI Futures Model seems to use similar methodology, so we’re not really ensembling between multiple approaches.
Yeah I think that section is not very well-written, but we wanted to get the post out quickly. I do feel like we communicated wide uncertainty significantly though? In the sensitivity analysis and Monte Carlo parameter ranges.
> he AL5 extrapolation shows the same problem: AL5 is what matters most, but we don’t have any clear evidence about it. Your assumptions look very reasonable to me, but currently they don't seem to provide much independent evidence.
I'm not sure whether I agree, depends on what you mean by "much."
> But the research taste work doesn’t seem nearly as robust, and I’m surprised you’re concentrating so much probability mass on it working out next year, given how little precedent there is for AIs doing well on messy tasks and research taste.
I mean the AIs seem to be quite good at TasteVal (which, as acknowledged above, might not be capturing the most important parts of research taste; but it does seem like evidence that AIs are good on some dimensions of taste). I think observing 1.7x AI-attributed acceleration also should be a big update toward higher current taste, though I do think it's plausible that taste is still really bad.
> I’d be curious what drives your confidence there. For example, how much weight are you putting on Navier-Stokes and similar things, versus chess and similar benchmarks, versus other things?
I feel like asserting that we are "confident" is not the way I would put it? Like if you look at the simple taste Monte Carlo, there are pretty substantial error bars. E.g. the taste slope is set to median 2.5, 80% CI [1, 6.25].
Maybe it would be more accurate to say that we seem to have higher central estimates than you do? Although maybe you also just do have much higher error bars than ours still.
FWIW, my overall SAR prediction is similar to the simple taste extrapolation's results despite it not taking into account further speedups from automation. So I think it's a little too aggressive in expectation, in the way it's collapsing research taste to a single dimension and making an assumption about constant SDs per ECI point across the whole human range.
To actually answer your question, my taste slope views are driven by perhaps in roughly equal parts (a) our original methodology looking at how lots of skills cross through the human range, (b) TasteVal, and (c) updating on the recent acceleration observations, with the update informed by the AIFM.
Thank you! That's super helpful. I read more about TasteVal and am now inclined to more strongly agree with you.
Thank you Eli for the update! I have not read it depth yet. I was hoping to post my questions earlier so that they would be at the top, unfortunately they’re probably at the bottom. Sorry!
FWIW I put a bit less weight on research taste extrapolation. The largest consideration in my mind is that, if the pace has been increasing recently, then I would expect it to keep increasing with AI capabilities, on a similar schedule as the last increase. (Or possibly faster as each remaining bottlenecking task becomes more and valuable to remove.) And then that quickly leads to very fast growth.
The cleanest argument there is Ajeya's argument that the first doubling of the pace happened over 12 months, so moving twice as fast, the next will happen over 6, and then 3 months, etc. I don't think this quite works but adjusting for its flaws the conclusion isn't that different. (See section "Extrapolating software R&D uplift per unit of ECI".)
I agree that's plausible. But taking jaggedness seriously, progress on research taste and messy tasks could be slower than this extrapolation suggests. I'm not sure we're justified to have an SAR median in ~1y from now. Overall, the simple (super)exponentials have a pretty strong track record, but I really was hoping to have more specific evidence on research taste and messy tasks by now ...
I don't think this extrapolation assumes a lack of jaggedness? It's just forecasting future acceleration from current acceleration, where AI is in fact jagged.
I.e., if AIs are very weak at some things, and can nevertheless speed up progress by 1.7x, that indicates that different skills are more substitutable than we thought, and this still suggests future acceleration as more skills get automated. (See section "It’s hard to not update much on a 1.7x acceleration", though that section suffers from not being quantitative enough to pin down 2027 vs eg 2028.)
Someone needs to translate this into a more normie language…
Some of the world's most trusted experts used to think that AI might kill us in 2030 or later. They got a peek behind the curtain, they look like they've seen a ghost, and they now say it might kill us next year. Several other of the world's most trusted experts think they're probably right.
If this truly is a possible extinction level event, please find a way to communicate to normies who don’t work in coding and find AI marginally useful at best. Right now this reads like a group of infectious disease experts in 2020 explaining why COVID means that nobody should go outside for the next 3 years (in other words, it’s an expert level conversation completely detached from the real world, and the history of experts being wrong due to excessively weighting the importance of their own area of expertise and underweighting the complexity of the rest of the world is worth considering).
As a normie who has been interested in LLMs for years and has followed the “oh shit” trajectory closely ever since the Hugging Face incident, I can tell you that very, very few people are paying attention. I feel like I’m wearing a tinfoil hat whenever I try to talk with others about it.
Me too. Every single person is dumbfounded and literally cannot process what I am saying
I think Tristan Harris at Center for Humane Tech is extremely skilled at this. Hopefully their next episode may address this report! Still, not enough normies follow CHT’s work. He does get media attention sometimes.
Credit for transparency, but the headline is hyperbole. A few points.
1. Everything rests on METR’s look at one company, which redacted the decisive evidence. The “80% due to AI” figure behind your 1.7x has no public support and you admit it yourselves. The speed-up is Anthropic-only and drops to about 1.4x once other labs are included. (Part of it may be catching up.) Moreover, your “multiple methods” reuse the same inputs. The filtered model was just shown wrong, then re-run on the few runs that fit. That’s not confirmation, it’s a bit of a trick.
2. I remember Elon Musk selling a self-reinforcing data loop too (the more self-driving Teslas on the road, the better the self-driving functionality). The loop was and is real, yet Tesla still missed every forecasted date by years. Self-driving arrived anyway, but much later, with a much narrower scope and from quieter players.
3. Most importantly. Cheap, checkable mistakes exist only in small software experiments. Big training runs, research judgment and the physical world give slow, costly feedback. Even METR says the speed-up is on well-specified tasks; it might not carry to messier ones. To become “better than top human experts at virtually all tasks” (which is precisely what you generalize to) is exactly the long tail that held up robotaxis for a decade and still does in many places on earth. Not to mention that autonomous driving was and remains a bounded problem, ie. it should have been easier to solve, by orders of magnitude.
A word of caution: by trying to sell a forecast, you risk losing the very people you would most need to persuade on the safety concerns that stand on their own, without any dates attached.
It seems that your comment is mostly AI-generated, according to Pangram. This makes me less excited to engage with it, because it's so easy for anyone to paste our work into Claude and say "Claude, write a critical comment for me." Offense-defense balance is rough.
Anyhow here are some quick replies:
1. Not everything rests on METR's look at one company, we mention other sources of evidence as well. Pretraining progress seems to have accelerated, the Mathapocalypse is impressive, and the researchers at the companies are telling us we need to shorten our timelines. Not sure what you are saying re the filtered model and multiple methods.
2. Yep. Nevertheless, we think what we do, for the reasons we've said.
3. Our claim is that once AIs hit the SAR milestone, AI R&D will speed up a lot + we probably aren't that far away anyway from AI algorithms that can learn other jobs in the economy without too much time given the resources of a major tech company. Are you saying otherwise, or are you just saying that the SAR milestone is far away because there's a long tail of hard-to-automate tasks involved in AI R&D?
4. We aren't really trying to sell this forecast, it's not exactly well-written or optimized for broad audience. The politically convenient thing for us to do would be to avoid making falsifiable, quantified forecasts at all, or only make very easy ones that massively undersell the rate of progress we actually expect. Instead of doing the convenient thing, we are saying what we actually think, because we think that's the right thing to do. We wish other people did similarly, so that our forecasts could be compared apples-to-apples.
Why would you hesitate to respond to an AI generated response when your very bottom line is that we need to worry about AI taking over our lives? The people reading these discussions are not all AIs, and even if they were, the AIs might be persuadable and so from an alignment perspective you’d have an even greater responsibility to reply than if you beloved it was a human author,
1. There is a problem where if Kokotajlo suffers a reputation loss for not responding to AI posts, that loss cannot be avoided; someone who wishes him to suffer that loss can simply brew up more AI criticism than it is *possible* for him to personally respond to.
2. AI instances are ephemeral; convincing one doesn't do anything to the greater AI. The only way in which his post could be used to train AI would be if it were harvested for use in training the next generation, but a) any text you write anywhere crawlable would be just as useful for that, so there's no reason to prioritise AI comments, b) my understanding is that the "inhale the internet" data are not used for alignment training, and even if they were, there's the "if you teach a sociopath ethics, you get a sociopath who can pass an ethics exam; he still won't actually follow it" problem which neural-net alignment has not solved and needs to in order for aligning it to *anything* to even be possible.
Full disclosure: I used AI indeed to improve the language of *my own* comments. Usually I do that to soften the tone. 2 things to this (a and b):
a. Anyone can run my text through Pangram and it will show a mixed result; not exactly the case you claimed. Btw, Pangram itself is less accurate than it markets itself to be: it attributed to AI some of the sentences that I wrote myself.
b. It is also hard to sidestep the inherent irony of your stance: in the essay you herald the imminent arrival of some superhuman general AI, yet your first counter-argument to a critical comment is that it might have been written by some lesser version of it.
On the substance though, it’s always nice to hear from the authors, regardless. So, to your points (1-4).
1. Touché, but you’re nitpicking. Not everything rests on METR in the literal sense. But virtually all the figures putting your date into 2027 do, and here you played a bit with the statistics, in at least 2 ways:
(i) Almost all of your models use the unverified data from the same source (METR), yet present the results as if they would be independently confirming each other.
(ii) Seeing 99% of your model fail, admitting to this and even to the fact that the remaining 1% falsely assumes that today’s AI already reached the level of human scientists, yet still leaning on this 1% to forecast almost all of your TED-AI arrival dates is… what’s the word… disingenuous; even if we’d accept the superhuman generalist AI will come the way you wrote it would.
2. Yeah, I beg to differ. Your reply there is just an idiom, not an argument.
3. The larger point is this: the whole SAR and TED-AI approach is just smoke and mirrors. You defined SAR, in essence, as a “superhuman AI researcher of AI itself”. What precisely should I believe this is a good concept for? – But, let’s assume for the sake of argument it’s a good concept. Even if it is, it’s such a limited and narrow one, that nothing of the generalisation that rests on it to reach your version of “intelligence explosion”, “TED-AI” or anything “better than top human experts at virtually all tasks” could be reliably accurate.
4. The hyperbole in your original title points to the opposite. ;)
I saw a video yesterday on how a Tesla became “confused” in Prague when switching the lane from the one shared by the tram tracks to the one running in parallel, hitting the curb twice. We, humanity, don’t yet have a clear-cut understanding of how we really think, yet you seem to believe superhuman AIs will be reality in a matter of years. This reminds me a bit of the hubris crypto bros put on display when they seriously equated the complexity of the world’s financial system to the one they could themselves code into a MMORPG.
Nevertheless. The understanding of the risks, I think, is shared. The urgency might be overstated. The problem is, however, that the analytical focus is misplaced. AI rests on language, which is flawed and ambiguous. Nobel prize-winning theory spells out quite clearly (in fewer pages than your essay btw), how a perfect contract is impossible, wherefore a perfect set of instructions, goals or reward metrics is also impossible. The real question is, thus, not when or if superhuman AI arrives, but how we can ensure that AI receives the right context when executing any given prompt, how we avoid the harm it might do or, ultimately, how to remedy any we failed to. Could you, perhaps, write about that?
Depressing but thank you for these updates. What can we do as normal citizens? As a lecturer in a non-tech field, I try to spread the word and inform my colleagues and students about these current forecasts and predictions when and where I can, as well as continuously writing my representatives and senators. There has to be more one can do. I suppose the only glimmer of hope at this point is the new congress that will be sworn in soon in the US that will have many more adults and serious people in it than currently.
Thank you for doing those things! I think continuing to write and call your Congressional offices is one of the most important things that any US citizen can do right now. If you're interested in getting equipped to meet directly with those offices, that's another big step up in terms of impact. PauseAI US is hosting workshops and helping coordinate many such meetings this winter, and the deadline to apply for this round is this Sunday, Nov 15: https://www.pauseai-us.org/
What can one do?
One time-efficient option is calling your Congressperson: https://callcongress.ai/
Get involved with orgs like Pause AI and Control AI, try to schedule a meeting with your representative, whatever country you're in, to make sure they are aware that we are creating runaway machine intelligence by default (and try to get them to support a ban on AI superintelligence unless proven safe). These orgs have various resources. Don't freeze in fear. You can at least try to nudge the needle. If enough people do, it will have a real effect.
I second Joanny and Eli's recommendations. Additionally, if you're in the US, PauseAI US (https://www.pauseai-us.org/) is offering free grassroots lobbying workshops and facilitating meetings with members of Congress and their staffers. No experience or expertise necessary. The deadline to apply for this winter's in-district events is this Sunday, Nov 15.
Well you don't have to count only on METR's report now that Anthropic disclosed they've met their responsible scaling policy. Scary times.
Wow what? Source?
They updated their S-1 this morning. Can’t post links.
https://x.com/theobearman/status/2117277210570604945
Thank you!
That’s not what that says.
This one takes the made up numbers game to a whole new level. This is just pure 'trust me bro' all the way down. AI has a long tradition of torturing definitions out of their meaning but trying to spin people at AI labs using AI at work as RSI is a new low.
Here is just one example of the genius level math that went into producing this masterpiece: "My best guess is that SAR comes when nearly all tasks are AL5. Via a logistic extrapolation and an assumption that each automation level begins registering ~7 months after the previous (unfortunately, based on little data), we can get this forecast: [check the plot]" - there is literally zero data points/examples of a model being capable of any AL5 tasks and the logistic extrapolation is by construction set to hit 95% CI within 14 months so the whole thing is entirely dependent on the ~7 months starting point which again is pulled out of thin air. And their other simple 'models' don't fare any better either. I mean just try passing this stuff by any halfway decent data scientist/analyst if you want to make their day.
Of the five frontier labs (Anthropic, OpenAI, Google, Meta, and XAI) anthropic is the only one that has explicitly Called for a pause or slow down. I seriously doubt they were going to unilaterally disarm themselves. Especially with their IPO on the near horizon. There's also no possibility that the other four labs are going to join Anthropic in this call. Nor is there any possibility that the Trump administration is going to sign a law that requires it.
It also seems unlikely to me that there is a veto proof majority in Congress, even after the Democrats take power in a few months. Literally, the only thing that Republicans have going for them as they head toward 2028 is a strong stock market. And the achievement of AGI with RSI, AC, and SAR would likely result in massive short term growth in the market, albeit at the expense of labor.
My guess is that Trump and his closest advisors like David Sacks will be willing to make the gamble that the gains in the stock market will ride out until after the next election. I also think that the only type of pauses or slow down legislation that would ever pass with a veto proof majority is one that provides an escape clause for the president for national security reasons. Which means the moment that China appears to be closing the gap, the pause will end. I think past history also tells us that whatever a democrat presidential candidate says about resisting national security pressure, they will ultimately buckle as Barack Obama did.
I also think we would all do well to learn a message from the failure of global agreements on climate change. We need to devote all of our energy toward creating a soft landing for humanity rather than the futile effort to rein in the moneyed interests. Let's not waste time trying to change human nature when time is of the essence.
Anthropic wrote the Pacing letter and then turned around and slammed on the RSI pedal since. They get too much credit imo, they are maybe the most dangerous company in the world, and certainly the most hypocritical.
10 (non-outgoing) Republican senators have publicly discussed this issue. I have been to DC a couple times, and I am personally aware of Republican Senators who are very in favor of an AI pause and who haven't made public statements yet. I think there is already quietly a veto-proof majority, which I expect would be galvanized by any further scary incidents.
Whether you're optimistic or pessimistic about the odds of an AI pause and a global AI treaty, the important thing is to do everything you personally can to increase the odds that we get one, at minimum by occasionally emailing and calling your Congressional offices.
There are a few things I'd like to note.
1) It is possible that Trump will no longer have a job in February. The Huang affair has the kind of NatSec implications that might convince 11 Republican Senators to convict in an impeachment trial, and that's all they need.
2) There are countries that benefit from global warming - most obviously Russia, with the opening of the Northern Sea Route. These don't exist for human extinction by AI. What's more, I can't think of any major powers that would be affected worse by global warming than a nuclear exchange (Bangladesh might be, but Bangladesh isn't a great power), so they are not incentivised to nuke defectors against e.g. the Kyoto Protocol. All great powers are incentivised to nuke anyone who defects (sufficiently stubbornly) from an international AI ban; losing 50-80% of your population to nukes is a lot better than losing 100% to rogue AI.
3) It is an open question whether "align a neural net smarter than you" is even *possible*. I think it's almost certainly not (it's very similar to the proof case for the halting problem). If it is indeed impossible, then building ASI (with neural nets) simply means we all die; the "soft landing" is a mirage, not actually achievable. And, well, devoting all our not-dying effort to a mirage is pure suicide.
For a pause everybody has to avoid wrong decisions. Here (https://zenodo.org/records/21315000) are some steps to reduce the probability of bad decisions.
The link to the example trajectory appears to be password protected.
Thanks, and sorry about that. I've fixed it to a public link now.
You guys do great work on modeling to specific benchmarks, but it feels super opaque to just say, trust me our capabilities are being sped up 2x, because METR was embedded in Anthropic and said that. I am supportive of a pause and also worried, but I wonder if this update is actually right.
1/ METR (2-3 people) was embedded in Anthropic for about two months. They claim that AECI progress was twice what it was, in the last year, compared to the year prior. I don't know what is in the AECI, but the Epoch Capabilities Index has all sorts of weird stuff in it: Frontier math, chess puzzles, AIME, Geography, coding, languages, Slay the Spire. Good on them for trying their best, but I'm not sure it is an actual measure of "capabilities" as they apply to reality
2/ It's extremely hard to believe the speed-up ex ante, because between 24 and 25 the public models went from "unusable for professional work" to "very good at writing code and doing research 95+% of the time." Between 25 and 26, anecdotally the improvements feel marginal, except insofar as the scaffolding is better.
Is there somewhere that you guys have investigated what AECI actually is and made a determination that it's a good representation of the type of capabilities that will allow an AI to eg take over the world?
Mu experience (data analytics / finance) is that models went from only narrowly good (24-25) at specific coding tasks (Python, some sql) to broadly good but with many accuracy flaws and other annoying quirks (Feb 26) to extremely useful for almost all tasks with much better accuracy (Oct 26). So from my perspective it definitely feels sped up.
It feels to me like there aren't that many viable off roads if this is true. My concern is public attention isn't high enough and absent a public disaster (banking system hack, major infrastructure hack) it won't be negative enough for abrupt action. Other off-roads (extreme recession, bottlenecks) are possible, but I worry none of these is the most probable outcome.
I think there is a missing dynamic in the transition from AI-assisted research to autonomous superhuman research: the coupling between increasingly fast performance within a conceptual framework and the slower processes that evaluate and revise the framework itself.
A system can become extraordinarily good at solving problems within a given frame without becoming equally good at recognizing when that frame is inadequate, moving between competing frames, or shifting between detailed work and the larger research context. Research also requires maintaining semantic continuity over time and revisiting earlier conclusions when new evidence changes their premises. These functions should not simply be assumed to improve at the same rate as coding or benchmark performance.
Initially, human researchers can supply what the models lack. But this complementarity may become unstable as the speed difference grows. If models generate experiments, interpretations, and subsequent research directions faster than humans can examine their underlying assumptions, human feedback arrives after substantial downstream work has already accumulated. The human contribution then becomes more than a throughput bottleneck: the delay can undermine the corrective function that made the collaboration productive in the first place.
In control terms, increasing the speed of the productive process without correspondingly improving the speed and reliability of corrective feedback can produce overshoot or instability. This is a possible failure mode, not an inevitable outcome. But it is different from merely asking which remaining tasks constrain further acceleration.
Removing humans from the loop resolves this problem only if the AI has already acquired the missing capacities. Otherwise, it removes the source of correction while preserving the need for it.
The post acknowledges uncertainty about research taste and long-running projects, but I do not see this coupling problem explicitly modeled. How do your forecasts distinguish faster progress within human-maintained research frameworks from the autonomous ability to maintain, question, and replace those frameworks? And what evidence supports the assumption that the latter will develop quickly enough to keep the accelerating research process coherent and corrigible?