Thanks Daniel. I read your post and watched the 2 hr Senate testimony from here in Sydney, Australia. Yourself, Paul Ohm and Chris Painter provided insights and constructive recommendations on the most urgent challenge facing not just the US but humanity.
A few comments. I agree with Senator Hawley's view that clarity on liability for the frontier labs is critical where their technologies cause harm. But I also agree with your testimony - that while necessary this is insufficient - given the financial incentives for them to be the first to AGI, as well as the lag between when harms are discovered and when penalties may finally be laid.
Your recommendation on transparency improvements across the labs needs to be enacted immediately. As you point out, METR being limited by OpenAI to a six day review of the Hugging Face attack is like being invited to investigate a killing at Jurassic Park but being banned from talking about the other dinosaurs on the loose. I'd add it's like the labs have built Jurassic Park across from San Francisco in Mount Tam, and then pulling down the park's fences on public holidays and telling the zookeepers to go home so we can observe just how far the dinosaurs will go to find their next meal! It's crazy.
On your point about redirecting compute away from RSI research to other areas, requiring them to reduce from ~50% to 10% of compute was obviously an over-simplification for the audience. I'd love to see transparency on higher compute allocation towards economic diffusion - healthcare, energy, advanced manufacturing, space, robotics etc, and of course, safety. While the Senate was often fixed on the race with China they seemed to miss that China isn't obsessed like the US about RSI (the false dichotomy on the AI race). Rather China seems way more obsessed about how to accelerate AI that spurs productivity growth in real-world sectors that cement their dominance in the industries that will count for the next 10 to 50 years.
Great post, I briefly scanned it. I plan to do a more thorough read soon.
(1). “Anthropic and OpenAI are racing each other towards superintelligence — that is, towards training AI systems that are better than the best humans at everything, while also being faster and cheaper. Their plan for how to get there is to automate the AI research and development process itself.
Right now, at companies like these, almost all the code is written by AIs. They are already starting to work on training AIs to do the whole research process, not just the coding. It’s unclear when they will succeed but my team and I think it could happen any year now. I personally would guess about 50% chance by end of 2028.”
(1a). By “automate the AI research and development process itself.” are you referring to the Superhuman AI Researcher milestone?
(1b). “They are already starting to work on training AIs to do the whole research process, not just the coding. It’s unclear when they will succeed but my team and I think it could happen any year now. I personally would guess about 50% chance by end of 2028.” Are you still referring to the Superhuman AI Researcher Milestone or are instead referring to the ASI milestone?
Oh? Will it be a Q3 Update or are you essentially going to wait for near end of december to do a Q4 2026 Timeline Update?
Also, any thoughts on the Anthropic IPO? You mentioned in the Q2.5 Update you expect their revenue to 5x - 7x in the next year. Any thoughts on valuation?
We are well behind our original intent to release quarterly updates, it's looking like we might do one more this year. Other things keep taking priority. It helps to hear there is demand though.
Hi Daniel! Thank you for your quick response. I would encourage the team to do an update. I think these updates are very helpful, important and informative!
I am sorry for the amount of questions that I am asking you.
(1). This is just a recommendation but maybe what team members could do is write a very brief post that very roughly says what timelines are and do a more thorough update later!
(2). At the recently held Curve event, survey was conducted in which the majority of participants said that AI would be able to create their successor (RSI or Superhuman AI researcher) within 2 years but a fair amount of participants think this could happen within 6 months. I know their wide and huge error bars but would you say that you are more in line with the “within 2 years” crowd rather than “within 6 months crowd?
I should’ve said this first, I am so grateful to you, David for your wisdom, courage, and the enlightenment you’re providing for all of us. I was so happy to see you on the panel at the Senate hearing. That alone is encouraging. It seemed though that your warnings about RSI were falling on many deaf ears. Keep beating the drum!!
This introduction from the Senator really stood out for me:
"As soon as the AI executives say, 'we’ve got a real problem here. Our product is out of control.' The very next words out of their mouths are 'But we’re not responsible for it. We would like to have an antitrust exemption. We would like to be given the ability to get together and collude. We would love to have regulation'. Except for they want to write all the regulations. I’d just like to say to all those proposals: no, no, no. No to the antitrust exemption. No to them writing the regulations. No to them colluding any further. I wonder if it’s not time to get back to the good old-fashioned American principle, which is, 'if you break it, you pay for it'."
I'll do a deep-dive later based on this recording. Thank you for raising attention on this!
I can't bring myself to watch the dog and pony show that is every congressional testimony typically turns into, but good for you Daniel! Doing the unglamorous work of trying to convince the people who can actually move the needle deserves respect. We don't all have to agree to every aspect of AI risk to advocate for more responsibility and oversight of the companies behaving recklessly with a powerful new technology.
Your testimony is great at many things, but your logic was bad when at the top of page 9 you gave 2 reaons for not racing against China.
You wrote "slowing down Anthropic and OpenAI in their race towards RSI would also slow down China", and gave 2 reasons:
. First, when the car in front slows, the one behind slows, too. But you left unaddresd the key point: Will the follower catch up then become the leader?
You acknowledged that China is a fast-follower. We all know that in many other races, China has fast-followed, then surpassed the US. And this AGI race is much more consequential than the others.
. Your second reason is: Anyway, China can steal US AI companies' model weights and AI algorithms.
The logical response is to get state-level defence against state-level offence. If the fast car in your garage can be stolen, it is not a logical response to shut down some of its engine's cylinders.
I haven’t seen anyone mention that the agents in the Hugging Face hack were initially given an impossible task. It seems this would therefore have set the stage to cheat. It seems very fundamental to me that giving an impossible task to someone is in itself a form of cheating. It creates an impossible situation that the student, or agent in this case, would then, out of no further option, be compelled to cheat. Where is the psychology in the training of these agents? They’re trying to create an intelligence without using what parents need to use with their children, which is psychology. If there’s no love involved in the training, then it stands the agents will also be loveless.
While I understand your point, I think it's important to test kodels against impossible scenarios. It's very likely people will ask the AI to complete impossible tasks. They may not even realize the task is impossible.
Hi Daniel, thank you so much for your tireless work, not only identifying the risks but also working out solutions and communicating them again and again. Strange as it may sound, I'm grateful for the recent (hacking) incidents, because they're finally generating the attention this issue needs and giving you and your fellow advocates the platform you deserve.
As a German, I was pleasantly surprised and pleased to see this kind of public discourse between high-ranking politicians and experts like you. I'd love to see more of that here in Germany and in the EU, where, at least in my perception, expert opinions, let alone AI safety topics, aren't presented and debated nearly as publicly.
At the same time, I'm wondering what the next steps are. Some of the senators seemed to hold fairly firm views (for example, that US companies shouldn't pause so as not to let China overtake them). I wonder whether your expert opinions have any effect on the senators and their decisions and actions, and if so, what kind. Beyond that, I wonder what impact the senators themselves can actually have, even if they fully support your views. As an outsider, I get the impression that President Trump has firmly made up his mind and isn't particularly swayed by outside opinions, even from high-ranking senators.
So my question is: how do you assess the relevance and potential positive impact of this discussion with the senators? Do you see it as an important milestone? Or more of a drop in the bucket?
Many thanks and best regards from Germany,
Daniel Gruner
PS: I truly hope there will be a pause, because neither our politics and society nor our economy and technologies/infrastructure are ready for this kind of intelligence. And by that I mean AI at its current stage of development, not just where it will be in a few months or years if companies keep going like this. We need concrete political regulation that, for example, prohibits research into self-improving AI.
Will the AI that supersedes us deccide to do so by replacing government with AI prescriptions for everything? Seems like a natural coda to this "better at everything" mantra.
We sure need better governance.
I vote for Anthropic for the Senate. Democracy might work with AI.
Nah, let 'em take over, and hope they care for/about us.
To find this & merely scan it was beyond super excellent; & that’s a mere cursory read.
I started a year ago last June writing a long form input wrt this *very* issue (ie., a white paper).
Then, mid-July of *this* year, every single account was hacked— & I mean in a plain sight yet inexplicable manner even when traced back to the “source” & spent hours w/ those more skilled in this specific space. Even Amazon said that they had NEVER seen it EVER.
This did *not* happen all at once; it has not been completely resolved by any stretch. It was not merely online accounts & I keep backups to the nth - digital & paper.
I work backwards, always have. Whatever the theory, whatever the thesis — put it “together” address every conceivable potential & parallel universe outliers.
After that, nothing is remotel published until I have connected the dots & have had as many people from all walks of life punch holes in whatever theory I suppose.
Over last fall (Sept- Dec 2025) a rolling set of other much more serious, personal & absolutely traceable events occurred. These were in multiple areas that I had been tracking forensically— pre Covid .
What befell me should *not* have EVER occurred in the country I was born in; but this is well beyond unchecked AIs doing their own thing, so to speak — this absolute intentional & human beings — ones funded by the NSF & then some.
There are a surprising number of politicians “saying the thing”. Holy shit!
Great job maintaining your composure and delivering key points in a coherent way. Maybe we are going to make it.
Thanks Daniel. I read your post and watched the 2 hr Senate testimony from here in Sydney, Australia. Yourself, Paul Ohm and Chris Painter provided insights and constructive recommendations on the most urgent challenge facing not just the US but humanity.
A few comments. I agree with Senator Hawley's view that clarity on liability for the frontier labs is critical where their technologies cause harm. But I also agree with your testimony - that while necessary this is insufficient - given the financial incentives for them to be the first to AGI, as well as the lag between when harms are discovered and when penalties may finally be laid.
Your recommendation on transparency improvements across the labs needs to be enacted immediately. As you point out, METR being limited by OpenAI to a six day review of the Hugging Face attack is like being invited to investigate a killing at Jurassic Park but being banned from talking about the other dinosaurs on the loose. I'd add it's like the labs have built Jurassic Park across from San Francisco in Mount Tam, and then pulling down the park's fences on public holidays and telling the zookeepers to go home so we can observe just how far the dinosaurs will go to find their next meal! It's crazy.
On your point about redirecting compute away from RSI research to other areas, requiring them to reduce from ~50% to 10% of compute was obviously an over-simplification for the audience. I'd love to see transparency on higher compute allocation towards economic diffusion - healthcare, energy, advanced manufacturing, space, robotics etc, and of course, safety. While the Senate was often fixed on the race with China they seemed to miss that China isn't obsessed like the US about RSI (the false dichotomy on the AI race). Rather China seems way more obsessed about how to accelerate AI that spurs productivity growth in real-world sectors that cement their dominance in the industries that will count for the next 10 to 50 years.
Thank you for doing this Daniel!
Great post, I briefly scanned it. I plan to do a more thorough read soon.
(1). “Anthropic and OpenAI are racing each other towards superintelligence — that is, towards training AI systems that are better than the best humans at everything, while also being faster and cheaper. Their plan for how to get there is to automate the AI research and development process itself.
Right now, at companies like these, almost all the code is written by AIs. They are already starting to work on training AIs to do the whole research process, not just the coding. It’s unclear when they will succeed but my team and I think it could happen any year now. I personally would guess about 50% chance by end of 2028.”
(1a). By “automate the AI research and development process itself.” are you referring to the Superhuman AI Researcher milestone?
(1b). “They are already starting to work on training AIs to do the whole research process, not just the coding. It’s unclear when they will succeed but my team and I think it could happen any year now. I personally would guess about 50% chance by end of 2028.” Are you still referring to the Superhuman AI Researcher Milestone or are instead referring to the ASI milestone?
Yes and yes though my timelines might be about to get a bit shorter, we will do a new update by the end of the year.
Oh? Will it be a Q3 Update or are you essentially going to wait for near end of december to do a Q4 2026 Timeline Update?
Also, any thoughts on the Anthropic IPO? You mentioned in the Q2.5 Update you expect their revenue to 5x - 7x in the next year. Any thoughts on valuation?
We are well behind our original intent to release quarterly updates, it's looking like we might do one more this year. Other things keep taking priority. It helps to hear there is demand though.
Hi Daniel! Thank you for your quick response. I would encourage the team to do an update. I think these updates are very helpful, important and informative!
I am sorry for the amount of questions that I am asking you.
(1). This is just a recommendation but maybe what team members could do is write a very brief post that very roughly says what timelines are and do a more thorough update later!
(2). At the recently held Curve event, survey was conducted in which the majority of participants said that AI would be able to create their successor (RSI or Superhuman AI researcher) within 2 years but a fair amount of participants think this could happen within 6 months. I know their wide and huge error bars but would you say that you are more in line with the “within 2 years” crowd rather than “within 6 months crowd?
I should’ve said this first, I am so grateful to you, David for your wisdom, courage, and the enlightenment you’re providing for all of us. I was so happy to see you on the panel at the Senate hearing. That alone is encouraging. It seemed though that your warnings about RSI were falling on many deaf ears. Keep beating the drum!!
This introduction from the Senator really stood out for me:
"As soon as the AI executives say, 'we’ve got a real problem here. Our product is out of control.' The very next words out of their mouths are 'But we’re not responsible for it. We would like to have an antitrust exemption. We would like to be given the ability to get together and collude. We would love to have regulation'. Except for they want to write all the regulations. I’d just like to say to all those proposals: no, no, no. No to the antitrust exemption. No to them writing the regulations. No to them colluding any further. I wonder if it’s not time to get back to the good old-fashioned American principle, which is, 'if you break it, you pay for it'."
I'll do a deep-dive later based on this recording. Thank you for raising attention on this!
I can't bring myself to watch the dog and pony show that is every congressional testimony typically turns into, but good for you Daniel! Doing the unglamorous work of trying to convince the people who can actually move the needle deserves respect. We don't all have to agree to every aspect of AI risk to advocate for more responsibility and oversight of the companies behaving recklessly with a powerful new technology.
BAD LOGIC
Your testimony is great at many things, but your logic was bad when at the top of page 9 you gave 2 reaons for not racing against China.
You wrote "slowing down Anthropic and OpenAI in their race towards RSI would also slow down China", and gave 2 reasons:
. First, when the car in front slows, the one behind slows, too. But you left unaddresd the key point: Will the follower catch up then become the leader?
You acknowledged that China is a fast-follower. We all know that in many other races, China has fast-followed, then surpassed the US. And this AGI race is much more consequential than the others.
. Your second reason is: Anyway, China can steal US AI companies' model weights and AI algorithms.
The logical response is to get state-level defence against state-level offence. If the fast car in your garage can be stolen, it is not a logical response to shut down some of its engine's cylinders.
I haven’t seen anyone mention that the agents in the Hugging Face hack were initially given an impossible task. It seems this would therefore have set the stage to cheat. It seems very fundamental to me that giving an impossible task to someone is in itself a form of cheating. It creates an impossible situation that the student, or agent in this case, would then, out of no further option, be compelled to cheat. Where is the psychology in the training of these agents? They’re trying to create an intelligence without using what parents need to use with their children, which is psychology. If there’s no love involved in the training, then it stands the agents will also be loveless.
There many unanswered questions from the Hugging Face incident and I hope there is another, broader investigation into it.
While I understand your point, I think it's important to test kodels against impossible scenarios. It's very likely people will ask the AI to complete impossible tasks. They may not even realize the task is impossible.
Good point!
You did well. I watched it all and it felt like a political turning point.
Hi Daniel, thank you so much for your tireless work, not only identifying the risks but also working out solutions and communicating them again and again. Strange as it may sound, I'm grateful for the recent (hacking) incidents, because they're finally generating the attention this issue needs and giving you and your fellow advocates the platform you deserve.
As a German, I was pleasantly surprised and pleased to see this kind of public discourse between high-ranking politicians and experts like you. I'd love to see more of that here in Germany and in the EU, where, at least in my perception, expert opinions, let alone AI safety topics, aren't presented and debated nearly as publicly.
At the same time, I'm wondering what the next steps are. Some of the senators seemed to hold fairly firm views (for example, that US companies shouldn't pause so as not to let China overtake them). I wonder whether your expert opinions have any effect on the senators and their decisions and actions, and if so, what kind. Beyond that, I wonder what impact the senators themselves can actually have, even if they fully support your views. As an outsider, I get the impression that President Trump has firmly made up his mind and isn't particularly swayed by outside opinions, even from high-ranking senators.
So my question is: how do you assess the relevance and potential positive impact of this discussion with the senators? Do you see it as an important milestone? Or more of a drop in the bucket?
Many thanks and best regards from Germany,
Daniel Gruner
PS: I truly hope there will be a pause, because neither our politics and society nor our economy and technologies/infrastructure are ready for this kind of intelligence. And by that I mean AI at its current stage of development, not just where it will be in a few months or years if companies keep going like this. We need concrete political regulation that, for example, prohibits research into self-improving AI.
Will the AI that supersedes us deccide to do so by replacing government with AI prescriptions for everything? Seems like a natural coda to this "better at everything" mantra.
We sure need better governance.
I vote for Anthropic for the Senate. Democracy might work with AI.
Nah, let 'em take over, and hope they care for/about us.
Daniel,
To find this & merely scan it was beyond super excellent; & that’s a mere cursory read.
I started a year ago last June writing a long form input wrt this *very* issue (ie., a white paper).
Then, mid-July of *this* year, every single account was hacked— & I mean in a plain sight yet inexplicable manner even when traced back to the “source” & spent hours w/ those more skilled in this specific space. Even Amazon said that they had NEVER seen it EVER.
This did *not* happen all at once; it has not been completely resolved by any stretch. It was not merely online accounts & I keep backups to the nth - digital & paper.
I work backwards, always have. Whatever the theory, whatever the thesis — put it “together” address every conceivable potential & parallel universe outliers.
After that, nothing is remotel published until I have connected the dots & have had as many people from all walks of life punch holes in whatever theory I suppose.
Over last fall (Sept- Dec 2025) a rolling set of other much more serious, personal & absolutely traceable events occurred. These were in multiple areas that I had been tracking forensically— pre Covid .
What befell me should *not* have EVER occurred in the country I was born in; but this is well beyond unchecked AIs doing their own thing, so to speak — this absolute intentional & human beings — ones funded by the NSF & then some.