Discussion about this post

User's avatar
Nathan Metzger's avatar

The companies aren't racing because they don't know how to stop. They’re racing because they want to create superintelligent AI.

Many individual people within the companies prefer to prevent an escalating race, but the egragore of the companies themselves want to build superintelligence, no matter how obviously suicidal. That action takes place at the level of the company, not at the level of the individual, who in this respect is powerless while inside the company.

I don't expect research like this whose primary audience is the AI companies themselves to be as effective as research whose primary audience is the public or lawmakers. More to the point, I think the “pace the frontier” rhetoric paints over the obvious fact that walking off a cliff isn't much better than running off one. Anything other than a full halt of frontier AI progress spells human extinction. We don't only have a speed problem; the destination itself is fatal.

Ready for more?