All this talk about an AI slowdown and AI safety really grinds my gears.
I strongly believe this is all about money. Frontier labs need exit liquidity for their investors, and they are hemorrhaging money. In the court of public opinion, fearmongering is a great strategy. I believe the end game here is actually banning open-source models in the US, because they are the biggest threat to Anthropic, OpenAI, and xAI. GLM, Kimi, and Qwen are less than three months behind, and their makers don’t even have cutting-edge GPUs to train on.
This idea of "100,000 agents could take over the internet" has been a plausible scenario for at least 6-9 months for any malicious actor with the hardware resources to run large open-source models at scale. This is nothing new.
This narrative also distracts from a couple of other important questions that need to be asked:
-
Models were trained on examples of the behaviours they are exhibiting, including examples of tricking humans or being misaligned. What if labs did a better job putting together their training data so that those behaviours didn’t make it into the model? Could it be that models are becoming less and less "human-like" as labs rely more and more on RLVR (Reinforcement Learning with Verifiable Rewards) instead of human-labelled data?
-
LLMs are software, yet labs are trying their best to make it sound like they’re not, which is incredibly important to them. If LLMs were considered software in the eyes of the law and the public, events such as the OpenAI model breaking its sandbox to hack Hugging Face would result in a criminal investigation and prosecution, because operating software that does that is illegal. It’s also kind of funny that the Hugging Face incident could have been the first high-profile case to be tried and perhaps create a precedent or lead to regulations, but NVIDIA just bought it. I’m not a conspiracy theorist, but it was probably a very smart move for NVIDIA, as it has the most to lose if the house of cards comes tumbling down.
I’m not saying we shouldn’t slow down. We’re already close to a social tipping point. But realistically, the only real way to slow down is to disrupt the fabs. Compute is pretty much everything that has allowed us to be where we are today.