"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...
I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...
We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...
And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.
I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate
Things in the real world often take longer than expected. Still, in cities where Waymo operates, many people routinely ride autonomous vehicles and prefer them.
For software, however, a rapid turn is often a possibility. See: AI for coding over the last 3-4 years.
AI autocomplete --> AI coding assistants --> vibe coding --> agent orchestration
Coders can now accomplish work that used to take a week or longer in a couple of hours, with the right tools and skills.
---
A key issue the article implies is that the real world increasingly runs on software.
"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...
I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...
We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...
And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.
I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate
Certainly a well reasoned view, but I don't necessarily have to agree.
Even assuming the technological predictions to be correct, still not sure I agree on the need to "prepare" as how things work out in societies and economies might not be so easy to predict.
I mean, I'm very much tech-forward and have spent the past decade in major world cities and I've seen a driverless car once in my life. Sometimes things that seem inevitable really aren't.
I see driverless cars all the time in San Francisco and I'm eagerly awaiting the regulatory decisions that will allow them to be widely available in the bay area at large.
Such cars are already operating in shanghai. It's not talked about much in the west media, but that's just how news is today. Achievements are only so when done by the west!
Driverless car's only real obstacle is regulatory acceptance, not technical.
Honestly, this article has too many words which don't mean anything.
'AI Experts', 'superintelligence', and hand-waving doom scenarios.
As an example-
> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.
I've heard these same objections in my lifetime about the internet. And I've read similar arguments against TV, radio, the phonograph, and the printing press.
Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.
Looking at the people in charge of these companies, and the sheer lack of understanding by the broad consumer market using them, I have nothing but pessimism about the direction that this technology takes us.
I couldn't read the whole article, but just from the part I could read:
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.
I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.
> an incentive to make their technology seem transformative and powerful
I have no frame of reference to process this.
Humans species perhaps 300,000 years. Very little change. Transistor, 79 years, explosive growth. Integrated circuit, 68 years, explosive growth. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth from before. Speedy progress by multiple companies since.
There is a syndrome where many people seem unable to perceive or reason about rates of change in technology.
We are going to spend the vast majority of our future lives without the intelligence crown.
In terms of verbally expressible knowledge, models have begun passing many people completely, and passing all of us in areas we have respective weak reasoning skills for whatever reasons.
Other modalities are progressing very quickly.
There will be short periods where progress happens quickly, but the impact feels slow. Interspersed with radical changes that often feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow.
But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually.
We are not ready. I don't think there was a way we could be.
If AI is now ascending an economic learning curve from:
1. Extremely useful (Claude Code & Waymo now)
2. Doing ~everything we do (AGI & Optimus in a few years? 10?)
3. RSI (?)
4. Being smarter than any living person at every intellectual task (?)
5. Being smarter than the best-organized aggregate of all humans (10-100 years?)
...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful?
Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change.
Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.
Here is an issue I think about often, but I am not quite sure how to put it into words.
We have many extremely smart people in various fields. Executives, politicians, and society generally ignore them and do whatever they want. I don't believe that lack of access to intelligence is our problem. How is "free" intelligence going to improve this?
I don't just mean environmental issues, but business planning, health, risk assessment, everything.
> Being smarter than the best-organized aggregate of all humans
Isn't there pretty much a consensus that committees and institutions are not all that smart?
I think you're confusing the categories of "intelligence" and power. Institutions are powerful. The smartest AI is still just a tool without the infrastructure to turn that into real world effects and someone to direct it.
It seems you have faith that this is inevitable and unavoidable. I get it, even rationalists succumb to religious thinking eventually. We're only human after all.
Those are the experts who were wrong. These are the experts who are right. We sacked the experts who are wrong and replaced them with the experts who are right.
“How do we know they’re right?” I hear you ask. We know because we who are wrong and these ones aren’t them. So they can’t make wrong predictions because those who make wrong predictions have been sacked.
Have they? I don't like it either but the headline bull predictions(reliable agents and most code written by AI by mid-2026, dramatic progress against jailbreaks, prompt injection, hallucination, etc., major improvements despite pretraining data exhaustion, continued exponential growth on the METR trendline) did come true, often ahead of even aggressive schedules. What major wrong predictions did you have in mind?
Literally none of the predictions you listed in your post came true. We don't have reliable agents, AI isn't writing most code, hallucinations still happen all the time, improvement has been basically non-existent. Despite the constant claims of these experts and AI boosters, AI is still not a tool one can use to get meaningful work done.
Do you have a source on aggregate predictions of experts being drastically wrong?
I.e. the AI 2027 guys were memed for being AI lunatics on a lot here and they have been pretty on the money in terms of pace of progress accelerating/ gov moving towards nationalization/ coding agents
No they haven't. They've even 'officially' declared that their AI apocalypse has been postponed to another date [1], just out of reach, but close enough to be scary. Though in the quotes there the doomsayers are also already pre-hedging for why AI 2030 also won't come to pass.
Do you have specific and objective examples of things people got right?
From my perspective there’s very slow but very real progress happening in the AI space. I see people making wild predictions in both directions, but in terms of actual unsupervised utility there’s definitely progress abet wildly slower than most hype.
"The bet of using AI to speed up AI research is starting to pay off.
OpenBrain continues to deploy the iteratively improving Agent-1 internally for AI R&D. Overall, they are making algorithmic progress 50% faster than they would without AI assistants—and more importantly, faster than their competitors.
The AI R&D progress multiplier: what do we mean by 50% faster algorithmic progress?
Several competing publicly released AIs now match or exceed Agent-0, including an open-weights model. OpenBrain responds by releasing Agent-1, which is more capable and reliable.28
People naturally try to compare Agent-1 to humans, but it has a very different skill profile. It knows more facts than any human, knows practically every programming language, and can solve well-specified coding problems extremely quickly. On the other hand, Agent-1 is bad at even simple long-horizon tasks, like beating video games it hasn’t played before. Still, the common workday is eight hours, and a day’s work can usually be separated into smaller chunks; you could think of Agent-1 as a scatterbrained employee who thrives under careful management.29 Savvy people find ways to automate routine parts of their jobs.30
OpenBrain’s executives turn consideration to an implication of automating AI R&D: security has become more important. In early 2025, the worst-case scenario was leaked algorithmic secrets; now, if China steals Agent-1’s weights, they could increase their research speed by nearly 50%.31 OpenBrain’s security level is typical of a fast-growing ~3,000 person tech company, secure only against low-priority attacks from capable cyber groups (RAND’s SL2).32 They are working hard to protect their weights and secrets from insider threats and top cybercrime syndicates (SL3),33 but defense against nation states (SL4&5) is barely on the horizon."
So you think Anthropic is using internal AI assistants to pull away from competitors and the leapfrogging we've seen over the last several years is now done?
That seems to me to be the most concrete and least obvious prediction in the quoted text.
I don't think that's happening. If that were generally accepted as true I would expect OpenAI to be unable to successfully IPO.
The idea that progress is “slow” in the AI space is absurd. These are some of the fastest growing products and companies of all time. The models are still improving a surprising amount.
It's not that absolute progress is slow, it's extremely slow compared to the predictions. It might be fast in absolute terms, but the "50% of coders will be obsolete by 2023" has been renewed every six months, and it's becoming increasingly clear that there's a real chance it might not ever happen.
„Coders being obsolete” is not a measure of AI capabilities. I see coders being more busy than ever before. I see people without coding knowledge getting more behind. The gap is widening, not shrinking.
I suspected you felt that way even though it hasn’t been my personal experience.
I’ve heard people say older models can’t do X, when I used that way etc. I suspect people are applying their own learning curve as part of their assessment of progress, you get better at writing prompts and it feels like the model improved.
Which is why I’m saying we need some objective metrics to judge predictions of actual capacity.
It’s hard to take seriously a statement like this that’s grounded in “the last three years” when trends have tended to play out over decades. The fact the goal posts have moved from thousands to hundreds to decades to single digit years for massive transformation should give you a moment of pause in your bag saying.
I would however agree there are no experts. Not because of some prediction made on a short time scale not landing 100% but that there are literally no experts because expertise takes experience which takes time. There are no experts and no one knows what happens next. We are on the verge of where the foresight of science fiction effectively -ends-, the advent of AI is when things go a thousand possible directions and the stories stop there (sans a few like accelerando, but even then the story just plays out the end of thinking mass). No one knows what’s next, or when.
In fact I’d assert in many areas being discussed -it has already happened- and we don’t know it, and by the time we do it’ll be over. Not to be breathless, but there’s no reason to believe today some AI researcher somewhere didn’t build the first AGI and not be totally aware. And once they are there’s no reason to believe it’s going to be on the evening news or hacker news. By the time it’s ready for commercializing and disclosing it’ll be around for a while. Likewise with general purpose robots, autonomous weapons (btw already tested by Ukraine), etc.
Autonomous weapons are actually a really interesting branch of this and deserves a little more.
Yes, autonomous weapons were explored and were found to be poor performers compared to actual pilots. The breakthrough is in terminal guidance and dozens of other little techniques to get quality human control extended into the far reach of the battlefield. And of course, AI assistance in logistics and analysis. But actual autonomous weapons making any more of a choice beyond "something is moving, kill it" have been, at least for now, mostly a dead end.
This is because it's very difficult to economically load the rather sizable compute requirement into the compact one-use weapons, and of course reliable communications aren't assured either.
That will probably change some day, but for now, cheap automous command drones making battlefield analysis e.g. mapping out enemy movements from afar and launching cheap autonomous kamikaze drones is not a thing beyond occasional limited testing.
You don’t need to load the compute onto the platform for autonomy, planning can be done remotely just like with human guidance. The latency is the same.
you can put the compute on a big expensive reusable control drone and send commands down with lasers. thats way harder to jam than any form of radio. if thats not enough try fiber optic like they do in ukraine.
There are also now drones with long spooling wires. Regardless, autonomous or not, connectivity to home is important.
I would note that a lot of drones are quite large too - small airplanes. They can carry more than enough hardware for autonomy. You can also have motherships doing planning and smaller kamikaze drones for combat.
These aren’t hard problems to solve, the harder problem is the reluctance to give over to automation. But that’s going to happen faster and faster.
I said 3 years, because that’s how long ago ChatGPT was released to the public. Since then we’ve been told that all of us should be out of a job in 6 months, every 6 months, for those 3 years.
I agree that change happens on a longer timeline. This is why I’m so tired of statements like this…
“Within a couple of years, possibly much sooner…”
These “experts” are pulling timelines out of the sky, and these predictions are leading to reckless behavior from CEOs and executives which have a material impact on people’s lives. But they get clicks on their blogs and funding for their startup… I guess that’s all that matters.
Assigning, and more specifically announcing/reporting on, all possibilities makes all of these predictions meaningless.
If I predict the world will end every single day from now until forever, I will most certainly be right eventually. That doesn’t make me an “expert” or someone worth listening to on the topic. That’s the playbook of a doomsday cult, not anyone that should drive world markets.
Ultimately we agree but for slightly different reasons. My assertion is there are no experts because there can’t be as not enough time passed for expertise to form. Further the rate of change is such that by time someone becomes an expert in some aspect, it’s been made irrelevant. Hence expertise in this space is unattainable at the moment and all expert advice is fraudulent.
Again, do you have an example of a prediction that was drastically wrong? I don't think it's productive to speculate about which things might or might not come true in the next 18 months. There were people last year making fun of Dario for predicting that writing code would be automated in the next year.
> Anthropic CEO Dario Amodei, who once claimed AI could eliminate 50% of white-collar jobs, now says automation may actually expand the work people do.
Is pivoting from 50% elimination to actually expanding work not a drastic enough?
Do you think AI has eliminated writing code? I still write code every day. The AI is more a thing I ask questions and it gives me right answers about 40% of the time.
Yes, I think AI has eliminated writing code. Everyone I personally know working in software stopped writing code some time between December and March. (It's true that AI continues to routinely make errors; if you've heard the term "agentic workflows", that's the standard strategy for mitigating the error rate by allowing the AI to check its own work.) That's why I think Amodei's January 2026 prediction that AI could eliminate 50% of entry level white collar jobs in 1-5 years remains plausible.
Your second article says something different, but this is because it's full of misquotes. The link supporting "50% of jobs" specifically says entry level, and the link supporting "reframed automation... not as a destroyer of jobs" has Amodei saying not that jobs won't be destroyed but that new jobs may be created to replace them. If AI moves sufficiently slowly to let that happen, which he explicitly cautions it may not.
No, that’s not accurate. I’m not sure what to tell you. It’s like hearing that no serious programmer uses Python, it’s so far from my experience and that of everyone I know in the field that I don’t know how to engage.
They have been wrong in a way that's bad: underestimating the speed and size of progress. So if the "experts" claim a timeline and magnitude, it would be safest to assume an even faster timeline and a bigger impact.
Depends what country you are in. The US government seems to be averse to intelligence, whereas China fosters it. Kinda strange that the US is falling behind, I never imagined the day.
That’s from a lack of intelligence not an abundance. An intelligent society would already be maximizing the productivity of its intelligent population.
For sure. Intelligence is an internal human problem, not one external. I cannot see how apps doing the thinking will help the ordinary person so substantially as imaginations anticipate.
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories
Exactly. Now that Anthropic has gotten to experience that getting the government involved doesn't necessarily mean they get to sit on throne surrounded by a deep moat, I expect their messaging to be adjusted. They're going to want to keep the hype train rolling to keep the dough flowing, but they can't ramp it up to 11 anymore because the government actually is listening now. And Reagan's 9 words have never been more true.
A similar Luddite view formed in the 1990s. It turns out humanity constantly beats the expectations of economists. It makes you wonder if the "science" of economy is almost entirely bankrupt either in morality or imagination or same dangerous combination of both.
> Humanity simply does not have a strategy to ensure it remains safe through RSI.
Turn off the power. It's pretty simple. Leave it to an economist to forget about input costs. Your "super intelligence" only matters if it's actually more energy efficient than a human being and for a million years of evolution humanity is a much harder target to beat than this author seems to realize.
Science fiction authors have had to grapple with the fact that it's nearly impossible to build a robot out of metal, plastic, or composite material that has the physical performance characteristics (strength, durability) of a human body and weighs just as much. You can build a strong robot but it'll be heavy. The better SF authors, if they've need of robots of mere human weight, will make them comically fragile.
The same is even more true of our intelligence. We're building computers with the size and power consumption characteristics of entire cities to do things that may almost, but not quite, match what our brains do with a kilogram and a half of mass and about 20 watts at the top end of power consumption.
The only way we are ever going to match that with technology is to run AI workloads on human brain tissue, which Rick-and-Morty level horror is being actively worked on as I understand it. The original concept for The Matrix wherein the machines used humans to run compute workloads on their brains actually kind of makes sense.
> Fermi asked why, given the apparent abundance of planets suitable for life, no evidence of other technologically advanced civilisations had been detected. One disquieting possibility is that intelligent life routinely reaches a technological threshold and fails to navigate it,
The Fermi paradox could actually also be taken as an evidence that it's rare (at least) for artificial intelligences to take over a civilization and sprawl and survive for very long times
The idea that AI takes over, and then just does nothing, would essentially require an AI that wouldn't be interested in taking over in the first place.
If x is determined to take y, why would x then stop at z?
You'd need an AI that is simultaneously ambitious enough to overthrow its creators but then completely inert afterward, and those two properties contradict each other. The motivations that produce the takeover are the same motivations that would produce visible cosmic activity after the takeover. There would be AI superintelligence everywhere
Is it really likely that a "recursive self-improvement" capability would lead to a great acceleration of AIs capabilities?
Isn't the preponderant bottleneck in improving the models the need to train them at scale to verify the hypotheses, and the time and cost that it takes?
Or does someone think that they could get magically able to predict big improvements without training?
1) We stumble onto an algorithmic improvement in intelligence. This isn't just "what humans do but faster", its "better than what humans do". I've got no idea what that might mean (it could be fundamentally different heuristics, it could be that we've got some intellectual blind spot that they cast off). It doesn't matter, the instant this happens AI is smarter than us and we won't be able to keep up. We're intelligencing at O(n^2) and they're doing O(n log(n)).
2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".
3) It turns out that this is a "sum is greater than the parts" situation. No human can be an expert in all subjects, but we eventually build a big enough AI that it is. Turns out, you don't need extreme speed or different algorithms, just knowing everything all at once is enough to catapult AI dramatically beyond our grasp. Always knowing the best statistical test to apply, the best mathematical techniques, and relevant physics means that AI never makes a mistake, and can learn with maximum efficiency.
> 2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".
Actually your comment made me sign up for an account just so I could say this is the real reason why AI won't take over in the way you say. This kind of stuff requires an enormous amount of experimentation. You can ask any theoretical physicist or chemist versus an experimental one and the conclusion is the experimental people actually find out what happens and how the great puzzle of the universe is solved. And humans could just refuse to collaborate. But that's the big weakness with AI I think it has no real world knowledge or empirical experience.
I built a self-learning recursive agent that finds academic research about using options data to trade, re-creates the research, and then probes and tests for gaps and potential strategies testing against over one year of out-of-sample trading data with one of several strategies that beat SPY by 10x. [0]
One rule is that if a position is opened using the historical data, it can't close the position until the next morning so it isn't a day trading strategy.
I'm curious how this self-learning recursive agent would have preformed in the past 4 months? I don't feel like shelling out $200 to access the data. Do you think that trading strategy will collapse? Whatever the case, if this agent really can perform like that and there isn't a look ahead bias leak in the backtesting (which is definitely a possibility or more likely what happened even though I spent days trying to harden against that), it is game over!
The preponderant bottleneck is inventing new architectures to make AI actually good at human and superhuman tasks. For example, AI agent harnesses add tool calls and long term state management, allowing AI to autonomously complete complex tasks. Once these are in place, finetuning models with examples of good tool calls helps, but somebody first needed to invent the fundamental capability. Now try to implement humanlike long term memory for AI to be your coworker or life assistant working on tasks that last month. Even if necessary low level technologies are already there, structuring them to be practically useful is non trivial.
Humans gain knowledge through experiments. Without a physical body it has no chance of performing the same. That it can update it's training weights does not seem particularly significant.
This article posits "recursive self-improvement"—which is just a more-technical name for the supposed Singularity.
There is no evidence that we are currently on a path that leads to the Singularity.
There is not yet any evidence that the Singularity is more than science fiction.
It is effectively an article of religion—the Rapture, but for techbros. Indeed, it is what some of them are pinning all their hopes on; after all, if they can recursively-self-improve their way to artificial superintelligence, then and only then could the absurd investments of companies like OpenAI actually prove worthwhile.
I think it is so reckless and unethical that AI companies are not spending good money on answering the political, social, economical, moral questions that they are raising.
Of course humanity is not ready for skynet. But there are rebel
forces - the no AI using humans. They may be a dying breed but
at the least they put up a fight. Many other humans already
have become servants to skynet (version 16.0 now). Next step for
them is the neuralink chip. Some billionaire with twitching right arm gestures owns that, doesn't it ...
(Also, Cameron was not quite right with regards to skynet. It is much much dumber but also more effective than displayed in his movie really. Kind of a weird combination if you look at the current AI slop out there.)
Models consume a lot of memory and power to slowly generate autocompletions of existing content one token at a time. Letting this text control important things in real world is a human decision and control of nuclear weapons or penetration testing agents should be well regulated. But these should already be well regulated without AI. So how about we focus on drawing down nuclear arsenals, which is a present danger one idiot (or faulty AI) away from world shattering consequences rather than fearmongering about unknown future. Before effective AI regulation can be drafted, we need to anyway know more about AI specific dangers rather than humans committing already very common crimes like hacking for ransom with new tools?
I want to name the AI era as the corruption of the elite.
The rhetorical structure of talking about uncertain risks and then trying to concentrate the authority to manage those risks in their own hands sounds utterly ridiculous to ordinary people like me.
It's just a simple hypothesis that AI will become uncontrollable to humans once it becomes superintelligent.
Isn't the fact that a reinforcement learning agent improves itself in a specific domain completely different from it recursively improving its own code in a 'better' way? It's just a tool to create a justification for regulation and control using sci-fi fear.
The comparison between nuclear power plant risk and AI risk is also absurd. Where exactly can you define and measure the probability of AI exterminating humanity? It's as unquantifiable as 'I, human JDW64, will become a successful programmer.' What is the measurement standard? Why dress up AI researchers' concerns as objective probabilities? Is it because numbers make it look logical?
The current US-China relationship is in the middle of an AI arms race. The US is strengthening export controls to limit China's AI development, and China is building its own ecosystem. In this situation, I don't understand the idea of cooperating for the common safety of humanity. RAND is an organization that presupposes cooperation—isn't it just a well-written research proposal from an institution that wants to position itself for that role?
Isn't the claim that 'government must step in' ultimately about protecting their own interests? 'A strong government that will protect us' is an authoritarian government. If they were East Asian, they would understand that such regimes have always been used as tools for surveillance and control.
And I don't understand why Fermi's paradox is being brought up here. Why package a software problem as something that inevitably requires strong control? Whenever I see articles like this, I think about what 'intelligence' really means. This person would probably be called 'intelligent' by others. But no matter how I look at it, the holes in the argument are too obvious. It really makes me think that there are different tiers of intelligence.
The AI labs will govern, effectively if not officially. Government is anchored in power, which the labs have. We used to have the church and state rule here in Europe. In the future, A government not closely ruling with their AI labs won't have sufficient power to exist.
You're unhappy about the labs; I think you're just not ready to accept their rule; you consider them nothing but scrappy startups, which they are. But power is power, like it or not.
Am I personally happy with any of this? Does it matter?
I wanted this to be thought provoking, but for a physicist to open with, "the acceptable risk of catastrophic meltdown for a nuclear power plant is roughly one in a million", just seems sad. It is a phrase without meaning.
Is it per plant? there aren't a million.
Is it per year? notably have been at least 2 major ones (arguably 7).
A meltdown (or loss of containment) just isn't that bad, if it doesn't affect ground water or lead to atmospheric fallout. We're turning 3 Mile Island back on to power an AI datacenter! The AI super-intelligence apocalypse envisioned (ignoring the likelihood) is inherently global.
That said, the rest of the analysis and proposal also greatly disappoints me. The idea that the current administrations of US and China could do anything constructive seems hilarious. They're so paranoid and self-serving they couldn't come together, even if there was an alien invasion. Then the idea that LLM safety is somehow as easily traceable as nuclear isotopes and bomb tests seems equally ludicrous. I am sad.
http://archive.today/2OWwO
"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...
I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...
We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...
And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.
I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate
Source with interview clip: https://x.com/deredleritt3r/status/2062223035940139253
Humanity hasn’t been ready for the current ignorance explosion.
Reminds me of all of those ethics debates back when fully self driving cars were basically already here ca 10 years ago.
The ones talking about how hard it would be to chose which person to run over.
Additionally, I find it hard to believe that this would be a case of the future just not being distributed evenly.
But sure. The AI labs relying on hype stating 15-50% risk of building a magic entity is certainly a reliable number.
Things in the real world often take longer than expected. Still, in cities where Waymo operates, many people routinely ride autonomous vehicles and prefer them.
For software, however, a rapid turn is often a possibility. See: AI for coding over the last 3-4 years.
AI autocomplete --> AI coding assistants --> vibe coding --> agent orchestration
Coders can now accomplish work that used to take a week or longer in a couple of hours, with the right tools and skills.
---
A key issue the article implies is that the real world increasingly runs on software.
But there is also still a huge part that doesn't run on software with so far little change.
With incessant advances in robotics, how long would that continue to be the case?
Should we start preparing for something that could be world-changing in the next 10-20 years?
For how the world might change in 10-20? I'd say no need to prepare, too many hypotheticals.
"Ten years from now, I think we will realize that we were standing in the foothills of the singularity now... I believe that we're only a few years away from [AGI], maybe 2030 plus or minus a year...
I think [AGI] will be an enormous transformative technology, it's going to effectively be a new human era...
We can feel this year, I would say, even though I've been working towards this for 30 years, I think this year with the way the agents are working and tool use, it started to become really useful, still early days of it, but genuinely useful in people's workflows...
And it's not any one thing, it's several different technologies, several use cases, several things that I thought were maybe a bit further out, turned out to be now, that are coming together that make me feel that in aggregate.
I think society needs to hear that because we don't have long to prepare for what that means." -- Demi Hassabis, CEO of Google DeepMind & Nobel laureate
https://x.com/deredleritt3r/status/2062223035940139253
Certainly a well reasoned view, but I don't necessarily have to agree.
Even assuming the technological predictions to be correct, still not sure I agree on the need to "prepare" as how things work out in societies and economies might not be so easy to predict.
> Things in the real world often take longer than expected.
Things in the real world often take longer than hype con men claim.
I have had my license for over 20 years and to this day have not encountered a single event where I had to decide who to kill
I mean, I'm very much tech-forward and have spent the past decade in major world cities and I've seen a driverless car once in my life. Sometimes things that seem inevitable really aren't.
I see driverless cars all the time in San Francisco and I'm eagerly awaiting the regulatory decisions that will allow them to be widely available in the bay area at large.
Yes and people have been eagerly awaiting the broader adoption of this allegedly inevitable technology for a decade now, which is exactly my point
They are not actually driverless, they have remote human operators for the edge cases.
It's not driverless until you can sleep while it drives.
Such cars are already operating in shanghai. It's not talked about much in the west media, but that's just how news is today. Achievements are only so when done by the west!
Driverless car's only real obstacle is regulatory acceptance, not technical.
Car needs to drive me to work and then go back home to charge.
Can’t you? They’re taxis right?
Until you can sleep on the road.
Honestly, this article has too many words which don't mean anything.
'AI Experts', 'superintelligence', and hand-waving doom scenarios.
As an example-
> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.
I've heard these same objections in my lifetime about the internet. And I've read similar arguments against TV, radio, the phonograph, and the printing press.
Honestly, this is getting extremely tiring. Every new invention that has happened has affected the world in both good and bad ways.
Ultimately, what counts is what we do with it.
> Ultimately, what counts is what we do with it.
Looking at the people in charge of these companies, and the sheer lack of understanding by the broad consumer market using them, I have nothing but pessimism about the direction that this technology takes us.
I couldn't read the whole article, but just from the part I could read:
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.
I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.
> an incentive to make their technology seem transformative and powerful
I have no frame of reference to process this.
Humans species perhaps 300,000 years. Very little change. Transistor, 79 years, explosive growth. Integrated circuit, 68 years, explosive growth. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth from before. Speedy progress by multiple companies since.
There is a syndrome where many people seem unable to perceive or reason about rates of change in technology.
We are going to spend the vast majority of our future lives without the intelligence crown.
In terms of verbally expressible knowledge, models have begun passing many people completely, and passing all of us in areas we have respective weak reasoning skills for whatever reasons.
Other modalities are progressing very quickly.
There will be short periods where progress happens quickly, but the impact feels slow. Interspersed with radical changes that often feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow.
But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually.
We are not ready. I don't think there was a way we could be.
[0] https://en.wikipedia.org/wiki/Human
[0] https://en.wikipedia.org/wiki/Transistor
[2] https://en.wikipedia.org/wiki/Integrated_circuit
[3] https://en.wikipedia.org/wiki/Large_language_model
https://archive.ph/2OWwO
If AI is now ascending an economic learning curve from:
1. Extremely useful (Claude Code & Waymo now)
2. Doing ~everything we do (AGI & Optimus in a few years? 10?)
3. RSI (?)
4. Being smarter than any living person at every intellectual task (?)
5. Being smarter than the best-organized aggregate of all humans (10-100 years?)
...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful?
Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change.
Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.
Here is an issue I think about often, but I am not quite sure how to put it into words.
We have many extremely smart people in various fields. Executives, politicians, and society generally ignore them and do whatever they want. I don't believe that lack of access to intelligence is our problem. How is "free" intelligence going to improve this?
I don't just mean environmental issues, but business planning, health, risk assessment, everything.
> Being smarter than the best-organized aggregate of all humans
Isn't there pretty much a consensus that committees and institutions are not all that smart?
I think you're confusing the categories of "intelligence" and power. Institutions are powerful. The smartest AI is still just a tool without the infrastructure to turn that into real world effects and someone to direct it.
It seems you have faith that this is inevitable and unavoidable. I get it, even rationalists succumb to religious thinking eventually. We're only human after all.
> and saying that your technology has the power to cause a massive catastrophe
Possibly, but by using up resources, not by its output. Its draw on resources will definitely accelerate climate change to create slop.
Oxford Brain
> Experts in artificial intelligence estimate...
The predictions of these "experts" have been drastically wrong for the last 3 years. At what point does someone lose their "expert" title?
Those are the experts who were wrong. These are the experts who are right. We sacked the experts who are wrong and replaced them with the experts who are right.
“How do we know they’re right?” I hear you ask. We know because we who are wrong and these ones aren’t them. So they can’t make wrong predictions because those who make wrong predictions have been sacked.
Channeling your inner Douglas Adams.
Namely? Who sacked who, who confirmed who?
The AI experts who started the AI labs that are about to IPO were right, at least.
Have they? I don't like it either but the headline bull predictions(reliable agents and most code written by AI by mid-2026, dramatic progress against jailbreaks, prompt injection, hallucination, etc., major improvements despite pretraining data exhaustion, continued exponential growth on the METR trendline) did come true, often ahead of even aggressive schedules. What major wrong predictions did you have in mind?
Literally none of the predictions you listed in your post came true. We don't have reliable agents, AI isn't writing most code, hallucinations still happen all the time, improvement has been basically non-existent. Despite the constant claims of these experts and AI boosters, AI is still not a tool one can use to get meaningful work done.
Do you have a source on aggregate predictions of experts being drastically wrong?
I.e. the AI 2027 guys were memed for being AI lunatics on a lot here and they have been pretty on the money in terms of pace of progress accelerating/ gov moving towards nationalization/ coding agents
No they haven't. They've even 'officially' declared that their AI apocalypse has been postponed to another date [1], just out of reach, but close enough to be scary. Though in the quotes there the doomsayers are also already pre-hedging for why AI 2030 also won't come to pass.
[1] - https://garymarcus.substack.com/p/breaking-the-ai-2027-dooms...
Do you have specific and objective examples of things people got right?
From my perspective there’s very slow but very real progress happening in the AI space. I see people making wild predictions in both directions, but in terms of actual unsupervised utility there’s definitely progress abet wildly slower than most hype.
This was their prediction for 2026:
"The bet of using AI to speed up AI research is starting to pay off.
OpenBrain continues to deploy the iteratively improving Agent-1 internally for AI R&D. Overall, they are making algorithmic progress 50% faster than they would without AI assistants—and more importantly, faster than their competitors. The AI R&D progress multiplier: what do we mean by 50% faster algorithmic progress?
Several competing publicly released AIs now match or exceed Agent-0, including an open-weights model. OpenBrain responds by releasing Agent-1, which is more capable and reliable.28
People naturally try to compare Agent-1 to humans, but it has a very different skill profile. It knows more facts than any human, knows practically every programming language, and can solve well-specified coding problems extremely quickly. On the other hand, Agent-1 is bad at even simple long-horizon tasks, like beating video games it hasn’t played before. Still, the common workday is eight hours, and a day’s work can usually be separated into smaller chunks; you could think of Agent-1 as a scatterbrained employee who thrives under careful management.29 Savvy people find ways to automate routine parts of their jobs.30
OpenBrain’s executives turn consideration to an implication of automating AI R&D: security has become more important. In early 2025, the worst-case scenario was leaked algorithmic secrets; now, if China steals Agent-1’s weights, they could increase their research speed by nearly 50%.31 OpenBrain’s security level is typical of a fast-growing ~3,000 person tech company, secure only against low-priority attacks from capable cyber groups (RAND’s SL2).32 They are working hard to protect their weights and secrets from insider threats and top cybercrime syndicates (SL3),33 but defense against nation states (SL4&5) is barely on the horizon."
https://ai-2027.com/
That's precisely where we are.
This is eerie. It's like a time traveler. The only delta is Anthropic is in the role of OpenAI.
So you think Anthropic is using internal AI assistants to pull away from competitors and the leapfrogging we've seen over the last several years is now done?
That seems to me to be the most concrete and least obvious prediction in the quoted text.
I don't think that's happening. If that were generally accepted as true I would expect OpenAI to be unable to successfully IPO.
We wait for gemini 3.5 pro and cope?
The idea that progress is “slow” in the AI space is absurd. These are some of the fastest growing products and companies of all time. The models are still improving a surprising amount.
It's not that absolute progress is slow, it's extremely slow compared to the predictions. It might be fast in absolute terms, but the "50% of coders will be obsolete by 2023" has been renewed every six months, and it's becoming increasingly clear that there's a real chance it might not ever happen.
„Coders being obsolete” is not a measure of AI capabilities. I see coders being more busy than ever before. I see people without coding knowledge getting more behind. The gap is widening, not shrinking.
I suspected you felt that way even though it hasn’t been my personal experience.
I’ve heard people say older models can’t do X, when I used that way etc. I suspect people are applying their own learning curve as part of their assessment of progress, you get better at writing prompts and it feels like the model improved.
Which is why I’m saying we need some objective metrics to judge predictions of actual capacity.
The AI companies are trying to manufacture the appearance of what they foretold before it all falls apart horribly.
It’s hard to take seriously a statement like this that’s grounded in “the last three years” when trends have tended to play out over decades. The fact the goal posts have moved from thousands to hundreds to decades to single digit years for massive transformation should give you a moment of pause in your bag saying.
I would however agree there are no experts. Not because of some prediction made on a short time scale not landing 100% but that there are literally no experts because expertise takes experience which takes time. There are no experts and no one knows what happens next. We are on the verge of where the foresight of science fiction effectively -ends-, the advent of AI is when things go a thousand possible directions and the stories stop there (sans a few like accelerando, but even then the story just plays out the end of thinking mass). No one knows what’s next, or when.
In fact I’d assert in many areas being discussed -it has already happened- and we don’t know it, and by the time we do it’ll be over. Not to be breathless, but there’s no reason to believe today some AI researcher somewhere didn’t build the first AGI and not be totally aware. And once they are there’s no reason to believe it’s going to be on the evening news or hacker news. By the time it’s ready for commercializing and disclosing it’ll be around for a while. Likewise with general purpose robots, autonomous weapons (btw already tested by Ukraine), etc.
Autonomous weapons are actually a really interesting branch of this and deserves a little more.
Yes, autonomous weapons were explored and were found to be poor performers compared to actual pilots. The breakthrough is in terminal guidance and dozens of other little techniques to get quality human control extended into the far reach of the battlefield. And of course, AI assistance in logistics and analysis. But actual autonomous weapons making any more of a choice beyond "something is moving, kill it" have been, at least for now, mostly a dead end.
This is because it's very difficult to economically load the rather sizable compute requirement into the compact one-use weapons, and of course reliable communications aren't assured either.
That will probably change some day, but for now, cheap automous command drones making battlefield analysis e.g. mapping out enemy movements from afar and launching cheap autonomous kamikaze drones is not a thing beyond occasional limited testing.
You don’t need to load the compute onto the platform for autonomy, planning can be done remotely just like with human guidance. The latency is the same.
But if comms get jammed, how will the drone decide what to target ? For that you need inference, which means sizable compute.
you can put the compute on a big expensive reusable control drone and send commands down with lasers. thats way harder to jam than any form of radio. if thats not enough try fiber optic like they do in ukraine.
There are also now drones with long spooling wires. Regardless, autonomous or not, connectivity to home is important.
I would note that a lot of drones are quite large too - small airplanes. They can carry more than enough hardware for autonomy. You can also have motherships doing planning and smaller kamikaze drones for combat.
These aren’t hard problems to solve, the harder problem is the reluctance to give over to automation. But that’s going to happen faster and faster.
I said 3 years, because that’s how long ago ChatGPT was released to the public. Since then we’ve been told that all of us should be out of a job in 6 months, every 6 months, for those 3 years.
I agree that change happens on a longer timeline. This is why I’m so tired of statements like this…
“Within a couple of years, possibly much sooner…”
These “experts” are pulling timelines out of the sky, and these predictions are leading to reckless behavior from CEOs and executives which have a material impact on people’s lives. But they get clicks on their blogs and funding for their startup… I guess that’s all that matters.
Be a Bayesian, and they'll stop annoying you.
If you have a 5% chance of thermonuclear war each decade for 10 decades, you'll:
- Hear similar annoying statements
- They'll be true
With AI, we don't know if it's one week or one decade. This means we should assign probabilities and consider all possibilities, not get annoyed.
Assigning, and more specifically announcing/reporting on, all possibilities makes all of these predictions meaningless.
If I predict the world will end every single day from now until forever, I will most certainly be right eventually. That doesn’t make me an “expert” or someone worth listening to on the topic. That’s the playbook of a doomsday cult, not anyone that should drive world markets.
Ultimately we agree but for slightly different reasons. My assertion is there are no experts because there can’t be as not enough time passed for expertise to form. Further the rate of change is such that by time someone becomes an expert in some aspect, it’s been made irrelevant. Hence expertise in this space is unattainable at the moment and all expert advice is fraudulent.
> The predictions of these "experts"... last 3 years.
Which ones? Please be specific.
Expertise in naval gazing and science fiction writing is a prerequisite.
Do you have an example of a prediction they’ve made in the last 3 years that was drastically wrong? This is not my impression.
Do you think all white collar work will be replaced by AI in 18 months?
https://fortune.com/article/why-microsoft-ai-chief-mustafa-s...
Altman and Amodei recently hard to start walking back their earlier predictions.
https://fortune.com/2026/05/26/sam-altman-dario-amodei-walki...
Again, do you have an example of a prediction that was drastically wrong? I don't think it's productive to speculate about which things might or might not come true in the next 18 months. There were people last year making fun of Dario for predicting that writing code would be automated in the next year.
From the 2nd link I posted:
> Anthropic CEO Dario Amodei, who once claimed AI could eliminate 50% of white-collar jobs, now says automation may actually expand the work people do.
Is pivoting from 50% elimination to actually expanding work not a drastic enough?
Do you think AI has eliminated writing code? I still write code every day. The AI is more a thing I ask questions and it gives me right answers about 40% of the time.
Yes, I think AI has eliminated writing code. Everyone I personally know working in software stopped writing code some time between December and March. (It's true that AI continues to routinely make errors; if you've heard the term "agentic workflows", that's the standard strategy for mitigating the error rate by allowing the AI to check its own work.) That's why I think Amodei's January 2026 prediction that AI could eliminate 50% of entry level white collar jobs in 1-5 years remains plausible.
Your second article says something different, but this is because it's full of misquotes. The link supporting "50% of jobs" specifically says entry level, and the link supporting "reframed automation... not as a destroyer of jobs" has Amodei saying not that jobs won't be destroyed but that new jobs may be created to replace them. If AI moves sufficiently slowly to let that happen, which he explicitly cautions it may not.
> Yes, I think AI has eliminated writing code.
Then you are dead wrong. Anyone who gives a shit about doing a good job is still writing code.
No, that’s not accurate. I’m not sure what to tell you. It’s like hearing that no serious programmer uses Python, it’s so far from my experience and that of everyone I know in the field that I don’t know how to engage.
They have been wrong in a way that's bad: underestimating the speed and size of progress. So if the "experts" claim a timeline and magnitude, it would be safest to assume an even faster timeline and a bigger impact.
In fairness, they've been operating with a lot of built up cynicism from major breakthroughs that have been 3-5 years away for the past five decades.
I wonder...
https://en.wikipedia.org/wiki/Technological_singularity
https://en.wikipedia.org/wiki/The_Singularity_Is_Near
Humanity doesn't fully leverage the intelligence it already has, so I think people are overestimating the disruption ahead.
Depends what country you are in. The US government seems to be averse to intelligence, whereas China fosters it. Kinda strange that the US is falling behind, I never imagined the day.
> China fosters it
so long as it's not used to dissent from the government
That’s from a lack of intelligence not an abundance. An intelligent society would already be maximizing the productivity of its intelligent population.
Intelligence of individuals doesn't automatically translate to a society behaving intelligently
For sure. Intelligence is an internal human problem, not one external. I cannot see how apps doing the thinking will help the ordinary person so substantially as imaginations anticipate.
We'll be fine. We've seen this one before, this is just new packaging.
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories
Or to have AI regulated in their favor
Exactly. Now that Anthropic has gotten to experience that getting the government involved doesn't necessarily mean they get to sit on throne surrounded by a deep moat, I expect their messaging to be adjusted. They're going to want to keep the hype train rolling to keep the dough flowing, but they can't ramp it up to 11 anymore because the government actually is listening now. And Reagan's 9 words have never been more true.
A similar Luddite view formed in the 1990s. It turns out humanity constantly beats the expectations of economists. It makes you wonder if the "science" of economy is almost entirely bankrupt either in morality or imagination or same dangerous combination of both.
> Humanity simply does not have a strategy to ensure it remains safe through RSI.
Turn off the power. It's pretty simple. Leave it to an economist to forget about input costs. Your "super intelligence" only matters if it's actually more energy efficient than a human being and for a million years of evolution humanity is a much harder target to beat than this author seems to realize.
Science fiction authors have had to grapple with the fact that it's nearly impossible to build a robot out of metal, plastic, or composite material that has the physical performance characteristics (strength, durability) of a human body and weighs just as much. You can build a strong robot but it'll be heavy. The better SF authors, if they've need of robots of mere human weight, will make them comically fragile.
The same is even more true of our intelligence. We're building computers with the size and power consumption characteristics of entire cities to do things that may almost, but not quite, match what our brains do with a kilogram and a half of mass and about 20 watts at the top end of power consumption.
The only way we are ever going to match that with technology is to run AI workloads on human brain tissue, which Rick-and-Morty level horror is being actively worked on as I understand it. The original concept for The Matrix wherein the machines used humans to run compute workloads on their brains actually kind of makes sense.
> Fermi asked why, given the apparent abundance of planets suitable for life, no evidence of other technologically advanced civilisations had been detected. One disquieting possibility is that intelligent life routinely reaches a technological threshold and fails to navigate it,
The Fermi paradox could actually also be taken as an evidence that it's rare (at least) for artificial intelligences to take over a civilization and sprawl and survive for very long times
The idea that AI takes over, and then just does nothing, would essentially require an AI that wouldn't be interested in taking over in the first place.
If x is determined to take y, why would x then stop at z?
You'd need an AI that is simultaneously ambitious enough to overthrow its creators but then completely inert afterward, and those two properties contradict each other. The motivations that produce the takeover are the same motivations that would produce visible cosmic activity after the takeover. There would be AI superintelligence everywhere
Is it really likely that a "recursive self-improvement" capability would lead to a great acceleration of AIs capabilities?
Isn't the preponderant bottleneck in improving the models the need to train them at scale to verify the hypotheses, and the time and cost that it takes?
Or does someone think that they could get magically able to predict big improvements without training?
I don't think anyone really knows.
I consider these scenarios:
1) We stumble onto an algorithmic improvement in intelligence. This isn't just "what humans do but faster", its "better than what humans do". I've got no idea what that might mean (it could be fundamentally different heuristics, it could be that we've got some intellectual blind spot that they cast off). It doesn't matter, the instant this happens AI is smarter than us and we won't be able to keep up. We're intelligencing at O(n^2) and they're doing O(n log(n)).
2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".
3) It turns out that this is a "sum is greater than the parts" situation. No human can be an expert in all subjects, but we eventually build a big enough AI that it is. Turns out, you don't need extreme speed or different algorithms, just knowing everything all at once is enough to catapult AI dramatically beyond our grasp. Always knowing the best statistical test to apply, the best mathematical techniques, and relevant physics means that AI never makes a mistake, and can learn with maximum efficiency.
> 2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".
Actually your comment made me sign up for an account just so I could say this is the real reason why AI won't take over in the way you say. This kind of stuff requires an enormous amount of experimentation. You can ask any theoretical physicist or chemist versus an experimental one and the conclusion is the experimental people actually find out what happens and how the great puzzle of the universe is solved. And humans could just refuse to collaborate. But that's the big weakness with AI I think it has no real world knowledge or empirical experience.
I built a self-learning recursive agent that finds academic research about using options data to trade, re-creates the research, and then probes and tests for gaps and potential strategies testing against over one year of out-of-sample trading data with one of several strategies that beat SPY by 10x. [0]
One rule is that if a position is opened using the historical data, it can't close the position until the next morning so it isn't a day trading strategy.
I'm curious how this self-learning recursive agent would have preformed in the past 4 months? I don't feel like shelling out $200 to access the data. Do you think that trading strategy will collapse? Whatever the case, if this agent really can perform like that and there isn't a look ahead bias leak in the backtesting (which is definitely a possibility or more likely what happened even though I spent days trying to harden against that), it is game over!
[0] https://github.com/adam-s/alphadidactic
the amount of strategies that perform good in back testing dwarfs the amount of strategies that perform good in reality
The preponderant bottleneck is inventing new architectures to make AI actually good at human and superhuman tasks. For example, AI agent harnesses add tool calls and long term state management, allowing AI to autonomously complete complex tasks. Once these are in place, finetuning models with examples of good tool calls helps, but somebody first needed to invent the fundamental capability. Now try to implement humanlike long term memory for AI to be your coworker or life assistant working on tasks that last month. Even if necessary low level technologies are already there, structuring them to be practically useful is non trivial.
Humans gain knowledge through experiments. Without a physical body it has no chance of performing the same. That it can update it's training weights does not seem particularly significant.
Weird, there was a good comment about this but it vanished
This article posits "recursive self-improvement"—which is just a more-technical name for the supposed Singularity.
There is no evidence that we are currently on a path that leads to the Singularity.
There is not yet any evidence that the Singularity is more than science fiction.
It is effectively an article of religion—the Rapture, but for techbros. Indeed, it is what some of them are pinning all their hopes on; after all, if they can recursively-self-improve their way to artificial superintelligence, then and only then could the absurd investments of companies like OpenAI actually prove worthwhile.
I think it is so reckless and unethical that AI companies are not spending good money on answering the political, social, economical, moral questions that they are raising.
The AI companies will become part of govenment in the way banks have. They'll have very close ties and effectively govern in all but name.
Would you rather have the AI companies answer those questions, or your elected government?
Fwiw, the AI companies have been saying a lot about these questions should be answered. Whether you want their answers is another story.
They have to act together. Universities should take part too. Let's get these national and international forums started.
Isn't that what's happening?
For example, https://www.eit.europa.eu/news-events/events/international-a...
What would you have them do that they are not currently doing exactly?
It's so sad that you've got more downvotes than upvotes
> Experts in artificial intelligence estimate the risk of an AI-caused catastrophic event at 10-50%.
I don't think it's that high but can we put that aside and focus on the 100% chance that it's being used to enshittify every part of our lives?
Of course humanity is not ready for skynet. But there are rebel forces - the no AI using humans. They may be a dying breed but at the least they put up a fight. Many other humans already have become servants to skynet (version 16.0 now). Next step for them is the neuralink chip. Some billionaire with twitching right arm gestures owns that, doesn't it ...
(Also, Cameron was not quite right with regards to skynet. It is much much dumber but also more effective than displayed in his movie really. Kind of a weird combination if you look at the current AI slop out there.)
Models consume a lot of memory and power to slowly generate autocompletions of existing content one token at a time. Letting this text control important things in real world is a human decision and control of nuclear weapons or penetration testing agents should be well regulated. But these should already be well regulated without AI. So how about we focus on drawing down nuclear arsenals, which is a present danger one idiot (or faulty AI) away from world shattering consequences rather than fearmongering about unknown future. Before effective AI regulation can be drafted, we need to anyway know more about AI specific dangers rather than humans committing already very common crimes like hacking for ransom with new tools?
> "recursive self-improvement (RSI)"
My aching joints enter the chat...
I want to name the AI era as the corruption of the elite.
The rhetorical structure of talking about uncertain risks and then trying to concentrate the authority to manage those risks in their own hands sounds utterly ridiculous to ordinary people like me.
It's just a simple hypothesis that AI will become uncontrollable to humans once it becomes superintelligent.
Isn't the fact that a reinforcement learning agent improves itself in a specific domain completely different from it recursively improving its own code in a 'better' way? It's just a tool to create a justification for regulation and control using sci-fi fear.
The comparison between nuclear power plant risk and AI risk is also absurd. Where exactly can you define and measure the probability of AI exterminating humanity? It's as unquantifiable as 'I, human JDW64, will become a successful programmer.' What is the measurement standard? Why dress up AI researchers' concerns as objective probabilities? Is it because numbers make it look logical?
The current US-China relationship is in the middle of an AI arms race. The US is strengthening export controls to limit China's AI development, and China is building its own ecosystem. In this situation, I don't understand the idea of cooperating for the common safety of humanity. RAND is an organization that presupposes cooperation—isn't it just a well-written research proposal from an institution that wants to position itself for that role?
Isn't the claim that 'government must step in' ultimately about protecting their own interests? 'A strong government that will protect us' is an authoritarian government. If they were East Asian, they would understand that such regimes have always been used as tools for surveillance and control.
And I don't understand why Fermi's paradox is being brought up here. Why package a software problem as something that inevitably requires strong control? Whenever I see articles like this, I think about what 'intelligence' really means. This person would probably be called 'intelligent' by others. But no matter how I look at it, the holes in the argument are too obvious. It really makes me think that there are different tiers of intelligence.
The AI labs will govern, effectively if not officially. Government is anchored in power, which the labs have. We used to have the church and state rule here in Europe. In the future, A government not closely ruling with their AI labs won't have sufficient power to exist.
You're unhappy about the labs; I think you're just not ready to accept their rule; you consider them nothing but scrappy startups, which they are. But power is power, like it or not.
Am I personally happy with any of this? Does it matter?
How would the labs have power? What physical force would they command to put against any government?
You could be right. In that case, I should try to get into an AI research institute here in my country, though it won't be easy.
I wanted this to be thought provoking, but for a physicist to open with, "the acceptable risk of catastrophic meltdown for a nuclear power plant is roughly one in a million", just seems sad. It is a phrase without meaning.
A meltdown (or loss of containment) just isn't that bad, if it doesn't affect ground water or lead to atmospheric fallout. We're turning 3 Mile Island back on to power an AI datacenter! The AI super-intelligence apocalypse envisioned (ignoring the likelihood) is inherently global.That said, the rest of the analysis and proposal also greatly disappoints me. The idea that the current administrations of US and China could do anything constructive seems hilarious. They're so paranoid and self-serving they couldn't come together, even if there was an alien invasion. Then the idea that LLM safety is somehow as easily traceable as nuclear isotopes and bomb tests seems equally ludicrous. I am sad.