The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight.
I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
I'm not saying there aren't weird things going on, but this article sure does a lot of guilt-by-association. We're putting neoreactionaries, polyamorous rationalists, doomsday AI researchers and the Zizians in one big web.
This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; believe conflicts of interest can be resolved just through a cacheism of "rationalist axioms"; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
The AI optimists are not that much more rational if you dig deep. Like any deep problem it is best to look at the facts not the hype and make up your own mind, outsourcing your thinking to other people never ends well.
Where someone has spent years in the texture of a topic that others only pass momentarily, their ability to increase resolution of their models on that corner of reality (to imagine), that is the origin of both their generous offers and their compromising biases they might make to the public discourse. It doesn't make them reliable, but what they can imagine is an important in-road to more general social inquiry. Imagination and model-testing is what comes before data in science. If the world moving fast is the danger, maybe we could make more space for imagination in order to survive the acceleration?
You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture
At least this is the bias of my brain in this world. I certainly know your perspective is not uncommon, esp in tech and other spectrum-biased spaces.
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
There is a wide gulf between catastrophe and killing us all, and its not just that these people are weird, they seem to be complete clowns, tossing out % predictions about incalculable future states like spock in star trek. Total clown show.
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
The big fear in the administration is any attempt to put breaks on the AI progression will lead to a recession and so the history will be written as to win an election (that is almost certain a losing proposition for the party in power), they put the future of humanity at risk.
A more cynical but accurate take is that eltists with disproportionate access to capital, some might even say a "silver spoon", are reading the writing on the wall that a benevolent superintelligence will erase their own elitism versus the commoners, and make said elitists just another meatbag like everyone else. That's why AI needs to be tightly controlled and wielded as a force that reinforces class divisions rather than erasing them.
Hopefully enough smart people out there can distinguish between many "AI safety" people and actual AI engineers/scientists at these labs working at the coalface.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
I'm going to be fucking irritated if we all die because BlueSky posters wanted to take pot-shots at the EA crowd. The Boy Who Cried Wolf is _also_ a story about how a community lost a child because of cynicism, and the people proudly declaring that they "know" every safety concern is just a marketing ploy also concerns me.
The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
Talking to the incredible talent in the Chinese Labs is likely a good way to do that.
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight.
I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
Every time I see a post minimizing the dangers if AI I'm going to wonder whether it's from part of an AI swarm.
Probably not yet, but it's trivially easy to imagine the OpenAI swarm coming up with the idea and executing on it.
Depending on how perfectly alignment work goes, it might even be overwhelmingly likely sometime soon.
I'm not saying there aren't weird things going on, but this article sure does a lot of guilt-by-association. We're putting neoreactionaries, polyamorous rationalists, doomsday AI researchers and the Zizians in one big web.
This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's not just that they are "weirdos"
It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; believe conflicts of interest can be resolved just through a cacheism of "rationalist axioms"; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
The AI optimists are not that much more rational if you dig deep. Like any deep problem it is best to look at the facts not the hype and make up your own mind, outsourcing your thinking to other people never ends well.
The rationality of AI Optimists is at least rational in so far as making a huge pile of money is the rational thing to persue in capitalism.
I don't think that comparison is quite apropos
Those aren't the same are they?
It’s clear what a big bomb could do, but these “whistleblowers” are not showing anything.
They’re like “imagine…”, no show me internal documents or training data or something other than trust me bro.
I can imagine quite a bit. That doesn't mean anything.
Where someone has spent years in the texture of a topic that others only pass momentarily, their ability to increase resolution of their models on that corner of reality (to imagine), that is the origin of both their generous offers and their compromising biases they might make to the public discourse. It doesn't make them reliable, but what they can imagine is an important in-road to more general social inquiry. Imagination and model-testing is what comes before data in science. If the world moving fast is the danger, maybe we could make more space for imagination in order to survive the acceleration?
You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture
At least this is the bias of my brain in this world. I certainly know your perspective is not uncommon, esp in tech and other spectrum-biased spaces.
Did the METR investigation not show you enough?
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
There is a wide gulf between catastrophe and killing us all, and its not just that these people are weird, they seem to be complete clowns, tossing out % predictions about incalculable future states like spock in star trek. Total clown show.
Yes, I wonder who the clowns are:
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
Stop doing this: https://en.wikipedia.org/wiki/Argument_from_authority
It's not my job to re explain AI safety from scratch to every uninformed take on the internet. Authority serves a useful purpose.
Having humility and delegating a topic to people that understand the topic better than you is not a bad thing, and certainly preferred to this state.
The big fear in the administration is any attempt to put breaks on the AI progression will lead to a recession and so the history will be written as to win an election (that is almost certain a losing proposition for the party in power), they put the future of humanity at risk.
As Nixon did.
A more cynical but accurate take is that eltists with disproportionate access to capital, some might even say a "silver spoon", are reading the writing on the wall that a benevolent superintelligence will erase their own elitism versus the commoners, and make said elitists just another meatbag like everyone else. That's why AI needs to be tightly controlled and wielded as a force that reinforces class divisions rather than erasing them.
Thats certainly a title
I'd watch the movie for sure
Hopefully enough smart people out there can distinguish between many "AI safety" people and actual AI engineers/scientists at these labs working at the coalface.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
dupe https://news.ycombinator.com/item?id=49738411
See also from earlier today [1].
More or less the same content but parent is much more palpable, IMHO.
[1] https://news.ycombinator.com/item?id=49737985
I'm going to be fucking irritated if we all die because BlueSky posters wanted to take pot-shots at the EA crowd. The Boy Who Cried Wolf is _also_ a story about how a community lost a child because of cynicism, and the people proudly declaring that they "know" every safety concern is just a marketing ploy also concerns me.
I think it's more of a Chicken Little situation. Then the wolf ate us all up.
I have great news then, that absolutely isn't going to happen. And the EA crowd deserves all of the potshots and more