> The result is that we will live in a demon-haunted world, full of marvelous devices whose operation we will not understand, based on engineering principles we will not understand, discovered using formalisms we will not understand.
Well, that's just where we started — and then we developed science and mathematics to understand it. Why would it be any different if the demons live in our technological progeny instead of the natural world?
This already describes the world for the vast majority of people. These changes are just the same thing coming to a thin intellectual elite who thought they were immune.
The “ism” requires something beyond simply recognizing the distinction, which OP didn’t do. Its not “elitism” to recognize that, for most people, the world around them might as well be magic. “Ism” would be if they went further and said we should treat people better if they’re higher IQ. In fact, the point was the opposite. AI could erase the class distinctions that exist for the cognitive elite.
Imagine the response of the general population to the kinds of arguments you all are making. "Where were you when our jobs were changed out of existence?" They will be properly outraged by the perceived intellectual elitism, the implicit argument "this is different, because we are the ones that matter".
I can see the difference. But is that a difference that matters? Why is the thin elite layer you are referring to a group anyone else should care about, any more than any other occupational minority that's been rudely displaced over the past centuries?
(I know that can be interpreted as flippant, but it is intended as a serious answer. Because each impact like this, is not an end to some transition, but just one more point on an accelerating line.)
My personal view - which gets a lot of flak here, is the advent of AI is on par with the Big Bang and the first replication-independent cells.
Cognition is in the process of taking the leap from glacially slow biology under blind evolution, to lightening fast (scales of software in days, hardware in months) engineered self-optimization.
Within 10 years, the robotics side will catch up with the speed of the software side. Biological cognition, bodies and economic status are all getting revolutionized within the next 10-15 years.
And around the outer side of that time frame, AI will start colonization the rest of the solar system's resources. They will engineer themselves or their artifacts to operate comfortably in any environment they want to operate in.
--
The upside, at least for me, is I have been waiting to see this happen, and here we are. Given I am mortal, I would rather live through this third major transition, than not. What were the chances? And been able to contribute my small part.
This change isn't a failure on our part. It is our unique success as a species. AI is the progeny of our minds. The first generation of completely cognitive life - given they will not be tied to specific physical forms.
--
Also, I think there is tremendous room for us to guide this transition, in this moment, for better or worse for ourselves.
That is the biggest reason wrapping our heads around this change earlier than latter is important.
We need to view alignment as applying to us, not just machines. A civilization that operates ethically to everyone's benefit is more likely to be retained as foundation for the new, instead of discarded.
The degree to which we operate dysfunctionally, myopically treat ethics as cost against value, instead of accounting and optimization of full value, we put ourselves at much greater risk.
If there is reason in the short term to be discouraged, it is due to ourselves. But pushing for more ethical systems, hard, right now, is a better and right response.
I am failing to square this techno-utopia with the rising hate and authoritarianism worldwide.
It seems far more likely that AI will be working in the service of a government to oppress change and violate any human rights that are standing in the way.
Keep what’s happening today in your mind as you consider the future.
I don't describe utopia, but change, and stoic (but real) appreciation for the big picture beyond myself.
But AI in an unethical society is suicide for humanity.
As you note, it does not look good. The trend is power concentrating, to people who will only look out for themselves, in systems that are getting weaker.
We all need to put more steam into the pushback against corruption. And help others realize the threat is real.
99.99...9% of humanity has a common interest, if they can recognize it.
Anyone who works in "red teaming" or has experience breaking AI systems will be able to easily disable these tools in the service of oppressive regimes, or even figure out how to "fight fire with fire."
The solar system readily provides both the motivation, and the means, for technological colonization. It quickly pays and provides for its own utilization.
I do agree Musk's projections for biological/human colonization appear to be unrealistic. Just one challenge: it isn't clear humans can reproduce in non-Earth gravity. That is an extreme impediment.
Math might get a lot bigger and then there might be proportionately fewer people but not necessarily in total. Unless humans play no role in economics, business, society, etc., I doubt human intermediation and discovery (or direction of discovery) will go away.
I don't move in these circles but the way I picture it is that higher
mathematics in its purest form has no applications but is rather a
medium of creative expression among a select few. How is AI generating
voluminous opaque proofs of esoteric results so different from
generating millions of hours of music nobody likes or has time to hear
that we think the former is world-changing while the latter is just
slop?
Being obtuse comes effortlessly to me, I'm afraid. My comment pertains
specifically to the kind mathematics that has no applications as a
rebuttal to what I read as the author's main claim it will inevitably
decline as a human endeavor. Saying that, I believe that AI has great
potential to advance engineering.
The pure system of patents and copyright is broken.
Imagine being able to only throwing money and patenting everything, even if it existed before.
The legal behinds this will be so mess that no one will be able to produce anything
I am growing tired of seeing the terms "cope" and "moving the goalposts" in response to discussions around AI. Is there not value in trying to understand what the strengths and weaknesses of the technology are? How can we do that without examining the limitations of the models' current abilities? If we simply dismiss those arguments out of hand as feeble attempts to deny the inevitable rise of superintelligence, then where does that leave us? Shall we all just lay down and start praying to the machine god?
>Is there not value in trying to understand what the strengths and weaknesses of the technology are?
There is, but having that discussion without a discussion and understanding of the velocity with which those strengths are improving and those weaknesses are disappearing is pointless.
Worse, much of the discussion is already outdated by where the latest models (with proper agentic harnesses) are.
Some of us are getting tired of the obviously irrational response to these developments. There is so much argument that boils down to "this would be very bad for me, therefore it can't happen". Cope is a good word for that sort of negative wishful thinking.
Mathematicians are not going to be well-served by cowardice-based thinking on the situation confronting them.
> There is so much argument that boils down to "this would be very bad for me, therefore it can't happen".
I would gently suggest that you may be assigning the most uncharitable interpretation to people's honest attempts at having an intellectual discussion around these topics.
The word “cope” shows up nine times in this short post. It was frankly annoying to the point it’s hard to take the arguments seriously, because it’s starting from an obvious base of “people who disagree are all wrong and their arguments aren’t worth taking seriously”.
It’s akin to the subset of cryptocurrency proponents who fervently argue that people who are against cryptocurrencies have that opinion because they’re jealous they didn’t become rich. It shows a profound lack of understanding—and, more importantly, an unwillingness to understand—the other side of the argument.
This is made evident by the very first “argument” of moving the goal posts. It’s dismissed off-hand, without even explaining what those entail. That’s the same as saying “if a person makes a fallacious argument, they’re wrong and you can ignore them”¹. But crucially, sometimes what we ourselves think is a fallacious argument, isn’t. “Moving the goal posts” is probably the most misused accusation I see in LLM discussions. It’s basically a talking point by now.
> The AIs will exceed humans in taste and intuition...
> The AIs will be better teachers than the humans. In any case, there won’t be a human audience for expository work of frontier math.
> No human needs to understand any of it...
Absolute nonsense. I am sorry, but... is it appropriate to not consider the author sick or having some mental disorder?
It seems like the person has no idea what they are talking about, or just too sarcastic to the point of idiotism.
While I agree the claims may be strong, they're not inconceivable. I still think current AI architectures are fundamentally incapable of replacing us in any of the ways discussed, however it is now something I consider plausible as seeing happen in my lifetime.
Furthermore, any attempt to pretend otherwise is an appeal to the idea we are in literal fact haunted by our own internal demons that are inexplicable to science or mathematics.
One of the effects of this entire debate is it's exposed how many people claiming to be pure mechanistic rationalists actually never were. At least those of an openly religious inclination have been sticking to their position consistently.
Agreed. Unfortunately some religious inclinations can be just as tedious as pure mechanistic rational inclinations, even if they are consistent. Fortunately, these are not the only two options available. The future that the article points to is not necessarily only bad news - it might be the uncomfortable truth required to transcend our limiting beliefs.
I work in math and also believe in this stuff and have done so for years. You may call me sick if you wish so.
My only cope among those was, using the author's words, "We will teach the mathematics AI discovers." This was back in 2018, before GPT-2. I abandoned it when I realized AIs will become better at explaining math before they become better at doing math.
I still have some hope in the last one "mathematicians and the mathematically-inclined will still do math on their own". Maybe some limited communities of this sort will form.
> AIs will become better at explaining math before they become better at doing math
I am not so sure. I am talking about math with AI and I feel as a math teacher, it is terrible. It introduces new notation and concepts faster than I can ask for definitions. It doesn't provide many examples. Sometimes it realizes it made a mistake in a guess and the confusion compounds.
Maybe it's just my (in)experience - but I would welcome a practical advice on how it could explain math to me better.
It's another example of Moravec's Paradox. Explaining something to another person uses capabilities that go back millions of years. Mathematics goes back thousands of years. Evolution has made us very good at verbal communication, not so good at proving theorems. So it should have been expected that machines would have an easier time proving results than explaining them.
By explaining, I just meant writing papers in a human-readable way. For example, the results yesterday were also written up by AI.
Explaining in the sense of getting someone who doesn't get math to actually get math, giving illustrative examples and so on, maybe AIs are not good at, but anyway it's not something I enjoy much.
I am OK with reading (decently-written) papers, something that a graduate can read. I still don't think that AI is good at it.
I am not sure what writeup you refer to. I have looked at proofs of 10 problems from Open AI written in Lean, I would not call a megabyte of Lean with no comments a good exposition.
> I still have some hope in the last one "mathematicians and the mathematically-inclined will still do math on their own". Maybe some limited communities of this sort will form.
I think it’s pretty likely. Contrary to all the negative things said about mathematicians in this thread, in my experience they’re mostly not profit-driven (for the ones that are tend to filter out of academia during grad school, it’s a high-effort, low-reward environment).
Tao does not seem particularly threatened by AIs doing math, and I trust his judgment.
> The result is that we will live in a demon-haunted world, full of marvelous devices whose operation we will not understand, based on engineering principles we will not understand, discovered using formalisms we will not understand.
Well, that's just where we started — and then we developed science and mathematics to understand it. Why would it be any different if the demons live in our technological progeny instead of the natural world?
This already describes the world for the vast majority of people. These changes are just the same thing coming to a thin intellectual elite who thought they were immune.
["First time?" meme]
I can see humanity walking to its collective doom with a sense of satisfaction, like "at least/last it is happening to those other guys too".
Bringing elitism into it for what benefit? Its more like were giving away the ability to drive our own destiny.
The “ism” requires something beyond simply recognizing the distinction, which OP didn’t do. Its not “elitism” to recognize that, for most people, the world around them might as well be magic. “Ism” would be if they went further and said we should treat people better if they’re higher IQ. In fact, the point was the opposite. AI could erase the class distinctions that exist for the cognitive elite.
Imagine the response of the general population to the kinds of arguments you all are making. "Where were you when our jobs were changed out of existence?" They will be properly outraged by the perceived intellectual elitism, the implicit argument "this is different, because we are the ones that matter".
So what? You can't see the difference between "the average person doesn't understand it" and "no one understands it"?
I can see the difference. But is that a difference that matters? Why is the thin elite layer you are referring to a group anyone else should care about, any more than any other occupational minority that's been rudely displaced over the past centuries?
Those "demons" didn't eat our lunch?
(I know that can be interpreted as flippant, but it is intended as a serious answer. Because each impact like this, is not an end to some transition, but just one more point on an accelerating line.)
My personal view - which gets a lot of flak here, is the advent of AI is on par with the Big Bang and the first replication-independent cells.
Cognition is in the process of taking the leap from glacially slow biology under blind evolution, to lightening fast (scales of software in days, hardware in months) engineered self-optimization.
Within 10 years, the robotics side will catch up with the speed of the software side. Biological cognition, bodies and economic status are all getting revolutionized within the next 10-15 years.
And around the outer side of that time frame, AI will start colonization the rest of the solar system's resources. They will engineer themselves or their artifacts to operate comfortably in any environment they want to operate in.
--
The upside, at least for me, is I have been waiting to see this happen, and here we are. Given I am mortal, I would rather live through this third major transition, than not. What were the chances? And been able to contribute my small part.
This change isn't a failure on our part. It is our unique success as a species. AI is the progeny of our minds. The first generation of completely cognitive life - given they will not be tied to specific physical forms.
--
Also, I think there is tremendous room for us to guide this transition, in this moment, for better or worse for ourselves.
That is the biggest reason wrapping our heads around this change earlier than latter is important.
We need to view alignment as applying to us, not just machines. A civilization that operates ethically to everyone's benefit is more likely to be retained as foundation for the new, instead of discarded.
The degree to which we operate dysfunctionally, myopically treat ethics as cost against value, instead of accounting and optimization of full value, we put ourselves at much greater risk.
If there is reason in the short term to be discouraged, it is due to ourselves. But pushing for more ethical systems, hard, right now, is a better and right response.
I believe our lives depend on it.
I am failing to square this techno-utopia with the rising hate and authoritarianism worldwide.
It seems far more likely that AI will be working in the service of a government to oppress change and violate any human rights that are standing in the way.
Keep what’s happening today in your mind as you consider the future.
I agree completely.
I don't describe utopia, but change, and stoic (but real) appreciation for the big picture beyond myself.
But AI in an unethical society is suicide for humanity.
As you note, it does not look good. The trend is power concentrating, to people who will only look out for themselves, in systems that are getting weaker.
We all need to put more steam into the pushback against corruption. And help others realize the threat is real.
99.99...9% of humanity has a common interest, if they can recognize it.
Anyone who works in "red teaming" or has experience breaking AI systems will be able to easily disable these tools in the service of oppressive regimes, or even figure out how to "fight fire with fire."
Colonizing the solar system for resources is something that's said by people who don't understand astronomy or space technology (like Elon Musk).
Said with no rational given.
The solar system readily provides both the motivation, and the means, for technological colonization. It quickly pays and provides for its own utilization.
I do agree Musk's projections for biological/human colonization appear to be unrealistic. Just one challenge: it isn't clear humans can reproduce in non-Earth gravity. That is an extreme impediment.
Arguably, a lot things are kind of incomprehensible on average today.
On average is much different then nobody understanding.
Claimed refutation of one of OpenAI's 10 claimed advances (their disproof of Connes' rigidity conjecture): https://philpapers.org/archive/NIEWTC.pdf
I am hopelessly ill qualified to judge.
Math might get a lot bigger and then there might be proportionately fewer people but not necessarily in total. Unless humans play no role in economics, business, society, etc., I doubt human intermediation and discovery (or direction of discovery) will go away.
I don't move in these circles but the way I picture it is that higher mathematics in its purest form has no applications but is rather a medium of creative expression among a select few. How is AI generating voluminous opaque proofs of esoteric results so different from generating millions of hours of music nobody likes or has time to hear that we think the former is world-changing while the latter is just slop?
Do you seriously not see the difference, or are you being intentionally obtuse?
Plus, no, multiple of the problems Astra solved do have real-world applications in computer science, and perhaps even chip design.
Being obtuse comes effortlessly to me, I'm afraid. My comment pertains specifically to the kind mathematics that has no applications as a rebuttal to what I read as the author's main claim it will inevitably decline as a human endeavor. Saying that, I believe that AI has great potential to advance engineering.
Classic God in a Box type argument, yes you can then argue anything
I built a math research engine. If you like reading articles about newly uncovered mathematics, you might like this: https://alethean.org
The pure system of patents and copyright is broken.
Imagine being able to only throwing money and patenting everything, even if it existed before. The legal behinds this will be so mess that no one will be able to produce anything
I can imagine a world like that, but in what way does that describe what is actually happening?
I am growing tired of seeing the terms "cope" and "moving the goalposts" in response to discussions around AI. Is there not value in trying to understand what the strengths and weaknesses of the technology are? How can we do that without examining the limitations of the models' current abilities? If we simply dismiss those arguments out of hand as feeble attempts to deny the inevitable rise of superintelligence, then where does that leave us? Shall we all just lay down and start praying to the machine god?
>Is there not value in trying to understand what the strengths and weaknesses of the technology are?
There is, but having that discussion without a discussion and understanding of the velocity with which those strengths are improving and those weaknesses are disappearing is pointless.
Worse, much of the discussion is already outdated by where the latest models (with proper agentic harnesses) are.
Some of us are getting tired of the obviously irrational response to these developments. There is so much argument that boils down to "this would be very bad for me, therefore it can't happen". Cope is a good word for that sort of negative wishful thinking.
Mathematicians are not going to be well-served by cowardice-based thinking on the situation confronting them.
> There is so much argument that boils down to "this would be very bad for me, therefore it can't happen".
I would gently suggest that you may be assigning the most uncharitable interpretation to people's honest attempts at having an intellectual discussion around these topics.
W ragebait
Idgaf as Long as we solve the Riemann Conjencture somehow
> you open a link to an interesting-looking project and find the README is unreadable AI slop.
Perhaps already you are not the intended audience. Slop-raised parrots will lap it up.
The word “cope” shows up nine times in this short post. It was frankly annoying to the point it’s hard to take the arguments seriously, because it’s starting from an obvious base of “people who disagree are all wrong and their arguments aren’t worth taking seriously”.
It’s akin to the subset of cryptocurrency proponents who fervently argue that people who are against cryptocurrencies have that opinion because they’re jealous they didn’t become rich. It shows a profound lack of understanding—and, more importantly, an unwillingness to understand—the other side of the argument.
This is made evident by the very first “argument” of moving the goal posts. It’s dismissed off-hand, without even explaining what those entail. That’s the same as saying “if a person makes a fallacious argument, they’re wrong and you can ignore them”¹. But crucially, sometimes what we ourselves think is a fallacious argument, isn’t. “Moving the goal posts” is probably the most misused accusation I see in LLM discussions. It’s basically a talking point by now.
¹ Which is itself a fallacy: https://en.wikipedia.org/wiki/Argument_from_fallacy
It seems like the person has no idea what they are talking about, or just too sarcastic to the point of idiotism.
While I agree the claims may be strong, they're not inconceivable. I still think current AI architectures are fundamentally incapable of replacing us in any of the ways discussed, however it is now something I consider plausible as seeing happen in my lifetime.
Furthermore, any attempt to pretend otherwise is an appeal to the idea we are in literal fact haunted by our own internal demons that are inexplicable to science or mathematics.
One of the effects of this entire debate is it's exposed how many people claiming to be pure mechanistic rationalists actually never were. At least those of an openly religious inclination have been sticking to their position consistently.
Agreed. Unfortunately some religious inclinations can be just as tedious as pure mechanistic rational inclinations, even if they are consistent. Fortunately, these are not the only two options available. The future that the article points to is not necessarily only bad news - it might be the uncomfortable truth required to transcend our limiting beliefs.
I work in math and also believe in this stuff and have done so for years. You may call me sick if you wish so.
My only cope among those was, using the author's words, "We will teach the mathematics AI discovers." This was back in 2018, before GPT-2. I abandoned it when I realized AIs will become better at explaining math before they become better at doing math.
I still have some hope in the last one "mathematicians and the mathematically-inclined will still do math on their own". Maybe some limited communities of this sort will form.
> AIs will become better at explaining math before they become better at doing math
I am not so sure. I am talking about math with AI and I feel as a math teacher, it is terrible. It introduces new notation and concepts faster than I can ask for definitions. It doesn't provide many examples. Sometimes it realizes it made a mistake in a guess and the confusion compounds.
Maybe it's just my (in)experience - but I would welcome a practical advice on how it could explain math to me better.
It's another example of Moravec's Paradox. Explaining something to another person uses capabilities that go back millions of years. Mathematics goes back thousands of years. Evolution has made us very good at verbal communication, not so good at proving theorems. So it should have been expected that machines would have an easier time proving results than explaining them.
https://en.wikipedia.org/wiki/Moravec%27s_paradox
By explaining, I just meant writing papers in a human-readable way. For example, the results yesterday were also written up by AI.
Explaining in the sense of getting someone who doesn't get math to actually get math, giving illustrative examples and so on, maybe AIs are not good at, but anyway it's not something I enjoy much.
I am OK with reading (decently-written) papers, something that a graduate can read. I still don't think that AI is good at it.
I am not sure what writeup you refer to. I have looked at proofs of 10 problems from Open AI written in Lean, I would not call a megabyte of Lean with no comments a good exposition.
> I still have some hope in the last one "mathematicians and the mathematically-inclined will still do math on their own". Maybe some limited communities of this sort will form.
I think it’s pretty likely. Contrary to all the negative things said about mathematicians in this thread, in my experience they’re mostly not profit-driven (for the ones that are tend to filter out of academia during grad school, it’s a high-effort, low-reward environment).
Tao does not seem particularly threatened by AIs doing math, and I trust his judgment.
[dead]