Usefulness relies on understanding, even if that means understanding what is possible to ask for. What is being generated right now is really just a translation from one inscrutable phenomenon into another. In neither case do we have power over the data, because we don’t know what it means, truly where it comes from, and even the beginning of the limits of what the discovery allows. We’ve traded one black box for another, and still need to unpack it.
> We’ve traded one black box for another, and still need to unpack it.
The human mind has been a black box, and largely still is.
So we traded one black box for another, in more than one way. The new black boxes are going to be able understand and keep up with their own work, much more quickly than we will.
The "alien" picnics, in this case, are a prelude to settlement and expansion, and we have only met the children.
Your point on understanding mostly holds (although our own mind's inscrutability but usefulness is a strong counter-case), but it will increasingly work against us.
We do have a "working hypothesis" how the mind works though, what psychology calls "theory of mind" or "folk psychology". While this is completely divorced from the level of neurons or brain regions and so might tell little of how the mind "really" works, it's detailed and useful enough that we can judge how other people would react, guess what attitudes, feelings and motivations drive them and in general can have a useful level of interaction.
The same is true for traditional computer programs. The standard argument against valuing understanding is "well, you don't understand the assembly, processor microcode and the electrical signaling when your program gets executed either, why do you care about understanding the high-level code then?"
I'd counter this by saying that high-level languages have (mostly!) self-contained abstractions that allow you to "understand" a program deep enough to make accurate predictions of its runtime behavior, answer "why" questions when errors happening and safely make changes to it - without having to understand all the stuff at lower levels.
In contrast, with AI, I see the risk that we lose understanding of both the low- and the high-level structure of things. Then we really end up in "unpredictable alien artifact" territory.
Usefulness doesn’t always rely on understanding. And it’s not a new thing in mathematics to have an answer without fully understanding it. We’re talking about a change in the rate of that happening, not a new phenomenon in itself.
I think Ted Chiang's story "The Evolution of Human Science" may be a significantly better metaphor here. What AI is dropping is (allegedly) just answers, not mysteries. In both cases, we don't really understand it, but in "Picnic", most of the things left behind aren't even useful, merely mysterious or more often dangerous. In "Evolution", answers (and only answers) and technologies arrive pre-packaged, with zero understanding.
> All this new knowledge is arriving at a pace much greater than the field’s ability to process it
That’s the whole issue: it’s not knowledge, it’s data. AI can not provide an account for the proofs, or be questioned about them, or organize a talk exposing the idea because there’s no understanding behind any of it.
This is what Tao has been saying; that the results, in the strictest sense, were never the goal. It is ultimately the understanding (that previously was required to form a proof) that is valuable.
But understanding to what end? I'm if it's for the pure aesthetic experience of engaging in the beauty of math than that's fine, but that's typically not the view from my experience.
The goal of understanding is to advance the field, since historically human understanding was the only way to do this.
For the vast majority of human history, the world we experienced was a strange and unintelligible space. Humans survived at the by the grace of forces beyond their control. For a brief window of time human existed in a world where they increasingly saw a relationship between their understanding of the world and there ability to steer their environment towards a desired outcome.
I believe now we're seeing that that was a historical anomaly. The world is once again entering a state where it becomes increasingly unintelligible to us. Nick Land already famously pointed out, many decades ago, that the global financial system is already a form of "artificial intelligence" we don't really control. We're seeing that spread wider and wider.
The world making sense peaked around 1960s and has continued to breakdown ever since.
Why wouldn’t the AI be able to answer questions about the proofs or organize a talk? Those are pretty much the things it excels at, exploring and summarizing previous knowledge. The problem is the volume of questions and data a human needs to step through to arrive at the same understanding.
Apparently, the knowledge isn't in the data set, and the explanations are gobbledygook. They certainly summarize well, but it seems that if the "understanding" isn't already baked in, then expanding the source description doesn't really clarify it. Maybe in the future?
What does apparently here mean? Is there an account of an expert sharing their attempts to get a model to explain the results and key ideas to them (for a topic they're familiar with) in an interactive session, and having it fail?
For somebody deeply familiar with a topic and all the nuance around the problem and approaches that didn't work, I'd be quite surprised if they couldn't cajole it into explaining what finally did work in terms they understand. Unless it turned out that these proofs were just well beyond anything humans were anywhere near being able to figure out, in which case, no great loss that the AI figured it out because we were nowhere near it.
I guess the issue with that is that the understanding is important to mathematicians. The majority of people outside mathematics don’t understand it whether it’s humans doing the work or AI. If AI can generate useful results more quickly, then to the general public it is more valuable.
Mathematical solutions aren’t valuable to the general public at all, other than the passing fancy that comes from reading headlines and news articles.
Mathematics is only useful to the extent that we can do something with it, and I would guess that still requires human mathematicians to understand it.
Does that mean AI is not doing something useful? If it provides a proof, it gives direction to human mathematicians who can then develop an understanding, give talks, be questioned about the topic, and teach. Is that not a useful service?
It might well be useful if the RIGHT proofs are evaluated and proven. Figuring out what will be useful has always been a real challenge in math. Finding certain symmetries and mappings between problems in different domains, might well be useful even if they aren't well understood? That has always been another part of the work of mathematicians. Of course there are those who just work on the beauty of things, and few in the 80s would have thought that prime factoring number theory or elliptic curves would be the basis for modern communications and banking.
That said, getting a point solution and not knowing the path to get there, is certainly much less useful, and generalization is much harder. Again, more work for mathematicians, but maybe without the easy "I solved it" proof metric for rewards. Goodhart's law strikes again.
Less useful, but still useful. If robot Ramanujan gives us answers with no mortal-understandable justification, but it's always right, that still gives decades or centuries of topics to try to build theory around, with the motivational benefit that you always know that what you're working on can be solved. Simply treat it as an oracle that dispenses magical opaque proof certificates.
I don't think that's the argument. In my opinion this is like paying to unlock characters, à la EA. Who's going to feel a sense of accomplishment by just trying to interpret what an LLM spat out? Sure,maybe the results will be used by someone in some field, the same way a product manager uses what the development team developed. Is the PM going to give talks about it? Are they going to expand on those developments and come up with something new?
> Who's going to feel a sense of accomplishment by just trying to interpret what an LLM spat out?
I think you have a point, but this is the reality we have to face. The genie is out of the bottle. Even if OpenAI stops publishing AI-generated proofs, it's only a matter of months before tools with similar capabilities are in everyone's hands.
Fair, “answers” is probably more precise than “knowledge.” That gap between answers and understanding is the whole point of the post, so I think we mostly agree.
I like this metaphor better: the mathematics community is like a group of ancient astronomers that are unexpectedly getting the output from a modern telescope. "Why even look at the sky anymore," some say. "And we don't even own the telescope!"
Meanwhile, the Rubin observatory is creating an archive that grows by 20 terabytes each night, and nobody really worries about this. Dealing with that amount of data is definitely a problem, but also a nice problem to have, provided that there's a scientific community that can deal with it. Mathematics has unexpectedly become Big Science and there needs to be a better relationship between the funders of the scientific instruments and the scientists themselves.
It's a bit confusing that the output from the scientific instrument looks like badly-written academic papers. Maybe it should be treated more like raw data? The format is not ideal, but historians (for example) often have to deal with far worse.
I think there is a difference here; the modern telescope doesn't output answers, claims or theorems - just data that still has to be interpreted to be even remotely useful.
I like that metaphor. But what it calls to my mind is an "alien artifact" which was found by a road, and "how did these refined metals get placed inside of a rock, in such an unnatural formation?" and there was much debate about this artifact. Years later it was determined that it was in fact a spark plug which had been remineralized by the effluent from a mine upstream: an unusual process acting on a discarded object, and people who want to believe. Nobody thought that the remineralization process could take less than hundreds of years, which ruled out "spark plug".
I tried "Monday Begins on Saturday" and had a pretty hard time getting through it. But I have to give it another chance - and check out the ones you mentioned.
Monday… is a very Russian book, heavily based on memes and folklore. You will have very hard time understanding it. I can’t recommend you to try it again.
The 2 books I’ve mentioned are very international on the contrary.
I am Russian, read it in the original. But I'm scarred by being forced to read meandering classics from a very young age, so I may just have a more visceral negative reaction to it that I have to get over.
It also reminds me, in a very similar way, of William Gibson's "Hinterlands". Some people are just hell bent on flying into the Alien Coordinates That Make You Crazy or Kill You but Sometimes Give You Cool Stuff you Don't Understand, from William Gibson's short story Don't Fly Into the...
It doesn't assume, it just observes. It has the capacity to use computers better than us, more persistently, in a more scalable way, to solve problems we haven't solved before like Navier-Stokes and many more to come. You can argue all you want about whether it's intelligent or not and what it means, but the solution is now here for us to process, understand, and integrate forward into our model of the world.
I really dislike this analogy. Trash and apple cores versus potential, objective discoveries?
I'm not saying that OpenAI's results are good or bad. But I can say that at least humans could try to understand them and get not only the discoveries themselves but some techniques that could help other progress down the road.
It's fair to ask whether we'll be better off as a result of AI Math. But many people are failing to see any sort of positive outcome and I think this is close minded and borderline irrational.
I don't blame Mathematicians for trying to protect their careers. Anyone would to the same in their place. But the rest of society should take a hard look at what's happening and, as the financiers of mathematical research, make an independent assessment of whether humanity needs "old school" mathematical research at all versus a model with a lot more AI.
That's probably not the point the author is trying to make. In the book humans are trying to make sense of the artefacts lefts by the aliens, but are only able to do so slowly. The gap between human understanding and the aliens who built the artefacts are just to large.
The book never tells us that this is actual trash, it's just a theory. If the theory is true however, the aliens might also not understand the damage they've done, it's just a bit of littering to them. Or they don't care, they just destroyed an ant hill, so what.
The author of the article might have a bit more faith in the current generation of AI than I do, I don't believe that LLMs are really capable of creating that big a gap. The discoveries they make are all based on human knowledge and understanding, so we can comprehend the foundations on which they are built. Now if models can train and iterate on their own, to a much greater extend, then maybe, some day, in the far future their discoveries could leap frog humans to the same degree as the aliens in "A Roadside Picnic" and not understand that their output is damaging to us. That's just not going to happen with our current LLMs, they aren't going to scale up to that extend. There are already articles suggesting that we're almost at the peak of what they can do, with the resources we allocate.
Usefulness relies on understanding, even if that means understanding what is possible to ask for. What is being generated right now is really just a translation from one inscrutable phenomenon into another. In neither case do we have power over the data, because we don’t know what it means, truly where it comes from, and even the beginning of the limits of what the discovery allows. We’ve traded one black box for another, and still need to unpack it.
> We’ve traded one black box for another, and still need to unpack it.
The human mind has been a black box, and largely still is.
So we traded one black box for another, in more than one way. The new black boxes are going to be able understand and keep up with their own work, much more quickly than we will.
The "alien" picnics, in this case, are a prelude to settlement and expansion, and we have only met the children.
Your point on understanding mostly holds (although our own mind's inscrutability but usefulness is a strong counter-case), but it will increasingly work against us.
We do have a "working hypothesis" how the mind works though, what psychology calls "theory of mind" or "folk psychology". While this is completely divorced from the level of neurons or brain regions and so might tell little of how the mind "really" works, it's detailed and useful enough that we can judge how other people would react, guess what attitudes, feelings and motivations drive them and in general can have a useful level of interaction.
The same is true for traditional computer programs. The standard argument against valuing understanding is "well, you don't understand the assembly, processor microcode and the electrical signaling when your program gets executed either, why do you care about understanding the high-level code then?"
I'd counter this by saying that high-level languages have (mostly!) self-contained abstractions that allow you to "understand" a program deep enough to make accurate predictions of its runtime behavior, answer "why" questions when errors happening and safely make changes to it - without having to understand all the stuff at lower levels.
In contrast, with AI, I see the risk that we lose understanding of both the low- and the high-level structure of things. Then we really end up in "unpredictable alien artifact" territory.
Usefulness doesn’t always rely on understanding. And it’s not a new thing in mathematics to have an answer without fully understanding it. We’re talking about a change in the rate of that happening, not a new phenomenon in itself.
I think Ted Chiang's story "The Evolution of Human Science" may be a significantly better metaphor here. What AI is dropping is (allegedly) just answers, not mysteries. In both cases, we don't really understand it, but in "Picnic", most of the things left behind aren't even useful, merely mysterious or more often dangerous. In "Evolution", answers (and only answers) and technologies arrive pre-packaged, with zero understanding.
- https://en.wikipedia.org/wiki/The_Evolution_of_Human_Science
- https://gwern.net/doc/fiction/science-fiction/2000-chiang.pd...
> All this new knowledge is arriving at a pace much greater than the field’s ability to process it
That’s the whole issue: it’s not knowledge, it’s data. AI can not provide an account for the proofs, or be questioned about them, or organize a talk exposing the idea because there’s no understanding behind any of it.
This is what Tao has been saying; that the results, in the strictest sense, were never the goal. It is ultimately the understanding (that previously was required to form a proof) that is valuable.
> It is ultimately the understanding
But understanding to what end? I'm if it's for the pure aesthetic experience of engaging in the beauty of math than that's fine, but that's typically not the view from my experience.
The goal of understanding is to advance the field, since historically human understanding was the only way to do this.
For the vast majority of human history, the world we experienced was a strange and unintelligible space. Humans survived at the by the grace of forces beyond their control. For a brief window of time human existed in a world where they increasingly saw a relationship between their understanding of the world and there ability to steer their environment towards a desired outcome.
I believe now we're seeing that that was a historical anomaly. The world is once again entering a state where it becomes increasingly unintelligible to us. Nick Land already famously pointed out, many decades ago, that the global financial system is already a form of "artificial intelligence" we don't really control. We're seeing that spread wider and wider.
The world making sense peaked around 1960s and has continued to breakdown ever since.
Why wouldn’t the AI be able to answer questions about the proofs or organize a talk? Those are pretty much the things it excels at, exploring and summarizing previous knowledge. The problem is the volume of questions and data a human needs to step through to arrive at the same understanding.
Apparently, the knowledge isn't in the data set, and the explanations are gobbledygook. They certainly summarize well, but it seems that if the "understanding" isn't already baked in, then expanding the source description doesn't really clarify it. Maybe in the future?
What does apparently here mean? Is there an account of an expert sharing their attempts to get a model to explain the results and key ideas to them (for a topic they're familiar with) in an interactive session, and having it fail?
For somebody deeply familiar with a topic and all the nuance around the problem and approaches that didn't work, I'd be quite surprised if they couldn't cajole it into explaining what finally did work in terms they understand. Unless it turned out that these proofs were just well beyond anything humans were anywhere near being able to figure out, in which case, no great loss that the AI figured it out because we were nowhere near it.
I guess the issue with that is that the understanding is important to mathematicians. The majority of people outside mathematics don’t understand it whether it’s humans doing the work or AI. If AI can generate useful results more quickly, then to the general public it is more valuable.
Mathematical solutions aren’t valuable to the general public at all, other than the passing fancy that comes from reading headlines and news articles.
Mathematics is only useful to the extent that we can do something with it, and I would guess that still requires human mathematicians to understand it.
Does that mean AI is not doing something useful? If it provides a proof, it gives direction to human mathematicians who can then develop an understanding, give talks, be questioned about the topic, and teach. Is that not a useful service?
It might well be useful if the RIGHT proofs are evaluated and proven. Figuring out what will be useful has always been a real challenge in math. Finding certain symmetries and mappings between problems in different domains, might well be useful even if they aren't well understood? That has always been another part of the work of mathematicians. Of course there are those who just work on the beauty of things, and few in the 80s would have thought that prime factoring number theory or elliptic curves would be the basis for modern communications and banking.
That said, getting a point solution and not knowing the path to get there, is certainly much less useful, and generalization is much harder. Again, more work for mathematicians, but maybe without the easy "I solved it" proof metric for rewards. Goodhart's law strikes again.
Less useful, but still useful. If robot Ramanujan gives us answers with no mortal-understandable justification, but it's always right, that still gives decades or centuries of topics to try to build theory around, with the motivational benefit that you always know that what you're working on can be solved. Simply treat it as an oracle that dispenses magical opaque proof certificates.
I don't think that's the argument. In my opinion this is like paying to unlock characters, à la EA. Who's going to feel a sense of accomplishment by just trying to interpret what an LLM spat out? Sure,maybe the results will be used by someone in some field, the same way a product manager uses what the development team developed. Is the PM going to give talks about it? Are they going to expand on those developments and come up with something new?
> Who's going to feel a sense of accomplishment by just trying to interpret what an LLM spat out?
I think you have a point, but this is the reality we have to face. The genie is out of the bottle. Even if OpenAI stops publishing AI-generated proofs, it's only a matter of months before tools with similar capabilities are in everyone's hands.
You're right.
Hopefully, this genie will grant us a few wishes
Fair, “answers” is probably more precise than “knowledge.” That gap between answers and understanding is the whole point of the post, so I think we mostly agree.
I like this metaphor better: the mathematics community is like a group of ancient astronomers that are unexpectedly getting the output from a modern telescope. "Why even look at the sky anymore," some say. "And we don't even own the telescope!"
Meanwhile, the Rubin observatory is creating an archive that grows by 20 terabytes each night, and nobody really worries about this. Dealing with that amount of data is definitely a problem, but also a nice problem to have, provided that there's a scientific community that can deal with it. Mathematics has unexpectedly become Big Science and there needs to be a better relationship between the funders of the scientific instruments and the scientists themselves.
It's a bit confusing that the output from the scientific instrument looks like badly-written academic papers. Maybe it should be treated more like raw data? The format is not ideal, but historians (for example) often have to deal with far worse.
I think there is a difference here; the modern telescope doesn't output answers, claims or theorems - just data that still has to be interpreted to be even remotely useful.
It might superficially look like answers, but apparently someone still needs to interpret the results? Maybe it would be better to treat it like data.
I like that metaphor. But what it calls to my mind is an "alien artifact" which was found by a road, and "how did these refined metals get placed inside of a rock, in such an unnatural formation?" and there was much debate about this artifact. Years later it was determined that it was in fact a spark plug which had been remineralized by the effluent from a mine upstream: an unusual process acting on a discarded object, and people who want to believe. Nobody thought that the remineralization process could take less than hundreds of years, which ruled out "spark plug".
I highly recommend also reading the “Inhabited Island” as well as “Hard to be a God” by these authors. These books are brilliant as well.
I tried "Monday Begins on Saturday" and had a pretty hard time getting through it. But I have to give it another chance - and check out the ones you mentioned.
Monday… is a very Russian book, heavily based on memes and folklore. You will have very hard time understanding it. I can’t recommend you to try it again.
The 2 books I’ve mentioned are very international on the contrary.
BTW, “The kid from hell” is also really good.
I am Russian, read it in the original. But I'm scarred by being forced to read meandering classics from a very young age, so I may just have a more visceral negative reaction to it that I have to get over.
Unrelated to the article: Roadside Picnic inspired a movie called Stalker, which in turn inspired the S.T.A.L.K.E.R. game series.
Also the game Pacific Drive.
Both of those are listed on the image of the book cover in the article.
Yep it's the OG literary version. Very much worth a read.
I wasn't a big fan of the book, tbh. I love the movie though, it's one of my favorites.
The movie is almost unrelated to the book. I really like the book though, even being as depressing as it is.
It also reminds me, in a very similar way, of William Gibson's "Hinterlands". Some people are just hell bent on flying into the Alien Coordinates That Make You Crazy or Kill You but Sometimes Give You Cool Stuff you Don't Understand, from William Gibson's short story Don't Fly Into the...
I can only recommend Strougastky brothers books’, I enjoyed ‘Hard to Be a God’ and wish to read more from them.
That is why I find the prospect of spoiling the plot without warning to be bad taste and didn’t finish the article.
No real spoilers in the article though. Just one of the ideas revealed very early in the book.
A new analogy for the HN audience that is susceptible to reinforcement learning applied to humans (daily)!
This rests on the (completely false) premise that AI is doing anything other than simply parroting our human knowledge back at us
It has no capacity to think, argue, demonstrate, or certainly do anything even remotely close to “intelligence”
It doesn't assume, it just observes. It has the capacity to use computers better than us, more persistently, in a more scalable way, to solve problems we haven't solved before like Navier-Stokes and many more to come. You can argue all you want about whether it's intelligent or not and what it means, but the solution is now here for us to process, understand, and integrate forward into our model of the world.
Completely false
I really dislike this analogy. Trash and apple cores versus potential, objective discoveries?
I'm not saying that OpenAI's results are good or bad. But I can say that at least humans could try to understand them and get not only the discoveries themselves but some techniques that could help other progress down the road.
It's fair to ask whether we'll be better off as a result of AI Math. But many people are failing to see any sort of positive outcome and I think this is close minded and borderline irrational.
I don't blame Mathematicians for trying to protect their careers. Anyone would to the same in their place. But the rest of society should take a hard look at what's happening and, as the financiers of mathematical research, make an independent assessment of whether humanity needs "old school" mathematical research at all versus a model with a lot more AI.
That's probably not the point the author is trying to make. In the book humans are trying to make sense of the artefacts lefts by the aliens, but are only able to do so slowly. The gap between human understanding and the aliens who built the artefacts are just to large.
The book never tells us that this is actual trash, it's just a theory. If the theory is true however, the aliens might also not understand the damage they've done, it's just a bit of littering to them. Or they don't care, they just destroyed an ant hill, so what.
The author of the article might have a bit more faith in the current generation of AI than I do, I don't believe that LLMs are really capable of creating that big a gap. The discoveries they make are all based on human knowledge and understanding, so we can comprehend the foundations on which they are built. Now if models can train and iterate on their own, to a much greater extend, then maybe, some day, in the far future their discoveries could leap frog humans to the same degree as the aliens in "A Roadside Picnic" and not understand that their output is damaging to us. That's just not going to happen with our current LLMs, they aren't going to scale up to that extend. There are already articles suggesting that we're almost at the peak of what they can do, with the resources we allocate.