One thing about human mathematicians is that they only publish positive results. Professors etc might have file drawers full of "negative results", but the incentives and bandwidth of human mathematicians makes publishing these useful results impossible.
In general though, LLMs do not have the same limitations and incentives as human mathematicians, and the next year's tsunami of change will make this abundantly. clear.
It's also "out-brute forcing them." It just never gets tired. If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc. This thing just does not ever get tired or discouraged or care; it's just onto the next thing until something ends up working.
loaned from German, where it's originally a way to say buttocks, literally "sitting flesh". If you have more Sitzfleisch you can sit for longer. Both in the literal sense (a bigger butt makes sitting more comfortable) and in the figurative sense (having the mental ability to sit for longer, get more desk work done)
I think there is an IT bit of humor from about 1 decade+ back where the people who's proposals won out in meeting were the ones that could keep from needing to go to the bathroom longer.
There's also a less flattering reading of the word, where Sitzfleisch means having a "flat ass" (from sitting too much, e. g. Sitzfleischparade describing a group of flat-arsed people, or something like Sitzfleischmaxxer, and so on).
Wow, what a great comparison. LLMs are great at reasoning but absolute dogshit at simple arithmetic. If there's a raw calculation involved I always tell it to use python to add it all up.
String theory is a great example of a dead end kept alive by ego and sunk cost fallacy. An AI would have declared it dead and moved on 10 years earlier.
When people make these comments about string theory, are they at the forefront of quantum physics theory and have spent years on modern string theory? Or did they just watch a YouTube video and then keep parroting this?
You could reductio ad-absurdum this logic quite broadly. We all form beliefs about the world and its aspects, and almost always through fallible sources. Should one be confident about ones of questionable provenance? I say no. But forming beliefs from the information we have is a useful skill.
This isn't a great argument. Researchers who do not think string theory is good are not going to spend years on it. There are plenty of experts that dismiss string theory. I have no skin in the game, and don't care either way, fwiw.
No, not morons, but people who have built a career on string theory. At this point, even if they regret their decisions, it’s too late to turn back now.
This is a pretty silly belief to hold about science. Plenty of strong, well held standards of the modern era were considered ridiculous fringe beliefs a century ago.
The key here is that it’s depending on the human inability to connect the sum of relevant knowledge, but said knowledge comes from humans.
Theres going to be this field day of low-hanging fruit that ML can round up, but after that I suspect it will be in fits and starts as a “connection maker” rather than some proof producer.
I think we're underestimating just how much low hanging fruit there is. I've been trying to apply this LLM research process to physics (QM and solid state) and there is so much missing in Physlib and the rest of the Lean ecosystem that most of my work has been trying to formalize the theories and validating them against the specification problem (and mostly failing badly).
It's not only going to be "connection maker". If and when robotics advance to a point where the LLMs are embodied, they can run experiments in the physical world and find new knowledge.
Robots in labs already exist, but mercifully they're not hooked up to anything as unpredictable as an LLM. Robots tend to work best as specialists doing high-throughput, extremely repetitive tasks which nonetheless require a degree of precision. Giving a robot a "human" body makes very little sense if we're talking about the needs and productivity of a non-human; humanoid robots are marketing for humans.
Humanoid robots are obviously more than marketing. The entirety of human civilization is human shaped. Making robots that are human shaped is easier and more efficient than redesigning and rebuilding everything that exists.
> The entirety of human civilization is human shaped.
That's the marketing pitch.
A plumbing robot doesn't need to be humanoid, an octopus shape may well be better for all the awkward corners. A robot police officer could be the municipality itself for sensory nodes (essentially the sales pitch of Flock etc.), plus some drones or robot dogs to perform arrests*.
The robot vacuum cleaners and lawnmowers we already have are nothing like a human. A robot taxi driver can be just the car. Robot dogs are already used for maintenance and security sweeps.
If you've got wheelchair access, you've got wheeled robot access. If you've got guide dog access, you've got access for Boston Dynamics' Spot.
* this may be a bad idea with current robotics, but I aver it's not improved by making those robotics humanoid.
Outside of sci-fi, marketing proposals, and niches like "Elder care in Japan" there are very few humanoid robots. By contrast non-humanoid robots have been mass produced and used in industry for decades. Arms. Carts. Trollies.
No people. If you want something with fine motor control and dexterity, it's easier to make that the robot and then have another robot bring the workpiece to the arm than it is to build a single robot that can walk around and do it. There are compromises in human features because we're generalists.
A new technology being able to do something better than humans does not mean it’s intelligent though. A calculation program is not intelligent just because it can remember more digits than me, work more than me
But the difference really does matter and is not just a case of "whittling down" what intelligence really is.
We have known for a very long time that computers and machines are much faster than humans, more accurate, are scalable in certain ways that humans aren't, and they don't tire. I think most people who are not in the "AI cult" would agree that LLMs and modern generative AI are really just an extension of those faster/more accurate/more scalable and never tiring traits. But there does seem to be (and I'm sure folks much smarter than I have quantified this or described it better than I can) a fundamental difference in how humans think, especially as it applies to what true "understanding" really entails, and for the ability to think up truly novel and unique things that are not just a rejiggering/recombination of training data. I believe those skills really are at the heart of human cognition, and as impressive as LLMs are in replicating what this looks like, there are plenty of "LLM failure modes" where it's clear that LLMs lack a true understanding of concepts or the ability to generate useful, completely novel ideas.
There's definitely a lot missing from the current state of the art in machine learning that all brains manage to beat, and we can observe this just because an animal that needs as many examples as an AI to learn motor functions would starve to death before learning to eat.
However I can only guess that this is important, I'm not absolutely certain. They're at risk of being an economic disruptor just by being extremely stupid (by how much they need to study) faster than us to the same ratio we jog faster than continental drift.
Because, what's really happening is we're saying "Oh these things are what defines intelligence" then implementing them and /discovering/ "oh wait, there's more to this than we knew".
We've known, for decades, for example that an IQ test is not a measure of Intelligence, even though people still refer to it as though it is. A computer passing an IQ test, therefore, would have been thought of as possessing intelligence way back when, but would not now.
> If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc.
Your timelines are a bit unambitious. There's nobody expecting to make significant progress with a week of work.
If a thousand monkeys typed at a character per second, on a keyboard with the 23 relevant characters, it would take roughly 10^136 years for them to come up with this counterexample. Though, to be fair to monkey scenario, there's a large family of them known now, so it's not quite this bad: suppose there are a trillion permutations and similar examples that fit in this string. Then we are down to 10^124 years.
If LLMs are monkeys, somehow trained LLM weights allow them to model and prune massive numbers of universes in parallel.
There are plenty of high value endeavors where being a superhuman knowledge remixer is right on target. But even capturing all of the knowledge is proving elusive.
I use coding agents. I think they're pretty good overall. They save me a lot of tedious coding. For example I probably wouldn't spend the time to implement native splash screens for all the build targets of a Flutter app, but I'll have the coding agent do it.
Nevertheless, for all the time that we've had coding agents, it's still trivially easy to find the jagged edges of their training. For example, Gemini evidently doesn't know if the Xcode part of a Flutter tool chain is misconfigured. That's not exactly a Millennium Prize problem. But it is shaped wrong for a training set for a coding agent.
While TFA itself makes sense I disagree with the title and the conclusion. I would not consider referencing working memory during thinking as “remembering” but as a part of thinking itself. Working memory is the RAM to the much larger but higher latency indexed database that is our long-term memory. As such I would say AI is out-thinking us, even if in a brute force sort of way.
I think where you could say it is out-remembering us is when it can contemplate the vast universe of patterns, gleaned from essentially all human disciplines, encoded in its weights, that may let it draw connections that a human could not, unless they just happen to be familiar with multiple disciplines.
Which is why I think the analogy with Von Neumann / Einstein is also a bit off. From TFA it seems Von Neumann was more akin to what AI does, than Einstein. I don’t get the impression that it was Einstein’s memory but his ability to look at things from a radically different perspective. So far I don’t know that we can categorically say that LLMs can or cannot do that.
Does it matter? It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
We don’t trillions of dollars in LLM investment to build things mathematicians don’t understand. We already have plenty of those, even from ancient times.
As to your second point, Terry Tao already has an answer [1]: the proof isn’t the contribution, shared understanding is. This issue was already raised back when the four-colour theorem was proved. Machine proving and machine proof checking are useful tools but they don’t mean anything without the interpretative work and the communication necessary to build shared understanding.
If humans have nothing to contribute then shared understanding is a pointless endeavor. It makes sense now in the "centaur" period where human + AI > AI alone, but when AI mathematicians are both more rigorous and more elegant, then taking the time dumbing down their proofs to a human level of understanding is like requiring that we ensure all our current proofs be understandable by a monkey.
Why not? We build cranes to hoist weights construction workers can't lift. We build electron microscopes to measure things physicists can't see.
Why is it so hard to imagine we can build tools to think thoughts we can't comprehend?
If there's commercial value, I think it's inevitable. We don't fund mathematicians because it's cute when they understand a problem, but because their work tends to have applications with commercial value. The value can be captured without understanding the details.
Can you give an example of an incomprehensible piece of writing (any writing, never mind a proof) that has commercial value commensurate with the costs involved here?
> Can you give an example of an incomprehensible piece of writing (any writing, never mind a proof) that has commercial value commensurate with the costs involved here?
It depends on what exactly you mean by "commercial value commensurate with the costs involved" but I'd volunteer the 3G/4G/5G specifications and the other documentation required to implement the mobile network protocols. 5G is currently sitting at over 50,000 pages and it's one of the reasons Qualcomm/Broadcom/Apple are the only ones who can realistically make a mobile radio.
I don't think there is a single human to whom more than a few thousand pages would be comprehensible at a time except for the occasional genius.
Yeah, how many people do you think understand the Linux kernel in full? What percentage of the people using it to great commercial effect can understand it?
How's your understanding of Schroedingers "An Undulatory Theory of the Mechanics of Atoms and Molecules"? You seem to be using the results of it as applied to semiconductor engineering just fine. And, I promise you, most semiconductor engineers haven't read it in full, they just accepted the results as passed on by several layers of teacher.
I have a paper on routing algorithms, which I have attempted to read to my cat. I don't think my cat retained much, but they seem to be enjoying the cat food that got delivered using the results.
I'd suggest that we're going to be a lot closer to the cat than the author of the paper when AI takes off.
There are plenty of artifacts that, if not impossible for humans to understand, then at least no human has ever completely understood. To start with, the universe as a whole. Despite that, we are able to choose legible pieces of it to model and perform useful actions from.
This will shift your argument--that doesn't count! etc., to the point where it's by construction unsatisfiable and vacuous. And it doesn't matter: an LLM might e.g. break some cryptographic algorithm in a way utterly unintelligible to humans, but the fact that it works would be sufficient on its own to make all of us choose to abandon that algorithm and choose different ones.
You asked for examples, for a technology that we're still building. Maybe you can see the issue with that?
Anyways, people benefitted greatly from Newton's laws of gravity, even though we still don't have a quantum-compatible set of laws for it. The laws of gravity are still incomprehensible for people, but the approximation that we've observed is still immensely valuable.
No, I gave you a lot more leeway than that. Take any utterly incomprehensible piece of writing from the entire history of civilization and demonstrate its value.
You keep falling back on "incomprehensible for some people" but that wasn't the claim. It was about a text which is incomprehensible in principle; that is, utterly impossible for any human to ever understand.
Can you give me an example single piece of writing that the author didn't claim they understood? I don't think that any utterly incomprehensible writing exists.
If you want examples where nobody but the author understands it, examples are a dime a dozen.
So you concede the point then. The age of humans comprehending things is not coming to an end. And therein lies the rub. When it comes to intellectual labour:
Understanding == Value
If a mathematician produces something incomprehensible then it has no value. It's meaningless. Indistinguishable from random noise.
An AI which produces incomprehensible text is producing no value. We didn't need to spend trillions of dollars on LLMs to figure that out. Markov chains can do that job perfectly well.
> how many people do you think understand the Linux kernel in full?
"Not many people understand some things fully" is so massively different from "the human mind is incapable of understanding some things that AI will understand for us"
Yeah. We're only starting the journey of building tools better at thinking than the human mind, so expecting me to have examples of things it produces is a little hard, don't you think?
The best I can do is things that are incomprehensible to nearly everyone, but still provide value. There's a small leap of imagination to consider an author that understands it and can show others how to leverage results without understanding be mechanical rather than biological.
I know right? And there’s only a market for maybe 5 computers in the whole world.
Paul Krugman (1998): predicted the internet’s economic impact would be no greater than the fax machine’s.
The 1876 Western Union memo dismissing the telephone as having too many shortcomings, and the banker telling Horace Rackham not to invest in Ford because the automobile was a novelty.
The entire point of writing proofs is for advancing human understanding. A giant dump of symbols that passes the lean compiler is meaningless besides human beings understanding it.
> The entire point of writing proofs is for advancing human understanding.
Proofs also enable AIs to direct search and generate knowledge. Verifiability is immensely useful for keeping AI grounded.
One might imagine AI generating enormous numbers of hypotheses and then trying to prove or disprove them, and then mine that data for new abstractions and heuristics.
Sometimes the purpose of the proof is simply to demonstrate that some construct is a safe assumption for other more interesting work-- and could still serve that purpose even if it was entirely a black box.
Is it the AI's fault we can't understand? If the GUT is beyond human comprehension does it matter less? We don't apply this reasoning to other animals or even to less capable humans. Besides, the robots may want to ponder maths for their pleasure.
> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
That sentiment makes me cringe. If you understand how LLMs work, you'd know it'll never be possible without a fundamental change in how these work.
We're also supposed to be reaching that point, somehow, without the LLMs ever being intelligent (in the dictionary definition sense, not the "high reasoning model" marketing sense).
Based on observations, the ones who are fooled by the supposed emergent properties, are just that, fools. Any sufficiently unintelligent agent will perceive transformer based LLM text predictors as possessing high intelligence.
First of all, the argument isn't that LLMs (with I assume some automation) cannot be used in searching a problem space. I'm assuming this is what you're referring to, in terms of contributions?
That's the part where LLMs are used as tools. Which there are plenty of places where they are useful.
Also, do you know what turning completeness is? Why are you bringing that up here?
The crowd that AI psychosis has brought to HN is interesting. But not in the "I'd love to learn more" kind
> That's something AI companies would really want you to believe.
Why would I care what they want me to believe?
Intuitively it would make sense that you can put math ability on a chart with a value for “general public” “smart high schooler” “smart undergrad” “smart PhD/ professional”. And you could place frontier AI somewhere on that chart over time from GPT 2 to now and see the trend.
Then you’d have to consider that either you believe there is a fundamental limit that is below peak human mathematician level or there’s not.
They can say random stuff with the goal of increasing their shareholder value. Things they spit out do not have to be true. It is not easy to verify things they say, therefore, everything they say should be taken with a huge grain of salt.
Consider Enron and Amazon at the turn of the millennium. They were both telling you what the future would look like. The right action would’ve been to just ignore what they are saying and try and get data and reason about the world. It didn’t really matter that both Bezos and Jeff Skilling wanted you to believe various things - one was right and one was a scammer.
So that’s what I’m doing here. For what it’s worth I find a lot of the AI people’s worldview very consistent. They believed AI would be the most important technology of our life times and committed their work to it. Some of these same people are total liars so yeah I won’t really hang onto their every word.
that does not make it not true, nor does it make those companies or their products not dangerous. I like looking at videos of animals that tear other animals apart and eat them; lion cubs are super cute; but that does not mean I want to be thrown into a cage with a model of a lion that has not been programmed to be disinterested when it is sated. AIs appear never sated; humans using or making AI wanting money, even less so. I suspect the AIs will understand the cost long before the humans will, not that anyone making money would care.
> It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
I agree.
> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
I don't know if I see this being true for quite a while, if ever.
> It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
> I agree.
There's an infinite space of possible statements and proofs. The only thing that makes certain proofs significant is that human mathematicians consider them significant; if AI came up with a proof of some statement that no humans could understand then no humans would bother investing further resources in building upon it, for the same reason we don't waste computational resources iterating over the infinite space of true statements in first-order logic.
It also creates and updates models, uses those models to make predictions, guides the stochastic generation by comparing the output to those models and reworks them in real time.
It also creates and updates models, uses those models to make predictions, and guides the stochastic generation by comparing the output to those models and reworks them in real time.
In limited experimentation: AI will certainly make statements that are extremely intricate and hard to understand, in part because they're overcomplicated and in part because they use a bunch of unnecessary terminology.
This is not to say that a human couldn't understand a streamlined version or that the AI would not be better if it made more streamlined statements to begin with.
(I am not saying that everything mathematical that an AI produces is in any sense trivial.)
It’s possible, but there’s a difference between vastness and difficulty.
Humans can’t compete with AIs on vastness of material they are familiar with, or the depth of effort they are willing and able to throw at a problem.
But scale isn’t the only aspect of difficult scientific endeavours. There’s also theory. And advancements sometimes come through hard graft of knotting together many things. And sometimes they come through the revelation of a deeper truth, or a new framework, a fundamental insight.
AI might help us reach the next level. But that doesn’t mean we won’t understand anything. It could be we have periods of vast intricacy we cannot follow, punctuated by profound elegance we (or at least experts) relatively easily can. And then the scaffolding we needed to get there falls away.
I don't think that's true. Human intelligence is limited, and our brains are inefficient machines.
The tools we built to replace muscles have mostly obsoleted raw strength for tasks like excavating earth.
There's no reason to think we can't do the same for brains. And then we'll never need to think for a living again. Some people may want to do it as a commercially insignificant hobby, of course, the way people lift and compete in strongman competitions today.
We'll have AI taking care of our needs, the way a good mother takes care of their children.
A good mother doesn't raise children to be dependent upon her for all their needs.
For this to actually work in a way that benefits our species, humans will need to become something else/next through their interaction with the technology.
You could be right, but you're making a lot of assumptions about how complexity, scientific understanding, and explanations scale. One of the features of a good scientific discovery is that it often simplifies and compresses things that were previously a bunch of scattered facts. Also, as AI systems improve they'll get better not only at making scientific discoveries, but also at producing understandable explanations.
>produce proofs far more intricate than humans can understand
Math is not magic, a proof is just a series of applications of a set of rules on some axioms. A mathematician could understand any proof given enough time to study it; the only way for AI to make proofs that a human couldn't understand is by making really, really long proofs.
Why would you want something you don't comprehend? How can you be sure it empowers you?
I think perfect rationality doesn't exist, because it is rational to reject something that you don't understand. So rationality of a given physical system will always be bounded.
Yes. That's how LLMs do programming, mostly. It's also why LLMs don't need abstractions or parsimony as much as humans. They can work on something complicated without simplifying it first.
This has major implications that haven't been fully realized yet. On the math side, there are long machine generated proofs. On the code side, there are high volumes of code with similar code not being folded into functions.
This is the exact opposite of what I’ve been dealing with for awhile. LLMs absolute cannot work on something without an understanding unless they can outsource the understanding to a verifier. If you’ve got an easy to check function to measure progress then “keep going” is all the prompt you need. But if you need it to figure out “I pushed the up button and it moved up and left” then it’ll find the same bug five ways without realizing it’s just one bug in the underlying math.
For greenfield projects LLMs don't need abstractions, but as the project gets more complex, the right abstractions save a pot on input tokens (less code to read) and reasoning tokens (less work to do to figure out the code), so they free the context window for higher purposes
Also I suspect that, apart from that, the results on smaller, cleaner codebases are better. LLMs degrade when following more than N instructions (where N depends on the model) even if the context window is not full yet; I suspect they also degrade when code has too many unnecessary concepts and details
LLMs use abstractions a ton in code though: standard library functions, popular libraries, etc. They just dont always make their own abstractions. At least not particularly good ones. LLMs work really well when they have well abstracted pieces to put together.
I've been working on generating a large code base for the last couple of weeks. Finally got around to generating a sort of code-duplication report and have spent the last week just having it de-duplicating logic that had been strewn all over the place (eg 11 different functions all doing date math to add x days to a date). dozens of items that had each been similar functions duplicated numerous times. crazy. (opus-5-utracode)
No, the problem is they're still really dumb, and lack the ability to make logical connections that are obvious to us. "should I walk or drive to the carwash" being a very recent example of the larger problem.
This means some AI proofs might be impossible to comprehend by humans, right?
I guess AI still lacks human intuition for many concepts, but AI might beat humans in narrow areas, such as discrete math and combinatorics.
Outside of math you can basically take the entire corpus of research papers on any topic and have the AI read all of it and provide an analysis cross referencing everything all at once. This applies to everyone and everything.
I love the term "Out-Remembering"! I have been trying to find a way to communicate that "intelligence", "creativity" and so on might be misleading about the true nature of LLMs, and they would better be described as genious "reproducers" as in, they are very capable at reproducing what they have already seen - and they are a bit less capable, but for many use cases still good enough, at reproducing a mix of concepts seen previously.
This also nudges into how to use it best: By knowing where the "piles" of if training data are (i.e. when it comes to a CLI in rust, I just briefly describe the use cases, and I have a very high confidence the code will work exactly as intended by me since there will be a multitude of examples in the training data), one can predict where the LLM is likely to go wrong an prompt/guard accordingly. This skill grows with domain expertise, and is one of the many reasons LLMs can be (and probably should be) used to outsource busy work, but never understanding and learning. ("never" is a not meant literaly of course - I for one am glad that I do not have to wrap my head around CSS and other frontend topics and go straight to the topics that interest me most)
"Out-Remembering" captures that perfectly, I feel. Also goes nice along with "asking it leading questions" as we know how to do in real live; if you want a person (LLM) to confess (produce output tokens) something, sometimes you do that by leading the interogation (chat, context) to where you think the truth lies.
What I'm looking forward to amidst all the negativity, fear, and loathing is for some 20something mathematician to outdo both humanity and machines by leaning hard into centauring to expand the frontiers of mathematics. Pretty much what I think the future will play out to be as well, but I don't think people are ready for that yet.
This is why education used to start with rote memorization.
Functional intelligence isn't abstract, it is based on useful information you can quickly recall.
Mathematics is typically concerned with "proofs" [1], which similarly to code, often allow for strict validation. Thanks to reinforcement learning techniques, it is now possible to train LLMs to perform very well on code generation, and mathematical proof generation.
Law and medicine are fundamentally harder fields to obtain decent training data for, and LLM results are therefore expected to be less powerful. Also, making mistakes in these fields is costly, but perhaps you were alluding to that already.
Mathematician here. There is a lot of recent work on the Lean project -- when a proof can be translated into Lean code, then it can be strictly and formally validated.
But otherwise, mathematical proofs are read and written by humans, and at the end of the day the relevant standard of proof is what other mathematicians will accept.
Occasionally, mathematicians don't agree. For a prominent example, you can read about Shinichi Mochizuki's claimed proof of the so-called ABC Conjecture:
> Law and medicine are fundamentally harder fields to obtain decent training data for
I think this bubble has given a lot of people software brain and are trying to apply it to fields it is wholly inappropriate for, though. Law is about argumentation and rhetoric. It is about providing a persuasive argument. This is how it is taught. The actual legal code is a way to formalize parts of it, but increasingly I see people angrily insisting that the only thing that matters is the text.
As you might imagine, I find textualism a load of applesauce, but I don’t think the vast majority of people making this argument even understand textualism as jurisprudence. It seems to stem from Crypto bros and the whole “code is law” argument which is just codswallop.
Well also you can't just throw AI slop as doctors or lawyers advice and fail multiple times until you find the right answer. With code and math you can have failures 1000 times for every success and still get rewarded.
A neighbor of mine whose husband is a lawyer said it's already part of his regular workflows. OpenAI also already have HIPAA compliant offerings targeting healthcare uses, etc. Of course they already do these things.
They can rely on compilers, solvers, theorem provers to validate the generated softwares and maths. That’s what makes it possible to iterate quickly in a loop and self correct. You cannot do that in soft industries like legal and medicine
That is not the point I was making. I am not talking about validating software or maths. It can generate stuff that is valid, but bad and incomprehensible.
What I’m saying is that AI labs are talking so much about software and maths because we already have tools that can say « it’s all good ». That makes it possible and worth it for them to spend 1 week of compute on a problem until the validator passes, then publish marketing pieces. You cannot do the same in medicine or laws (modulo some niche areas)
It is obvious that super intelligence comes from more working memory.
It is the scary thing actually. Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
We can decompose and write things but only up to a point. when Ai can have a working memory that spans hundreds of books, we are necessarily going to have to trust the system.
This is why we have hierarchies of abstraction. Pretty much every field of mathematics relies on constructing notations, models, and other tools to simplify things in a way that is verifiable. LLMs rely on the same basic technique, they can just pull from a wide variety of these abstractions at once. So far we've been able to understand their proofs just fine. Computer-assisted proofs in the past that relied on brute-force is where we have run into trouble. We cannot reason about millions of possibilities at once, and we had to trust that the computer program that analyzed them was correct, which is a really hard problem and leaves humans fairly unsatisfied. I think we are actually progressing in terms of understandability in computerized proofs.
>Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
That doesn't follow. We could still understand it just by studying it and committing it all to long-term memory, it just takes longer. And there's a hard cap on the working memory of LLMs, due to the quadratic scaling cost of the full attention layers that have proved unescapable for all SOTA LLMs.
Aka - it’s a stochastic parrot with a good memory, for anyone still struggling to understand this. It should be obvious, imo, but some people seem to have trouble with the concept.
I don’t understand why this is such big news. OpenAI has essentially made a bunch of marketing copy by gussying up an algorithm being given a near unlimited budget to stochastically permute through its lossy memory.
I think the real scandal is that we are almost 3-4 years into this (I think the release of GPT 3.5 is a good marker of when this public frenzy started) and all we’ve seen is OpenAI and the other major AI frontier companies constantly retracting their preposterous claims every time. We appear to have reach a local maxima in that it has some value in places that tend to be a little easier to scope and limit (computer programming, mathematical proofs). So, given the actual useful economic value this has provided, does this justify the investments? I think we are approaching 1 trillion in CapEx for AI [0]. For context, I believe the annual GDP of Norway is $600 billion.
That’s how HN works, I had that multiple times over the years with my own submissions. Sometimes it gets picked up quickly, sometimes not. A post can also down rank very, very fast. It depends a lot on the level of engagement and the type of engagement
100%. Context is big for AI, but it's nothing compared to everything a human can learn. If you efficiently represent everything in context, it may be many papers, but if AI is actively working through proofs, it will quickly fill up. They're no denying AI is making strides, but pinning it to memory is an oversimplification.
I don't have to open the article to be confident it's not worth reading. Anybody knowledgeable in the field should be familiar with AI writing tells and the message they send. It only takes a few seconds thought to transform the title into something like "AI beats mathematicians by out-remembering, not out-thinking." Regardless of whether the article is slop or not, I expect any competent writer to avoid slop phrasing in their titles. To do otherwise signals laziness.
Most mathematicians are quite simple creatures. I can do basic math, some derivations, but my bright days of solving differential equations are far gone!
Computers are simply better at math now, like in chess or go!
One thing about human mathematicians is that they only publish positive results. Professors etc might have file drawers full of "negative results", but the incentives and bandwidth of human mathematicians makes publishing these useful results impossible.
But AI agents have no such limitations and can publish and re-use negative traces easily. There have been some recent projects (https://www.theoremdb.org) aimed at exploiting this fact. https://news.ycombinator.com/item?id=49227505
In general though, LLMs do not have the same limitations and incentives as human mathematicians, and the next year's tsunami of change will make this abundantly. clear.
What is next years tsunami of change? Asics?
It's also "out-brute forcing them." It just never gets tired. If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc. This thing just does not ever get tired or discouraged or care; it's just onto the next thing until something ends up working.
> It's also "out-brute forcing them."
That is also approximately what people have always done to succeed.
AI has sitzfleisch
https://en.wikipedia.org/wiki/J._Robert_Oppenheimer#:~:text=...
My browser doesn't do the fancy URL text selection...
sitzfleisch: the ability to endure or carry on with an activity
Something Oppenheimer did not have, apparently.
loaned from German, where it's originally a way to say buttocks, literally "sitting flesh". If you have more Sitzfleisch you can sit for longer. Both in the literal sense (a bigger butt makes sitting more comfortable) and in the figurative sense (having the mental ability to sit for longer, get more desk work done)
I think there is an IT bit of humor from about 1 decade+ back where the people who's proposals won out in meeting were the ones that could keep from needing to go to the bathroom longer.
Fascinating crossover from literal to figurative that you find so often when you trace language back far enough.
This was such an interesting thing to learn
This is fantastic. Now I have a sophisticated-sounding german word for my attention deficit.
There's also a less flattering reading of the word, where Sitzfleisch means having a "flat ass" (from sitting too much, e. g. Sitzfleischparade describing a group of flat-arsed people, or something like Sitzfleischmaxxer, and so on).
Wow, what a great comparison. LLMs are great at reasoning but absolute dogshit at simple arithmetic. If there's a raw calculation involved I always tell it to use python to add it all up.
Ever heard of string theory.
People go whole lives without being able to make it pan out.
String theory is a great example of a dead end kept alive by ego and sunk cost fallacy. An AI would have declared it dead and moved on 10 years earlier.
When people make these comments about string theory, are they at the forefront of quantum physics theory and have spent years on modern string theory? Or did they just watch a YouTube video and then keep parroting this?
You could reductio ad-absurdum this logic quite broadly. We all form beliefs about the world and its aspects, and almost always through fallible sources. Should one be confident about ones of questionable provenance? I say no. But forming beliefs from the information we have is a useful skill.
This isn't a great argument. Researchers who do not think string theory is good are not going to spend years on it. There are plenty of experts that dismiss string theory. I have no skin in the game, and don't care either way, fwiw.
Ok Sabina, everyone is string theory is all just a bunch of egotistical morons.
No, not morons, but people who have built a career on string theory. At this point, even if they regret their decisions, it’s too late to turn back now.
As opposed to having built a career complaining about string theory on youtube?
And that's the reason she's wrong?
Spending your life's work on a theory that is widely agreed to be untestable is certainly a form of ego, or at least is more philosophy than science.
This is a pretty silly belief to hold about science. Plenty of strong, well held standards of the modern era were considered ridiculous fringe beliefs a century ago.
Also out-speeding them, and that was before high speed inference.
Out-ralphing them, you might say!
https://ghuntley.com/ralph/
AGI ≈ artificial stupidity × infinite persistence
The key here is that it’s depending on the human inability to connect the sum of relevant knowledge, but said knowledge comes from humans.
Theres going to be this field day of low-hanging fruit that ML can round up, but after that I suspect it will be in fits and starts as a “connection maker” rather than some proof producer.
I think we're underestimating just how much low hanging fruit there is. I've been trying to apply this LLM research process to physics (QM and solid state) and there is so much missing in Physlib and the rest of the Lean ecosystem that most of my work has been trying to formalize the theories and validating them against the specification problem (and mostly failing badly).
It's not only going to be "connection maker". If and when robotics advance to a point where the LLMs are embodied, they can run experiments in the physical world and find new knowledge.
Being embodied is not the important barrier to running experiments. It's having access to a body of resources (i.e. funding and infrastructure).
For some research funding is mostly the salaries of the people doing the research.
What research works that way?
yeah but LLMs might be even cheaper than grad students
Robots in labs already exist, but mercifully they're not hooked up to anything as unpredictable as an LLM. Robots tend to work best as specialists doing high-throughput, extremely repetitive tasks which nonetheless require a degree of precision. Giving a robot a "human" body makes very little sense if we're talking about the needs and productivity of a non-human; humanoid robots are marketing for humans.
Humanoid robots are obviously more than marketing. The entirety of human civilization is human shaped. Making robots that are human shaped is easier and more efficient than redesigning and rebuilding everything that exists.
> The entirety of human civilization is human shaped.
That's the marketing pitch.
A plumbing robot doesn't need to be humanoid, an octopus shape may well be better for all the awkward corners. A robot police officer could be the municipality itself for sensory nodes (essentially the sales pitch of Flock etc.), plus some drones or robot dogs to perform arrests*.
The robot vacuum cleaners and lawnmowers we already have are nothing like a human. A robot taxi driver can be just the car. Robot dogs are already used for maintenance and security sweeps.
If you've got wheelchair access, you've got wheeled robot access. If you've got guide dog access, you've got access for Boston Dynamics' Spot.
* this may be a bad idea with current robotics, but I aver it's not improved by making those robotics humanoid.
> Making robots that are human shaped is easier and more efficient than redesigning and rebuilding everything that exists.
In the last 100-200 years, that has been proven wrong at every single step.
Counterpoint: cars. It takes sustained effort to prevent civilization from being modified to accommodate new technology, e.g. Stop Kindermoord.
Outside of sci-fi, marketing proposals, and niches like "Elder care in Japan" there are very few humanoid robots. By contrast non-humanoid robots have been mass produced and used in industry for decades. Arms. Carts. Trollies.
No people. If you want something with fine motor control and dexterity, it's easier to make that the robot and then have another robot bring the workpiece to the arm than it is to build a single robot that can walk around and do it. There are compromises in human features because we're generalists.
We're just going to slowly deconstruct every element that could be a factor of intelligence.
It's not out-thinking, it's just out-remembering
It's not out-thinking, it's just out-working
It's not out-thinking, it's just able to consider more things simultaneously
It's not creative, it's just randomly generating things and then selecting viable ones
If you make that list comprehensive, there's probably a Nobel prize in it for you.
A new technology being able to do something better than humans does not mean it’s intelligent though. A calculation program is not intelligent just because it can remember more digits than me, work more than me
But the difference really does matter and is not just a case of "whittling down" what intelligence really is.
We have known for a very long time that computers and machines are much faster than humans, more accurate, are scalable in certain ways that humans aren't, and they don't tire. I think most people who are not in the "AI cult" would agree that LLMs and modern generative AI are really just an extension of those faster/more accurate/more scalable and never tiring traits. But there does seem to be (and I'm sure folks much smarter than I have quantified this or described it better than I can) a fundamental difference in how humans think, especially as it applies to what true "understanding" really entails, and for the ability to think up truly novel and unique things that are not just a rejiggering/recombination of training data. I believe those skills really are at the heart of human cognition, and as impressive as LLMs are in replicating what this looks like, there are plenty of "LLM failure modes" where it's clear that LLMs lack a true understanding of concepts or the ability to generate useful, completely novel ideas.
There's definitely a lot missing from the current state of the art in machine learning that all brains manage to beat, and we can observe this just because an animal that needs as many examples as an AI to learn motor functions would starve to death before learning to eat.
However I can only guess that this is important, I'm not absolutely certain. They're at risk of being an economic disruptor just by being extremely stupid (by how much they need to study) faster than us to the same ratio we jog faster than continental drift.
>that could be a factor of intelligence
Could is carrying a lot of weight here.
Because, what's really happening is we're saying "Oh these things are what defines intelligence" then implementing them and /discovering/ "oh wait, there's more to this than we knew".
We've known, for decades, for example that an IQ test is not a measure of Intelligence, even though people still refer to it as though it is. A computer passing an IQ test, therefore, would have been thought of as possessing intelligence way back when, but would not now.
It's almost as if we're building something that... mimics intelligence.
Can you mimic intelligence?
You just did.
> If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc.
Your timelines are a bit unambitious. There's nobody expecting to make significant progress with a week of work.
> There's nobody expecting to make significant progress with a week of work.
You underestimate my ADHD.
Source: I am mathematician.
Hell, I underestimate my own ADHD.
Source: the post-it notes, ALL OF THEM.
They were just illustrating their point, I wouldn't take that literally.
In other words: Thousand monkeys with a thousand typewriters...
https://news.ycombinator.com/item?id=48231974
Take something like
If a thousand monkeys typed at a character per second, on a keyboard with the 23 relevant characters, it would take roughly 10^136 years for them to come up with this counterexample. Though, to be fair to monkey scenario, there's a large family of them known now, so it's not quite this bad: suppose there are a trillion permutations and similar examples that fit in this string. Then we are down to 10^124 years.If LLMs are monkeys, somehow trained LLM weights allow them to model and prune massive numbers of universes in parallel.
There are plenty of high value endeavors where being a superhuman knowledge remixer is right on target. But even capturing all of the knowledge is proving elusive.
I use coding agents. I think they're pretty good overall. They save me a lot of tedious coding. For example I probably wouldn't spend the time to implement native splash screens for all the build targets of a Flutter app, but I'll have the coding agent do it.
Nevertheless, for all the time that we've had coding agents, it's still trivially easy to find the jagged edges of their training. For example, Gemini evidently doesn't know if the Xcode part of a Flutter tool chain is misconfigured. That's not exactly a Millennium Prize problem. But it is shaped wrong for a training set for a coding agent.
While TFA itself makes sense I disagree with the title and the conclusion. I would not consider referencing working memory during thinking as “remembering” but as a part of thinking itself. Working memory is the RAM to the much larger but higher latency indexed database that is our long-term memory. As such I would say AI is out-thinking us, even if in a brute force sort of way.
I think where you could say it is out-remembering us is when it can contemplate the vast universe of patterns, gleaned from essentially all human disciplines, encoded in its weights, that may let it draw connections that a human could not, unless they just happen to be familiar with multiple disciplines.
Which is why I think the analogy with Von Neumann / Einstein is also a bit off. From TFA it seems Von Neumann was more akin to what AI does, than Einstein. I don’t get the impression that it was Einstein’s memory but his ability to look at things from a radically different perspective. So far I don’t know that we can categorically say that LLMs can or cannot do that.
Does it matter? It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
We don’t trillions of dollars in LLM investment to build things mathematicians don’t understand. We already have plenty of those, even from ancient times.
As to your second point, Terry Tao already has an answer [1]: the proof isn’t the contribution, shared understanding is. This issue was already raised back when the four-colour theorem was proved. Machine proving and machine proof checking are useful tools but they don’t mean anything without the interpretative work and the communication necessary to build shared understanding.
[1] https://news.ycombinator.com/item?id=49056620
If humans have nothing to contribute then shared understanding is a pointless endeavor. It makes sense now in the "centaur" period where human + AI > AI alone, but when AI mathematicians are both more rigorous and more elegant, then taking the time dumbing down their proofs to a human level of understanding is like requiring that we ensure all our current proofs be understandable by a monkey.
But but but... Does not AI exists (or has to exist) to only serve us.
Why not? We build cranes to hoist weights construction workers can't lift. We build electron microscopes to measure things physicists can't see.
Why is it so hard to imagine we can build tools to think thoughts we can't comprehend?
If there's commercial value, I think it's inevitable. We don't fund mathematicians because it's cute when they understand a problem, but because their work tends to have applications with commercial value. The value can be captured without understanding the details.
Can you give an example of an incomprehensible piece of writing (any writing, never mind a proof) that has commercial value commensurate with the costs involved here?
> Can you give an example of an incomprehensible piece of writing (any writing, never mind a proof) that has commercial value commensurate with the costs involved here?
It depends on what exactly you mean by "commercial value commensurate with the costs involved" but I'd volunteer the 3G/4G/5G specifications and the other documentation required to implement the mobile network protocols. 5G is currently sitting at over 50,000 pages and it's one of the reasons Qualcomm/Broadcom/Apple are the only ones who can realistically make a mobile radio.
I don't think there is a single human to whom more than a few thousand pages would be comprehensible at a time except for the occasional genius.
Yeah, how many people do you think understand the Linux kernel in full? What percentage of the people using it to great commercial effect can understand it?
How's your understanding of Schroedingers "An Undulatory Theory of the Mechanics of Atoms and Molecules"? You seem to be using the results of it as applied to semiconductor engineering just fine. And, I promise you, most semiconductor engineers haven't read it in full, they just accepted the results as passed on by several layers of teacher.
I have a paper on routing algorithms, which I have attempted to read to my cat. I don't think my cat retained much, but they seem to be enjoying the cat food that got delivered using the results.
I'd suggest that we're going to be a lot closer to the cat than the author of the paper when AI takes off.
You've moved the goalposts. The original claim was about producing mathematics that are in principle impossible for any human to understand.
All the stuff you've listed is understood by some person, and that understanding is the source of its value.
Now that we've cleared that up, can you furnish an example that satisfies the original claim of incomprehensibility and value?
There are plenty of artifacts that, if not impossible for humans to understand, then at least no human has ever completely understood. To start with, the universe as a whole. Despite that, we are able to choose legible pieces of it to model and perform useful actions from.
This will shift your argument--that doesn't count! etc., to the point where it's by construction unsatisfiable and vacuous. And it doesn't matter: an LLM might e.g. break some cryptographic algorithm in a way utterly unintelligible to humans, but the fact that it works would be sufficient on its own to make all of us choose to abandon that algorithm and choose different ones.
You asked for examples, for a technology that we're still building. Maybe you can see the issue with that?
Anyways, people benefitted greatly from Newton's laws of gravity, even though we still don't have a quantum-compatible set of laws for it. The laws of gravity are still incomprehensible for people, but the approximation that we've observed is still immensely valuable.
No, I gave you a lot more leeway than that. Take any utterly incomprehensible piece of writing from the entire history of civilization and demonstrate its value.
You keep falling back on "incomprehensible for some people" but that wasn't the claim. It was about a text which is incomprehensible in principle; that is, utterly impossible for any human to ever understand.
Can you give me an example single piece of writing that the author didn't claim they understood? I don't think that any utterly incomprehensible writing exists.
If you want examples where nobody but the author understands it, examples are a dime a dozen.
So you concede the point then. The age of humans comprehending things is not coming to an end. And therein lies the rub. When it comes to intellectual labour:
Understanding == Value
If a mathematician produces something incomprehensible then it has no value. It's meaningless. Indistinguishable from random noise.
An AI which produces incomprehensible text is producing no value. We didn't need to spend trillions of dollars on LLMs to figure that out. Markov chains can do that job perfectly well.
> how many people do you think understand the Linux kernel in full?
"Not many people understand some things fully" is so massively different from "the human mind is incapable of understanding some things that AI will understand for us"
Yeah. We're only starting the journey of building tools better at thinking than the human mind, so expecting me to have examples of things it produces is a little hard, don't you think?
The best I can do is things that are incomprehensible to nearly everyone, but still provide value. There's a small leap of imagination to consider an author that understands it and can show others how to leverage results without understanding be mechanical rather than biological.
I know right? And there’s only a market for maybe 5 computers in the whole world.
Paul Krugman (1998): predicted the internet’s economic impact would be no greater than the fax machine’s.
The 1876 Western Union memo dismissing the telephone as having too many shortcomings, and the banker telling Horace Rackham not to invest in Ford because the automobile was a novelty.
We are in good company!
Past performance does not guarantee future returns.
The entire point of writing proofs is for advancing human understanding. A giant dump of symbols that passes the lean compiler is meaningless besides human beings understanding it.
> The entire point of writing proofs is for advancing human understanding.
Proofs also enable AIs to direct search and generate knowledge. Verifiability is immensely useful for keeping AI grounded.
One might imagine AI generating enormous numbers of hypotheses and then trying to prove or disprove them, and then mine that data for new abstractions and heuristics.
In the field of pure mathematics this might be true, but it has implications regardless for applied math, engineering, and physics.
An AI may still be able to apply the results without humans understanding the proof.
Sometimes the purpose of the proof is simply to demonstrate that some construct is a safe assumption for other more interesting work-- and could still serve that purpose even if it was entirely a black box.
Is it the AI's fault we can't understand? If the GUT is beyond human comprehension does it matter less? We don't apply this reasoning to other animals or even to less capable humans. Besides, the robots may want to ponder maths for their pleasure.
> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
That sentiment makes me cringe. If you understand how LLMs work, you'd know it'll never be possible without a fundamental change in how these work.
We're also supposed to be reaching that point, somehow, without the LLMs ever being intelligent (in the dictionary definition sense, not the "high reasoning model" marketing sense).
Based on observations, the ones who are fooled by the supposed emergent properties, are just that, fools. Any sufficiently unintelligent agent will perceive transformer based LLM text predictors as possessing high intelligence.
Maybe you can enlighten us. In what way aren't LLMs able to make new contributions.
LLMs in agentic harnesses are Turing complete.
To my best knowledge, we don't know of any greater computational model that the brain is a part of, that LLMs are not.
Obviously actual intelligence has an ineffable essential aspect, just like unicorn farts do.
First of all, the argument isn't that LLMs (with I assume some automation) cannot be used in searching a problem space. I'm assuming this is what you're referring to, in terms of contributions?
That's the part where LLMs are used as tools. Which there are plenty of places where they are useful.
Also, do you know what turning completeness is? Why are you bringing that up here?
The crowd that AI psychosis has brought to HN is interesting. But not in the "I'd love to learn more" kind
> That sentiment makes me cringe. If you understand how LLMs work, you'd know
> But not in the "I'd love to learn more" kind
I hope you are able to see the problem in your own communication here.
Computation classes are interesting because they say something about fundamental capabilities.
Two machine that are Turing complete are in theory able to carry out the same computations. They are isomorph mediums of computation.
Regardless. Please keep it sober. If you think you know something, enlighten us. But don't just propagate out lies.
"The age of humans comprehending things is coming to an end"
That's something AI companies would really want you to believe.
> That's something AI companies would really want you to believe.
Why would I care what they want me to believe?
Intuitively it would make sense that you can put math ability on a chart with a value for “general public” “smart high schooler” “smart undergrad” “smart PhD/ professional”. And you could place frontier AI somewhere on that chart over time from GPT 2 to now and see the trend.
Then you’d have to consider that either you believe there is a fundamental limit that is below peak human mathematician level or there’s not.
> Why would I care what they want me to believe?
How would you not care? Are you a robot?
They can say random stuff with the goal of increasing their shareholder value. Things they spit out do not have to be true. It is not easy to verify things they say, therefore, everything they say should be taken with a huge grain of salt.
Consider Enron and Amazon at the turn of the millennium. They were both telling you what the future would look like. The right action would’ve been to just ignore what they are saying and try and get data and reason about the world. It didn’t really matter that both Bezos and Jeff Skilling wanted you to believe various things - one was right and one was a scammer.
So that’s what I’m doing here. For what it’s worth I find a lot of the AI people’s worldview very consistent. They believed AI would be the most important technology of our life times and committed their work to it. Some of these same people are total liars so yeah I won’t really hang onto their every word.
He's saying he can connect the dots without their help.
Yeah, my bad.
that does not make it not true, nor does it make those companies or their products not dangerous. I like looking at videos of animals that tear other animals apart and eat them; lion cubs are super cute; but that does not mean I want to be thrown into a cage with a model of a lion that has not been programmed to be disinterested when it is sated. AIs appear never sated; humans using or making AI wanting money, even less so. I suspect the AIs will understand the cost long before the humans will, not that anyone making money would care.
> It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers.
I agree.
> The age of humans comprehending things is coming to an end: our brains just won't have the capacity to make meaningful contributions to science, math, or technology.
I don't know if I see this being true for quite a while, if ever.
> It's going to produce proofs far more intricate than humans can understand, outdoing humans and opening new frontiers. > I agree.
There's an infinite space of possible statements and proofs. The only thing that makes certain proofs significant is that human mathematicians consider them significant; if AI came up with a proof of some statement that no humans could understand then no humans would bother investing further resources in building upon it, for the same reason we don't waste computational resources iterating over the infinite space of true statements in first-order logic.
You’re prescribing elegance to a stochastic generator trained on the wealth of humanity, including 4chan. Let’s set our expectations a bit.
Your brain is a stochastic generator. Have you read 4chan?
Your brain is not just a stochastic generator.
It also creates and updates models, uses those models to make predictions, guides the stochastic generation by comparing the output to those models and reworks them in real time.
what is it?
It also creates and updates models, uses those models to make predictions, and guides the stochastic generation by comparing the output to those models and reworks them in real time.
Not sure if you're referring to LLMs or humans here
In limited experimentation: AI will certainly make statements that are extremely intricate and hard to understand, in part because they're overcomplicated and in part because they use a bunch of unnecessary terminology.
This is not to say that a human couldn't understand a streamlined version or that the AI would not be better if it made more streamlined statements to begin with.
(I am not saying that everything mathematical that an AI produces is in any sense trivial.)
It’s possible, but there’s a difference between vastness and difficulty.
Humans can’t compete with AIs on vastness of material they are familiar with, or the depth of effort they are willing and able to throw at a problem.
But scale isn’t the only aspect of difficult scientific endeavours. There’s also theory. And advancements sometimes come through hard graft of knotting together many things. And sometimes they come through the revelation of a deeper truth, or a new framework, a fundamental insight.
AI might help us reach the next level. But that doesn’t mean we won’t understand anything. It could be we have periods of vast intricacy we cannot follow, punctuated by profound elegance we (or at least experts) relatively easily can. And then the scaffolding we needed to get there falls away.
I don't think that's true. Human intelligence is limited, and our brains are inefficient machines.
The tools we built to replace muscles have mostly obsoleted raw strength for tasks like excavating earth.
There's no reason to think we can't do the same for brains. And then we'll never need to think for a living again. Some people may want to do it as a commercially insignificant hobby, of course, the way people lift and compete in strongman competitions today.
We'll have AI taking care of our needs, the way a good mother takes care of their children.
A good mother doesn't raise children to be dependent upon her for all their needs.
For this to actually work in a way that benefits our species, humans will need to become something else/next through their interaction with the technology.
It's inevitable -- we won't have the machinery to compete, so we either have an aligned AI taking care of us, or we end up with a big problem.
Anyways, sipping wine on the beach and doing puzzles when I feel like sounds nice.
looking for scraps in the gutter of never seen sun city more like
>our brains are inefficient machines
The human brain is exceptionally efficient.
You could be right, but you're making a lot of assumptions about how complexity, scientific understanding, and explanations scale. One of the features of a good scientific discovery is that it often simplifies and compresses things that were previously a bunch of scattered facts. Also, as AI systems improve they'll get better not only at making scientific discoveries, but also at producing understandable explanations.
That maybe true at some point, but i don't think we are there yet.
Yeah, it's probably a few years out.
If and only if that is actually true, then perhaps nothing matters. Until then, calling out shenanigans remains a noble art.
But apparently we can teach machines to do it for us
Yeah. We can also teach machines to move hundreds of miles an hour, but we could never do it ourselves.
>produce proofs far more intricate than humans can understand
Math is not magic, a proof is just a series of applications of a set of rules on some axioms. A mathematician could understand any proof given enough time to study it; the only way for AI to make proofs that a human couldn't understand is by making really, really long proofs.
Why would you want something you don't comprehend? How can you be sure it empowers you?
I think perfect rationality doesn't exist, because it is rational to reject something that you don't understand. So rationality of a given physical system will always be bounded.
Yes. That's how LLMs do programming, mostly. It's also why LLMs don't need abstractions or parsimony as much as humans. They can work on something complicated without simplifying it first.
This has major implications that haven't been fully realized yet. On the math side, there are long machine generated proofs. On the code side, there are high volumes of code with similar code not being folded into functions.
This is the exact opposite of what I’ve been dealing with for awhile. LLMs absolute cannot work on something without an understanding unless they can outsource the understanding to a verifier. If you’ve got an easy to check function to measure progress then “keep going” is all the prompt you need. But if you need it to figure out “I pushed the up button and it moved up and left” then it’ll find the same bug five ways without realizing it’s just one bug in the underlying math.
For greenfield projects LLMs don't need abstractions, but as the project gets more complex, the right abstractions save a pot on input tokens (less code to read) and reasoning tokens (less work to do to figure out the code), so they free the context window for higher purposes
Also I suspect that, apart from that, the results on smaller, cleaner codebases are better. LLMs degrade when following more than N instructions (where N depends on the model) even if the context window is not full yet; I suspect they also degrade when code has too many unnecessary concepts and details
LLMs use abstractions a ton in code though: standard library functions, popular libraries, etc. They just dont always make their own abstractions. At least not particularly good ones. LLMs work really well when they have well abstracted pieces to put together.
I've been working on generating a large code base for the last couple of weeks. Finally got around to generating a sort of code-duplication report and have spent the last week just having it de-duplicating logic that had been strewn all over the place (eg 11 different functions all doing date math to add x days to a date). dozens of items that had each been similar functions duplicated numerous times. crazy. (opus-5-utracode)
Can LLMs not do this for you? Or would they go too far?
No, the problem is they're still really dumb, and lack the ability to make logical connections that are obvious to us. "should I walk or drive to the carwash" being a very recent example of the larger problem.
They add and add new code to the point when adding more is going to become very messy and then spagetti
This means some AI proofs might be impossible to comprehend by humans, right? I guess AI still lacks human intuition for many concepts, but AI might beat humans in narrow areas, such as discrete math and combinatorics.
Outside of math you can basically take the entire corpus of research papers on any topic and have the AI read all of it and provide an analysis cross referencing everything all at once. This applies to everyone and everything.
I love the term "Out-Remembering"! I have been trying to find a way to communicate that "intelligence", "creativity" and so on might be misleading about the true nature of LLMs, and they would better be described as genious "reproducers" as in, they are very capable at reproducing what they have already seen - and they are a bit less capable, but for many use cases still good enough, at reproducing a mix of concepts seen previously.
This also nudges into how to use it best: By knowing where the "piles" of if training data are (i.e. when it comes to a CLI in rust, I just briefly describe the use cases, and I have a very high confidence the code will work exactly as intended by me since there will be a multitude of examples in the training data), one can predict where the LLM is likely to go wrong an prompt/guard accordingly. This skill grows with domain expertise, and is one of the many reasons LLMs can be (and probably should be) used to outsource busy work, but never understanding and learning. ("never" is a not meant literaly of course - I for one am glad that I do not have to wrap my head around CSS and other frontend topics and go straight to the topics that interest me most)
"Out-Remembering" captures that perfectly, I feel. Also goes nice along with "asking it leading questions" as we know how to do in real live; if you want a person (LLM) to confess (produce output tokens) something, sometimes you do that by leading the interogation (chat, context) to where you think the truth lies.
What I'm looking forward to amidst all the negativity, fear, and loathing is for some 20something mathematician to outdo both humanity and machines by leaning hard into centauring to expand the frontiers of mathematics. Pretty much what I think the future will play out to be as well, but I don't think people are ready for that yet.
People aren't ready to prove/disprove every one who access to LLM and claims has changed course of humanity
Takes the rogue having access to the $$$$ models though.
Nobody IRL cares about nerds btw.
Exactly, it's making connection across vast set, not bringing the magic intuition.
Has anyone tried feeding all of human knowledge to an LLM prior to Einstein's work and tried to have it reinvent physics?
Where would you get enough pre internet tokens to train a near-frontier model?
This is why education used to start with rote memorization. Functional intelligence isn't abstract, it is based on useful information you can quickly recall.
C'mon AI companies, pivot to lawyers or doctors already.
Trying to convince us that mathematics and software engineering are "solved" is getting very tiring.
The pushback would probably be too much for the soon-to-be IPO-ed companies.
Mathematics is typically concerned with "proofs" [1], which similarly to code, often allow for strict validation. Thanks to reinforcement learning techniques, it is now possible to train LLMs to perform very well on code generation, and mathematical proof generation.
Law and medicine are fundamentally harder fields to obtain decent training data for, and LLM results are therefore expected to be less powerful. Also, making mistakes in these fields is costly, but perhaps you were alluding to that already.
[1] https://en.wikipedia.org/wiki/Mathematical_proof
Mathematician here. There is a lot of recent work on the Lean project -- when a proof can be translated into Lean code, then it can be strictly and formally validated.
https://lean-lang.org/
But otherwise, mathematical proofs are read and written by humans, and at the end of the day the relevant standard of proof is what other mathematicians will accept.
Occasionally, mathematicians don't agree. For a prominent example, you can read about Shinichi Mochizuki's claimed proof of the so-called ABC Conjecture:
https://en.wikipedia.org/wiki/Abc_conjecture#Claimed_proofs
For that particular example, it's now been proved that Mochizuki's proof is incomplete: https://zeli.app/en/story/48963019
I guess whether he will eventually fix those gaps and resolve the issues remains to be seen.
> Law and medicine are fundamentally harder fields to obtain decent training data for
I think this bubble has given a lot of people software brain and are trying to apply it to fields it is wholly inappropriate for, though. Law is about argumentation and rhetoric. It is about providing a persuasive argument. This is how it is taught. The actual legal code is a way to formalize parts of it, but increasingly I see people angrily insisting that the only thing that matters is the text.
As you might imagine, I find textualism a load of applesauce, but I don’t think the vast majority of people making this argument even understand textualism as jurisprudence. It seems to stem from Crypto bros and the whole “code is law” argument which is just codswallop.
They try to sell AI as lawyer or doctor replacements as well. But because it’s HackerNews we are biased towards our domains to see them more often.
Well also you can't just throw AI slop as doctors or lawyers advice and fail multiple times until you find the right answer. With code and math you can have failures 1000 times for every success and still get rewarded.
A neighbor of mine whose husband is a lawyer said it's already part of his regular workflows. OpenAI also already have HIPAA compliant offerings targeting healthcare uses, etc. Of course they already do these things.
They can rely on compilers, solvers, theorem provers to validate the generated softwares and maths. That’s what makes it possible to iterate quickly in a loop and self correct. You cannot do that in soft industries like legal and medicine
That is not the point I was making. I am not talking about validating software or maths. It can generate stuff that is valid, but bad and incomprehensible.
What I’m saying is that AI labs are talking so much about software and maths because we already have tools that can say « it’s all good ». That makes it possible and worth it for them to spend 1 week of compute on a problem until the validator passes, then publish marketing pieces. You cannot do the same in medicine or laws (modulo some niche areas)
As AI keeps improving, the definition of intelligence will keep changing in order to exclude it without explicitly saying so.
It is obvious that super intelligence comes from more working memory.
It is the scary thing actually. Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
We can decompose and write things but only up to a point. when Ai can have a working memory that spans hundreds of books, we are necessarily going to have to trust the system.
This is why we have hierarchies of abstraction. Pretty much every field of mathematics relies on constructing notations, models, and other tools to simplify things in a way that is verifiable. LLMs rely on the same basic technique, they can just pull from a wide variety of these abstractions at once. So far we've been able to understand their proofs just fine. Computer-assisted proofs in the past that relied on brute-force is where we have run into trouble. We cannot reason about millions of possibilities at once, and we had to trust that the computer program that analyzed them was correct, which is a really hard problem and leaves humans fairly unsatisfied. I think we are actually progressing in terms of understandability in computerized proofs.
We offload working memory to paper if we want to understand something that does not fit into the regular meat bits.
>Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
That doesn't follow. We could still understand it just by studying it and committing it all to long-term memory, it just takes longer. And there's a hard cap on the working memory of LLMs, due to the quadratic scaling cost of the full attention layers that have proved unescapable for all SOTA LLMs.
“is obvious” -- that’s what my Russian math professor said in college before skipping the rest of a proof.
But was it?
Aka - it’s a stochastic parrot with a good memory, for anyone still struggling to understand this. It should be obvious, imo, but some people seem to have trouble with the concept.
I don’t understand why this is such big news. OpenAI has essentially made a bunch of marketing copy by gussying up an algorithm being given a near unlimited budget to stochastically permute through its lossy memory.
I think the real scandal is that we are almost 3-4 years into this (I think the release of GPT 3.5 is a good marker of when this public frenzy started) and all we’ve seen is OpenAI and the other major AI frontier companies constantly retracting their preposterous claims every time. We appear to have reach a local maxima in that it has some value in places that tend to be a little easier to scope and limit (computer programming, mathematical proofs). So, given the actual useful economic value this has provided, does this justify the investments? I think we are approaching 1 trillion in CapEx for AI [0]. For context, I believe the annual GDP of Norway is $600 billion.
[0] https://www.fool.com/research/ai-companies-spending-on-data-...
That's exactly what the strength of AI is, no? Reading all the world's knowledge and recalling it in an instant and seeing where it applies.
Precisely perfect for replacing lawyers, if nothing else..
Metacommentary: how did this post get to #5 on the front page with 1 upvote within 2 minutes of submission?
That’s how HN works, I had that multiple times over the years with my own submissions. Sometimes it gets picked up quickly, sometimes not. A post can also down rank very, very fast. It depends a lot on the level of engagement and the type of engagement
Follow the money.
How fast the upvote happened?
Specifically working memory. If you can’t hold enough concepts in your head then you can’t see how they all relate in one giant theory.
And the goalposts must move once again..
> But chunking does not eliminate the limit. It merely compresses the information.
Yeah, as expected, an article about AI that's at the very least been polished using AI. For fucks sake we need an LLM flag to filter out slop.
"It's not X, it's Y" hot take AI slop.
100%. Context is big for AI, but it's nothing compared to everything a human can learn. If you efficiently represent everything in context, it may be many papers, but if AI is actively working through proofs, it will quickly fill up. They're no denying AI is making strides, but pinning it to memory is an oversimplification.
I don't have to open the article to be confident it's not worth reading. Anybody knowledgeable in the field should be familiar with AI writing tells and the message they send. It only takes a few seconds thought to transform the title into something like "AI beats mathematicians by out-remembering, not out-thinking." Regardless of whether the article is slop or not, I expect any competent writer to avoid slop phrasing in their titles. To do otherwise signals laziness.
Most mathematicians are quite simple creatures. I can do basic math, some derivations, but my bright days of solving differential equations are far gone!
Computers are simply better at math now, like in chess or go!