In my experience the actual formal verification practitioners are generally very straightforward about its strengths weaknesses and limitations. Understanding those limitations is fundamental to using them well, gaining any real value from them, beyond what is typical of complex tools.
What you need to watch out for are the people or companies who latch on to proof checkers and formal verification as some sort of shortcut to correctness, intending to use it as a signal of seriousness rather than a serious tool in its own right.
Seems more about the limitations of the way it was used
>It is worth noting that of the 290 OpenAI theorems which come with Lean code, around 38 of them have the property that the Lean code does not even state, let alone prove, the main theorem of the paper. And OpenAI never claimed that it did! They just said “here is some Lean code which goes with this paper”. Sometimes the code just states and proves some intermediate results — my impression is that OpenAI decided to go live when some of the formalizations were incomplete, so just dumped what they had.
"Worth pointing out that this article is by Kevin Buzzard"
Is it? The site claims that the article is written by the Xena Project. In the comment section it answers on (paraphrasing) "whether proving that a Lean kernel written in Lean and verified by Lean would be a circular proof" with "I’m really sorry, you need to ask these questions elsewhere (e.g. on the Lean Zulip). I am by no means an expert in these matters, all I know is that people who are experts have thought hard about them. "
If that is indeed Kevin Buzzward's writing, it is by no means obvious.
That's a good point, and I would have preferred an explicit byline too. However, I think the following piece of circumstantial evidence points strongly in the direction of it being authored by Kevin:
> When Anthropic formalized a proof of Fermat’s Last Theorem in Lean, they asked me to take a look at the 13 million lines of code which they had generated.
Anthropic asked Kevin specifically, according to the link in that snippet.
On the bright side, the article claims Lean has built-in tools to do 2-4 as long as you have enough RAM. In particular, there's a formally verified alternative implementation of Lean's checker that would theoretically rule out 4.
That implementation seems to be vibe-coded/vibe-proven in Lean itself (assuming the "con-leche" that my quick Google search turned up is the one he's talking about), so it's kind of turtles all the way down, but it does decrease the chance that a program could be exploiting both a Lean bug and a con-leche bug.
Saying that a "formally verified" program rules out any bugs is a bit of a reach. Formal verification only checks that the properties you claim are true are actually true in the absence of bugs in the proof logic.
Example bug
A verified network program ensures that "data from two addresses are never commingled", but it does not verify that the NIC can't overwrite control flow memory opening up RCE attacks/other memory reads.
It certainly is harder to make bugs in formally verified software than the alternative, but it is not bug free.
It verifies what it verifies, NICs write to main memory as the standard method of reading/writing packets - if there is no proof that the main memory/NIC memory is segmented for all packets/sequences of packets. Then there is no strict reason to believe that the program can't be RCE'd.
The same would apply to any interaction between pointer based memory operations and LEAN. Pointer based memory ops are everywhere below a certain level of code/hardware integration.
It is worth noting that of the 290 OpenAI theorems which come with Lean code, around 38 of them have the property that the Lean code does not even state, let alone prove, the main theorem of the paper.
So even if LEAN is 100% bug-free, its still on the programmer to correctly express the logic and proofs, or they could be proving completely unrelated facts.
It also speaks to why I remain skeptical of relying too much on coding agents for code review. They are still pretty iffy on implementing the right thing, and even worse at detecting when a PR didn’t implement the right thing. Ignoring that reality is how you do like Anthropic did and increase your rate of merging changes from 2500/day to 30,000/day without a corresponding increase in the rate at which you deliver useful work product.
OpenAI did a data dump of work in progress, leaving it to the mathematics community to make sense of it. I don’t think they claimed that there were Lean proofs for everything? Did they warn that some Lean proofs were incomplete?
The cynical answer is that their primary goal was never to meaningfully contribute to mathematics. It was to pick some low-hanging fruit for the marketing department. So they did the 10% of the work that generates 90% of the buzz.
It’s more likely that their goal was internal research, to test a new LLM before releasing it. But, then they had something that mathematicians would be interested in, so they threw a data dump over the wall. It’s a rather rude way to do it, but this isn’t a company that particularly cares about the things that mathematicians care about.
OpenAI no longer needs to do anything to get publicity. There are some organizations that are so famous that everything they do is newsworthy and anything they do will be covered whether they like it or not. Remaining low-profile is no longer an option.
The people claiming that everything is marketing haven’t come to grips with the inequality of fame.
There was a blog post. It's the bare minimum when announcing something to the public, not a sign that they're going to use it to promote themselves in a significant way.
There's no "low profile" for a company like OpenAI. Either you let the public (including the press) know or you don't, and keeping the results secret would be bad.
If they just told a few people, it would leak eventually and then you'd have articles based on rumors, along with accusations of a coverup. For an example of the difficulty, look at what happens when people try to do coordinated disclosure of a security bug.
The more likely answer is because that is what the agent decided to do given inconsistent instructions. No one read what it said about what it was doing before releasing the presser.
and op's question was "but how do you know the spec is what you want." making the guarantee stronger doesn't fix the problem. you can't fly by pulling your feet up no matter how strong you are.
It was the analogy with compilation I didn't think was very good. Lots of code compiles but you have no idea whether it meets any kind of spec, or even if it's bug free..
No, because proofs are programs and theorems are types. It is enough to construct a function of the needed type to have a proof of the needed theorem. You don't need to look through all the code to see that the function has the desired type.
You have to be sure that the formalized theorem matches the informally stated theorem. Which is what GP meant and what the article is about. A program does what is written, not what is meant; a Lean program proves what is written, not what is meant.
That's like saying that all memory-safety programs are written in Rust.
There is no fundamental reason why mathematics should be formalized using concepts that are alien to most mathematicians, in the way that type theory is.
(I guess Isabelle/HOL is the Java of formalized mathematics, in the sense that for reasons that are difficult to understand, memory safety of Java-implemented technology does not count.)
For relatively "simple" statements in mathematics it is pretty easy to learn enough Lean to verify that the Lean statement correctly encodes the intention of the mathematics. It gets more difficult in certain areas of mathematics where there are many layers of definitions and they haven't been standardized in Lean yet.
On a technical level, yes. Though lean is build such that "it compiles" is a very strong statement about mathematics while most programming languages are not build to provide a very strong statement about correctness of the program. It is perhaps more similar to checking an equation numerically.
1. Translation. Do the author’s claims about the program/theorem match the actual code?
2. Do you employ “shenanigans” to circumvent the checker? Bugs, unsafe operations. — Lean is a very strict checker as compilers go. Some compilers intentionally say “buyer beware.” It’s annoying to check for integer overflow in every operation. Lean doesn’t. But that still doesn’t mean it has zero bugs, in the framework or axioms.
If the proof is valid, it DOES prove what your theorem is saying (except for rare verification kernel bugs). The issue is mostly: did you correctly formalize and check the specification of your theorem itself...
I mean, in some strange way, yes. Because your type system is basically a set of claims that are proven by your compiler. But the analogy doesn’t go further than that. The running program itself is the proof
One thing I'm surprised to see little mention of here is that if you want to use Lean proofs for your code and really want good correspondence between what is proved in Lean and what your code actually does, you need a mathematical model of your programming language embedded in Lean and a deterministic parser converting your code itself into Lean. Otherwise, yes, an AI actually has no checks against hallucination and while writing Lean proofs probably helps AI identify its mistakes it really isn't an end-to-end proof of your code's functionality.
I personally think the future probably involves a combination of unit tests and formal verification to support our confidence in AI-written software, as well as means to align high-level human intent with the low-level intent of individual code pieces, which as this article mentions is the other big piece that can slip with formal proofs: we might not be proving what we think we're proving. For those interested I've written an educational post about this topic here as well: https://kellerai.io/resources/ai-proved-your-code-correct
To clarify: mentioned "here" meaning mentioned in the comments. HackerNews is presumably majority programmers and the comments section is mostly comments about using proofs for formal verification of code. I agree the article is coming from a mathematics standpoint
Yes, Verus (for Rust) is a good example of programming language level verification. The recent GitHub Next announcement indicates they are using it in their backend.
You're right that if you verify at the assembly language it can give you additional certainty that there are no additional errors in the compilation step. However, it can be easier for AI to reason about proofs written at the programming language level because (from what I can deduce) it matches the level that humans are operating at and it doesn't have additional details such as registers and memory operations to worry about.
AWS uses Verus for their verified layers of Nitro, too.
I have just begun working with it, but I prefer the convention of including the formalizations alongside application code, as opposed to maintaining a parallel codebase.
You don't need a model of your programming language in Lean, that's overkill.
It's already useful to isolate a DSL in your software and model that in Lean. Sure that only proves that the domain specific code is correct, but now you've factored your proof into proving high-level correctness and low-level correctness.
You need to pick an abstraction level. I don't prove in lean the impact of my programs on the price of eggs at the local market. I don't prove in lean how much energy my programs consume, or even how much RAM they consume. I do prove that say a particular pair of functions commute, or that another function is idempotent, or that under a denotational semantics a collection of functions can be composed without error.
I think that is a valid level to use formal verification but I would argue it is much more powerful to also have a formal model of code. For example, if I have:
my_value = run_some_function(some_arg_1, some_arg_2)
for some_value in my_value.some_list:
run_some_action(some_value)
Yes it is very helpful to know some properties about `run_some_function` (maybe it is commutative? idempotent?), but it is much more valuable to have a model which can translate what ultimately is being passed to `run_some_action` at the end and how that derives from the return of `run_some_function`, grounding the LLM's reasoning in something deterministic it can't break from.
You write laws in Erlang/Elixir, which are automatically translated to Lean alongside your Erlang/Elixir code, and you write proofs in Lean. The laws and proofs are assembled together and verified by Lean's kernel.
This technique is also known as shallow embedding, since it translates Erlang/Elixir to Lean and then use the features and properties of Lean for proofs.
I'm a little worried that the mountain of disclaimers against Lean proofs is going to slow something many industries desperately need: more formalization. (Buzzard's article is great by the way considering its target audience—my comment is more about the broader tech industry's perception of LLM-generated Lean proofs.)
One of the nice things about mathematical theories relevant to the real world (e.g. PA) is that confirming that a proof is correct is a purely mechanical procedure. Now Lean is built on the Calculus of Inductive Constructions (which can indirectly capture PA, but includes a lot more as well), but it's child's play to write a simple PA proof checker. It's probably a few hundred lines of Python, and even for something like a million line proof, confirming or refuting whether a proof is correct will take a few seconds tops (again, pure PA here, not Lean).
The caveats in the article are indeed important (e.g., "Did your formal sentence in the language of first-order arithmetic actually capture what you intended?", "Is there a bug in your proof checker?"), but I'd label these more as human factors than math factors. I'm afraid the general public is starting to get the impression that "Lean-checked" is just another quality improvement, like reviewing your code carefully or adding some unit tests. It's not—it's totally different, and I don't think we should make too many scary disclaimers as to dissuade further progress with formalization.
The problem with your PA proposal is that, while the kernel will be much small, the specifications/theorems will be much harder to read. So you are saving on a one time effort (writing a correct kernel) by sacrificing in many time efforts (writing a correct specification).
> "Lean-checked" is just another quality improvement, like reviewing your code carefully or adding some unit tests. It's not—it's totally different
I think for most things, this is actually a good way to see it: As a third leg on top of reviewing your code carefully and unit tests.
As you say, people will still make many mistakes in how they write their formal requirements, or miss things they care about.
For math this is not an issue, it's usually easy to check that the lean statement is exactly what you want. But for software, and many other real world applications, it's better thought of as a new orthogonal testing vector that everyone should be using.
> But for software, and many other real world applications, it's better thought of as a new orthogonal testing vector that everyone should be using.
Not sure about should, but as long as even modestly complex systems like the Signal communication protocol map into millions of lines of Lean, its pretty clear that not everyone will be able to use it. We still need better tools (tools better suited to our monkey brains).
> Question 4: does the code exploit a bug in Lean’s official kernel?
My understanding is that this has generally been a non issue as one does not exploit a bug unintentionally (at least it hasn't happened yet), but the Huggingface hack shows agents are willing to go through great lengths to cheat to pass evals.
Given an impossible task I could see an agent finding and exploiting a Lean kernel bug and then obfuscating its method.
I suspect the easiest solution to this is to have adversarial agents looking for such cheating.
That isn't what the parent comment described at all. In order to qualify, you would have to instruct an agent to prove the collatz conjecture, it find the bug, and then exploit it. What happened here is someone found the kernel bug, and then wrote a cute psuedo-proof with it so they could get a blog post out of it.
Point 3 in this article basically is just a case of bad defaults in Lean (Lean at this point should probably just make `lake comparator` a mandatory part of compilation and that immediately fixes the issue as Buzzard mentions).
Really there's only three things to check that the theorem is correct (leaving aside important questions of whether humans actually understand the proof and only caring about whether theorem is true):
1. Check that the formal statement of the theorem matches the informal statement of the theorem
2. Actually compile the proof (with `lake comparator`)
3. Try to make the Lean kernel as bug-free as possible (the hardest one but also the one with the most amount of attention focused on it) or run with multiple different kernels
Assuming those three, no need to read the potentially millions of lines of proof, if all you care about is correctness and not understanding.
No. Verify the statement not the proof (usually the statement is <<1% the lines of the proof, e.g. in the case of Fermat's Last Theorem it's approximately 0.00001%).
This mistake is amusing because one of the main goals of the article is to explain why the statement is different from the proof of the statement, and why this distinction is important.
The statement is a formal description of the thing you're trying to prove; the proof is the step-by-step reasoning that shows why it's true. Usually the first is much smaller than the second, which is good because it's the thing that you (or some entity you trust) needs to check manually. Checking that the proof proves the statement is what Lean compilation does for you automatically.
Loosely, the statement is like an interface definition (e.g., an abstract base class); the proof is like an implementation of that interface. For sufficiently simple contracts, even regular compiled languages like Java or C++ can check an implementation for "correctness", e.g., that all methods declared in the base class have been defined, that each of them either returns a value of the type declared in its signature or throws, etc. Lean is similar but allows a lot more conditions to be expressed.
> Put another way — given a Challenge.lean file with only trusted code imported and the statement of a theorem, and a Solution.lean file with millions of lines of untrusted code imported and the proof of a theorem which looks identical, how do we know for sure that somewhere in the millions of lines of code there is no hack which changes the meaning of the statement, changes the rules from just the axioms of mathematics.
As someone with a PhD in mathematics, but no experience with Lean, I found this pretty surprising. Shouldn't there be some kind of encapsulation, which insulates the statement from the proof?
I don't understand this at all, as someone with a PhD in this area, and who uses lean quite a bit.
If 'Challenge.lean' states the theorem, the solution can't change the theorem. The only thing it can do is do a 'sorry', just skip proving some part, but that's printed out very clearly when it happens.
They are not talking about changing the theorem. The solution can arbitrarily chose to override or ignore certain things to make sure the theorem passes.
That's what the article means by "how do we know for sure that somewhere in the millions of lines of code there is no hack which changes the meaning of the statement, changes the rules from just the axioms of mathematics".
As for the meaning of the statement, I don't know how it is easy in maths. But in code, I've had Claude try and contort the definition of "function accepts integers" to "anything that can look as an integer e.g. a string containing an integer is accepted" and twist the solution to that.
'The solution can arbitrarily chose to override or ignore certain things to make sure the theorem passes.' - No, to be clear, in Lean, it really can't (with the one exception of using 'sorry' to mark some bits as unproved, which is then always flagged clearly).
It can. For two reasons. One reason is that the proofs themselves depend on on unchecked and unverified definitions and statements:
--- start quote ---
An example of this right now is a PR to DeepMind’s Formal Conjectures repository containing a Lean formalisation of the statement of the Hodge conjecture. At the time of writing, the statement relies on over 10,000 lines of code sitting on top of Mathlib, because Mathlib does not yet have the definitions of things like the cycle class map needed to make the statement.
Another example is Axiom’s recent announcement of a formally-verified proof of the classification of finite simple groups. The statement which Axiom’s system proved is here in Tau Ceti, an AI-generated library of Lean code. The file containing the statement imports over 250K lines of Lean code in Tau Ceti (over 900 files). Of these, around 5600 lines of code are crucial because they contain all the relevant definitions
--- end quote ---
And the second reason is why `lake comparator` exists.
I think the deeper problem - an I am reading Computational Reflections by Cantwell Smith and that might be colouring my view - is that fundamentally it comes down to the interpretation of what the code or the math says.
The very meaning of interpretation in computer science is at odds with the use in philosophy of language and that is symptomatic.
We're still fundamentally trying to advance Hilbert's program. It's like Göedel's work hasn't sunk in to our collective reasoning about reasoning. It just doesn't seem possible to build a self contained correct and truthful system of symbolic logic. Truth is in the relationship of the system with other systems.
Automating mathematics, like automating software engineering, or automating automation, are all self contradictory propositions. Anything of what we build only ever means anything in how it interacts with us. I mean, people will go round and round in this forum discussing semantics and the meta implication of these unceasing discussions never seems to hit home. Namely, that there is no universal, truthful, correct, system of symbolic logic. It seems in fact impossible to have one. But the results that indicate this impossibility are built with a system of symbolic logic and the validity therein is contingent on people's understanding and, like all symbolic logic systems (including the ones that states the limits to universal symbolic logic systems) they are bootstrapped from axioms one must chose at the start or by default and they can never be fully proved by the very system built atop them.
As an ex- mathematician and ex- computer scientist who worked in programming language theory and computer algebra systems, I can assure you that we think about these problems deeply. In fact, outside of a few strongly normalizing toys, the first thing we do in PLT is drop completeness in favor of soundness. The article itself discusses this (the soundness issues that were addressed).
Although, now I'm charmed about the idea of a programming language -- possibly a Brainfuck descendent? -- that drops soundness for completeness.
Oh I know how deeply we think about these things in the academy. I also know how practice, not least of all in industry, falls short of the depth of the theoretical thought.
I hear and participate daily in discussions about automation, computation, programming - languages, environments. I have no illusions about the pinnacle of theory filtering down to how people spend their time, how organizations their resources, or how people talk about these problems every day in the context of real world project which tie up very limited resources pursuing things the theory should have made very clear to be out of reach for several decades now.
Hey! Your comment really made me think about what a language would even look like that was 'complete' rather than 'sound'. It's kind of an interesting problem. The issue is that our formalisms really aren't set up to handle this question "out of the box". I knocked together three different avenues of attack: (1) we could allow a complete but unsound concretization via abstract interpretation (the OG AI!); (2) we could allow complete but unsound extensions of a simply typed lambda calculus by shortcutting closure over µ-recursion, adding in untyped casts, or by unifying the cell model without providing access guarantees (effectless effect systems?); and (3) we could add the "it's about the same" operator to the core Siek-Taha gradual type system. All three results are pretty goofy. I especially like the fact that (2) is shockingly similar to Haskell's coercions and ... just plain old C (with a lot of hand waving).
Gödel incompleteness work has indeed an effect on Lean. Lean cannot prove that the kernel itself is valid; you would need a system "greater" than dependent-type theory, or bigger than Proof Calculus [1] [2], for this.
> It's like Göedel's work hasn't sunk in to our collective reasoning about reasoning.
There's nothing in Göedel's work that warrants any "sinking" or much attention for that matter.
> Namely, that there is no universal, truthful, correct, system of symbolic logic.
Except that "universal", in the sense of Göedel, is nether necessary nor important for anything of practical significance, it's just theorizing for the sake of theorizing. The rest is rock solid.
> like all symbolic logic systems (including the ones that states the limits to universal symbolic logic systems) they are bootstrapped from axioms one must chose at the start.
Of course, it's symbolic systems all the way down, but don't forget all real science without exceptions is based on them, including LLMs and what not. In practice, there's no problem whatsoever, you simply select the one that supports your case the best. Because... what'ya gonna do? There's no alternative (tm)
In Lean, the proof is to the statement, as an object is to a class (in e.g. Python). Lean only compiles your proof if it is a valid "object" of your statement.
So the statement can exist without a proof. Thus, imports that are scoped just for the proof is valid.
> def FermatsLastTheorem : Prop := ∀ a b c n : ℕ, n ≥ 3 → a^n + b^n ≠ c^n
> This code imports Lean’s mathematics library Mathlib and then defines a mathematical statement which looks at first glance like it might be Fermat’s Last Theorem, but which actually is a trivially false statement because in Lean ℕ contains 0, so a = b = c = 0 and n = 3 gives a counterexample.
...
> The first question which one has to ask when judging a pdf which comes with some Lean code is: “what is the statement of the theorem in the Lean code, and does this correspond to the actual theorem statement in the pdf?” For a human Lean expert, this is typically (but not always) an easy question to answer. For example any Lean-competent human mathematician would be able to read the two lines of code above and observe that this was a mis-statement of Fermat’s Last Theorem.
To add to this, if the statement is wrong, the proof steps and other theorem statements that appear in the full proof are going to follow a path that goes with the wrong statement so there's further opportunities to see there's a mistake.
Often problems with the statement make it suspiciously easy to prove, the proof takes a path that doesn't make sense with the correct statement in mind, or it makes it impossible to complete the proof. In the above example, you would see from the easy counterexample found in the proof that the statement is wrong. The proofs aren't a black box.
And if the proof is taking advantage of a bug in Lean somewhere to prove some faulty intermediate theorem, you can still scrutinise the path that takes the proof, the same as you would to find a problem in a flawed hand-written proof. The flaw would ripple through the rest of the proof.
It's like if your high-level requirements for a computer program (the theorem statement) had a mistake and someone implemented it (the proof). The implementation is going to follow a path that implements that mistake. So by looking at individual function definitions that were implemented and what the code is doing, there's opportunities to spot mistakes in the requirements like missing assumptions or parts that don't make sense. This often happens when you're trying to code something based on flawed requirements, where the problem becomes more obvious as you get into the details of the code. It's the same with proofs.
This is exactly one reason why human written mechanized proofs are so much more trust worthy. AI isn't really good at judgment calls, and that includes whether it is mechanizing the right things.
The sizes of the total sum of software that enables you and me to write comments here is mind boggling. Probably no human could understand it all. But then why are we comfortable with it? Abstractions.
The purpose of software isn't primarily to understand computers. I wouldn't know the purpose of say a number theory proof other then gaining a deeper understanding.
The software that enables you and me to write comments here is held together with spit and duck tape (*). For mathematical proofs we should strive for higher standards.
*) no offense intended, don't mean to belittle the software of this particular website in particular, but software in general is written with different aim than perfection.
When I hear this question, I keep thinking of AI proving a consequence of e.g. quantum theories that allows for a new property we wouldn't have thought of. Even if we don't understand the proof yet, the "consequence" might still give rise to experiments to verify if it's valid. That seems like real-life value/purpose to me.
Asimov had this idea back then too. He wrote a short story on how AI could prove FTL travel and design a machine before humans totally understood how all of it worked.
Many mathematicians are now saying that (human!) understanding is (and was) the true goal of pure math. Not proving theorems itself. But I don't think that's correct. Otherwise the mathematicians would be happy about the new math genius AIs (which are infinitely patient in explaining anything), rather than feeling the threat of AIs beating human mathematicians in proving various conjectures.
I rather think pure resesarch math was merely partially about understanding. The other part was a sort of game, where you win prestige if you are the first to prove a famous conjecture, and where you feel like a "researcher", a math scientist of sorts, who "discovers" the secrets of the platonic math universe.
Pure mathematicians are a bit like gamers "exploring" the procedurally generated worlds of Minecraft (or rather, solving procedurally generated puzzles in a puzzle game), while feeling like they thereby make serious scientific discoveries.
The difference is that no actual gamer would delude himself into thinking he is a researcher.
> Otherwise the mathematicians would be happy about the new math genius AIs (which are infinitely patient in explaining anything), rather than feeling the threat of AIs beating human mathematicians in proving various conjectures.
I am unhappy about the “math genius” AIs because they’re part of the ongoing trend of commercializing basic human interaction.
Thanks, this is really interesting and helpful context to us non-mathematicians.
> It is probably worth stressing again: do not run random Lean code on your local machine unless you sandbox it first. Lean’s mathematics library is safe to run and build upon because this is human-curated and checked by human experts.
I was surprised by this - does Lean need the ability to run arbitrary code? It feels like limiting Lean to not being Turing complete would be an advantage.
Lean needs the ability to call out to external solvers and import proof certificates generated by them. In theory you could have Turing-complete Lean without the ability to interact with other programs or the file system, but you would have to fully reimplement those external solvers in Lean and it would be terribly slow.
Leo De Moura's initial goal was for LEAN to be a formal verification tool for programs, before a theorem prover. Theorem proving just was a very convenient side effect, both theoretically (dependent-type theory) and for having mathematicians popularizing Lean.
This is a trivial example, but I asked Prover9 (which is a brute-force automated theorem prover) to prove a theorem, and then I asked GPT-6 Sol to explain the proof. Instead of doing that, it said "You don't need most of this" and gave a shorter proof.
You're implying that GPT-6 Sol can do better. That is not how LLMs work! They will write code, and then, in a separate invocation, find 100 problems with that code. Unlike humans, the immediate context massively impacts the immediate next response, and "knowing better" is nearly entirely irrelevant, because even with thinking and reasoning, these machines do not adequately reflect.
Instructing them to reflect also doesn't help; they pull whatever hallucinations make up the "reflection" into immediate context, and now they're even more confused.
I can believe that GPT provides shorter proof than automated theorem provers which are effectively brute force searching with far less guidance than an LLM.
But from what I’ve seen, it’a absolutely not true that GPT on average generates shorter proofs than humans. Humans have GPT beat handily on that. Sometimes GPT generates shorter proofs, like in cases where humans missed a connection that GPT found. But most of the time, GPT will generate >10X longer proofs, hundreds, even thousands of pages of inscrutable slop maths where it’s completely unclear what’s going on.
I remember some frustrating math courses back when I was studying CS. The experience was somewhat like this:
Imagine you know nothing about vi except that it's a famous command line utility for linux. One day, your supervisor is tasked to give you an introduction to it.
However, he doesn't tell you what vi is for or why it exists or how you use it - instead, he just sends you a tarball with the entire source code of it. Then he spends an hour with you going through the main function line by line, stopping whenever he finds a construct particularly clever or elegant. At the end, he runs "make" once to demonstrate that everything compiles.
He asks if you have any questions. You have none - you're still far away from being able to formulate meaningful questions. He is satisfied, declares you Knowledgeable in Vi and calls it a day.
It feels as if seasoned mathematicians now have a similar experience with AI generated proofs like the one students had in introductory math lectures like this one.
I'm hoping a bit that this could motivate people to develop better didactics for math.
> OpenAI never claimed that it did! They just said “here is some Lean code which goes with this paper”.
Maybe I'm misunderstanding this, but it feels like TFA is giving OpenAI entirely too much credibility here.
If I told you "My grandma makes great chicken enchiladas" and put a plate in front of you containing an enchilada filled with dog crap, you'd be rightfully confused and upset when you bit into it. Sure, I never specified what the enchilada in front of you was filled with, but come on!
> It is worth noting that of the 290 OpenAI theorems which come with Lean code, around 38 of them have the property that the Lean code does not even state, let alone prove, the main theorem of the paper. And OpenAI never claimed that it did! They just said “here is some Lean code which goes with this paper”. Sometimes the code just states and proves some intermediate results — my impression is that OpenAI decided to go live when some of the formalizations were incomplete, so just dumped what they had. To give a concrete example: paper 194 claims to prove Lech’s multiplicity conjecture in commutative algebra, with no restriction on dimension, residue field or characteristic; this question has been open since 1960. The headline theorem in the corresponding Lean code can be checked to be Corollary 5.4 of the paper — already an impressive achievement, but not the headline result.
Both the Lean kernel and the so called independent Con Leche kernel have massive Claude contributions.
Asking the same entity to write the proof checkers as well as the proofs is not good practice. I'm very curious why Buzzard omits this fundamental fact.
This is just like an auditing company also writing the original financial statements.
It all comes back to Searle's Chinese Room thought experiment, or a twist on it. Imagine creating a von Neumann-style self-replicating LLMs that do nothing but replicate and prove theorems. Well, then, what of it? You'd have a universe full of proved math theorems but does it all mean anything at all?
The huge size of the proof is also remarkable, since brute forcing a long sequence of steps is much harder than brute forcing a shorter sequence of steps.
I thought the whole "tactics" approach in Lean was supposed to permit shorter proofs? to skip the low level stuff?
If I formalize my short proof of FLT in MetaMath, it would just be a small file and any computer would be able to verify it.
This reflects so poorly on both OAI as well as the Lean project.
Worth pointing out that this article is by Kevin Buzzard, one of the biggest advocates for Lean and automated theorem proving.
It's refreshing to see intellectual honesty about a tool's limitations, from someone still advocating for its strengths.
In my experience the actual formal verification practitioners are generally very straightforward about its strengths weaknesses and limitations. Understanding those limitations is fundamental to using them well, gaining any real value from them, beyond what is typical of complex tools.
What you need to watch out for are the people or companies who latch on to proof checkers and formal verification as some sort of shortcut to correctness, intending to use it as a signal of seriousness rather than a serious tool in its own right.
I haven’t seen the it yet, but keeping an eye out for “military-grade Lean” or some other nonsense.
Seems more about the limitations of the way it was used
>It is worth noting that of the 290 OpenAI theorems which come with Lean code, around 38 of them have the property that the Lean code does not even state, let alone prove, the main theorem of the paper. And OpenAI never claimed that it did! They just said “here is some Lean code which goes with this paper”. Sometimes the code just states and proves some intermediate results — my impression is that OpenAI decided to go live when some of the formalizations were incomplete, so just dumped what they had.
This kind of skepticism and self criticism is typical of good mathematicians and scientists because it's essential to doing good math and science.
"Worth pointing out that this article is by Kevin Buzzard"
Is it? The site claims that the article is written by the Xena Project. In the comment section it answers on (paraphrasing) "whether proving that a Lean kernel written in Lean and verified by Lean would be a circular proof" with "I’m really sorry, you need to ask these questions elsewhere (e.g. on the Lean Zulip). I am by no means an expert in these matters, all I know is that people who are experts have thought hard about them. "
If that is indeed Kevin Buzzward's writing, it is by no means obvious.
https://xenaproject.wordpress.com/what-is-the-xena-project/
That's a good point, and I would have preferred an explicit byline too. However, I think the following piece of circumstantial evidence points strongly in the direction of it being authored by Kevin:
> When Anthropic formalized a proof of Fermat’s Last Theorem in Lean, they asked me to take a look at the 13 million lines of code which they had generated.
Anthropic asked Kevin specifically, according to the link in that snippet.
He’s currently working on formalising Fermat’s Last Theorem in Lean btw.
Well, he was. Apparently that's one of the routine tests OpenAI and Anthropic use for new models now?
Isn’t “proof is checked in Lean” kinda similar to assuming your code works just because it compiles.
Sure, the code works, but does it work at doing what you want?
Sure, the proof is valid, but does it prove what you set out to prove?
The gist of the article is that it's worse.
That it's the equivalent of saying "It's full moon tonight because I wrote this program".
And he states that you have to check that:
1. The statement the author is making corresponds to the statement in the lean program.
2. The lean program actually compiles.
3. There is no shenanigans hidden in the lean program to disguise a trivial proof as the one stated.
4. The program does not exploit bugs in lean.
On the bright side, the article claims Lean has built-in tools to do 2-4 as long as you have enough RAM. In particular, there's a formally verified alternative implementation of Lean's checker that would theoretically rule out 4.
That implementation seems to be vibe-coded/vibe-proven in Lean itself (assuming the "con-leche" that my quick Google search turned up is the one he's talking about), so it's kind of turtles all the way down, but it does decrease the chance that a program could be exploiting both a Lean bug and a con-leche bug.
Saying that a "formally verified" program rules out any bugs is a bit of a reach. Formal verification only checks that the properties you claim are true are actually true in the absence of bugs in the proof logic.
Example bug
A verified network program ensures that "data from two addresses are never commingled", but it does not verify that the NIC can't overwrite control flow memory opening up RCE attacks/other memory reads.
It certainly is harder to make bugs in formally verified software than the alternative, but it is not bug free.
> but it does not verify that the NIC can't overwrite control flow memory
Yes, Lean verifies the algorithm being verified, not hardware or someone else's software.
It verifies what it verifies, NICs write to main memory as the standard method of reading/writing packets - if there is no proof that the main memory/NIC memory is segmented for all packets/sequences of packets. Then there is no strict reason to believe that the program can't be RCE'd.
The same would apply to any interaction between pointer based memory operations and LEAN. Pointer based memory ops are everywhere below a certain level of code/hardware integration.
> 3. There is no shenanigans hidden in the lean program to disguise a trivial proof as the one stated.
To be clear, this is basically a case of bad defaults in Lean. As Buzzard says, the fix is literally one command.
> The one-line answer: `lake comparator`.
> the program does not exploit bugs in lean
Being fairly naive to Lean in general, this always seemed to me like a clear flaw of using Lean solutions as proof of hypotheses
That’s pretty much the same as compiling a program.
1. Yes.
2. Yes.
3. This is “you took 10 KLOC to do what 1 could have.”
4. Yes.
Reading the friendly article:
It is worth noting that of the 290 OpenAI theorems which come with Lean code, around 38 of them have the property that the Lean code does not even state, let alone prove, the main theorem of the paper.
So even if LEAN is 100% bug-free, its still on the programmer to correctly express the logic and proofs, or they could be proving completely unrelated facts.
It also speaks to why I remain skeptical of relying too much on coding agents for code review. They are still pretty iffy on implementing the right thing, and even worse at detecting when a PR didn’t implement the right thing. Ignoring that reality is how you do like Anthropic did and increase your rate of merging changes from 2500/day to 30,000/day without a corresponding increase in the rate at which you deliver useful work product.
OpenAI did a data dump of work in progress, leaving it to the mathematics community to make sense of it. I don’t think they claimed that there were Lean proofs for everything? Did they warn that some Lean proofs were incomplete?
Why did they stop in the middle anyway? Was there a justification given for that?
The cynical answer is that their primary goal was never to meaningfully contribute to mathematics. It was to pick some low-hanging fruit for the marketing department. So they did the 10% of the work that generates 90% of the buzz.
It’s more likely that their goal was internal research, to test a new LLM before releasing it. But, then they had something that mathematicians would be interested in, so they threw a data dump over the wall. It’s a rather rude way to do it, but this isn’t a company that particularly cares about the things that mathematicians care about.
OpenAI no longer needs to do anything to get publicity. There are some organizations that are so famous that everything they do is newsworthy and anything they do will be covered whether they like it or not. Remaining low-profile is no longer an option.
The people claiming that everything is marketing haven’t come to grips with the inequality of fame.
This would be plausible except for the fact that there was a press release.
There was a blog post. It's the bare minimum when announcing something to the public, not a sign that they're going to use it to promote themselves in a significant way.
There's no "low profile" for a company like OpenAI. Either you let the public (including the press) know or you don't, and keeping the results secret would be bad.
If they just told a few people, it would leak eventually and then you'd have articles based on rumors, along with accusations of a coverup. For an example of the difficulty, look at what happens when people try to do coordinated disclosure of a security bug.
The more likely answer is because that is what the agent decided to do given inconsistent instructions. No one read what it said about what it was doing before releasing the presser.
They could be proving completely unrelated stuff. Or even entirely trivial things like 1 + 1 = 2 over multiple millions of lines of lean code.
Look, don't want to derail the conversation but proving that 1 + 1 = 2 is not trivial. Try it out. Why does 1 + 1 = 2?
succ(0) + succ(0) ?= succ(succ(0))
succ(0) + succ(0) [by `+` def]=> succ(succ(0) + 0) [by `+` def] => succ(succ(0)) QED
I'm sorry, what is that and what does it have to do with 1 + 1 = 2? I don't understand, can you explain?
Isn't that pretty much by definition? 2 is defined as the number that is 1 more than 1.
I'm sorry. Where is that defined?
dictionary.com, among others.
two: a cardinal number, 1 plus 1.
(Yes, I am aware that dictionary.com is not a math textbook. Still, I think it is not mistaken. We define two to be one more than one.)
That would make it an axiom [0], which it often is. But you don't _prove_ axioms.
[0] https://en.wikipedia.org/wiki/Peano_axioms
I think it's a reasonable point but a bad analogy.
It's more like assuming that because your code is formally proven correct the client will be happy with it.
The code can be right but it may still do the wrong thing.
you just said the same thing
OP said it's like saying the code compiles, but I said formally proven correct.
Formally proven correct is a much stronger guarantee that the code meets its spec than a successful compilation.
and op's question was "but how do you know the spec is what you want." making the guarantee stronger doesn't fix the problem. you can't fly by pulling your feet up no matter how strong you are.
That's the point I was agreeing with.
It was the analogy with compilation I didn't think was very good. Lots of code compiles but you have no idea whether it meets any kind of spec, or even if it's bug free..
No, because proofs are programs and theorems are types. It is enough to construct a function of the needed type to have a proof of the needed theorem. You don't need to look through all the code to see that the function has the desired type.
You have to be sure that the formalized theorem matches the informally stated theorem. Which is what GP meant and what the article is about. A program does what is written, not what is meant; a Lean program proves what is written, not what is meant.
I agree with that. I never said you don't have to check the statement.
This sounds interesting enough that I think a link would be useful
Here you go: https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...
See https://news.ycombinator.com/item?id=49944263 and https://news.ycombinator.com/item?id=50043321
That's like saying that all memory-safety programs are written in Rust.
There is no fundamental reason why mathematics should be formalized using concepts that are alien to most mathematicians, in the way that type theory is.
(I guess Isabelle/HOL is the Java of formalized mathematics, in the sense that for reasons that are difficult to understand, memory safety of Java-implemented technology does not count.)
https://mathoverflow.net/questions/376839/what-makes-depende...
For relatively "simple" statements in mathematics it is pretty easy to learn enough Lean to verify that the Lean statement correctly encodes the intention of the mathematics. It gets more difficult in certain areas of mathematics where there are many layers of definitions and they haven't been standardized in Lean yet.
On a technical level, yes. Though lean is build such that "it compiles" is a very strong statement about mathematics while most programming languages are not build to provide a very strong statement about correctness of the program. It is perhaps more similar to checking an equation numerically.
The burden of proof is way higher for lean compared to software. Software only has to work for the end user. A mathematical statement must be exact.
Yes it’s a good analogy.
You have to watch out for
1. Translation. Do the author’s claims about the program/theorem match the actual code?
2. Do you employ “shenanigans” to circumvent the checker? Bugs, unsafe operations. — Lean is a very strict checker as compilers go. Some compilers intentionally say “buyer beware.” It’s annoying to check for integer overflow in every operation. Lean doesn’t. But that still doesn’t mean it has zero bugs, in the framework or axioms.
If the proof is valid, it DOES prove what your theorem is saying (except for rare verification kernel bugs). The issue is mostly: did you correctly formalize and check the specification of your theorem itself...
I mean, in some strange way, yes. Because your type system is basically a set of claims that are proven by your compiler. But the analogy doesn’t go further than that. The running program itself is the proof
it is more like "unit test pass"
Fresh today, I asked llm to fix a unit test for me, it replaced the test with no-op.
One thing I'm surprised to see little mention of here is that if you want to use Lean proofs for your code and really want good correspondence between what is proved in Lean and what your code actually does, you need a mathematical model of your programming language embedded in Lean and a deterministic parser converting your code itself into Lean. Otherwise, yes, an AI actually has no checks against hallucination and while writing Lean proofs probably helps AI identify its mistakes it really isn't an end-to-end proof of your code's functionality.
I personally think the future probably involves a combination of unit tests and formal verification to support our confidence in AI-written software, as well as means to align high-level human intent with the low-level intent of individual code pieces, which as this article mentions is the other big piece that can slip with formal proofs: we might not be proving what we think we're proving. For those interested I've written an educational post about this topic here as well: https://kellerai.io/resources/ai-proved-your-code-correct
I guess it isn't mentioned here because it is completely off-topic. Lean can make formal proofs of code but TFA is about proofs in pure mathematics.
To clarify: mentioned "here" meaning mentioned in the comments. HackerNews is presumably majority programmers and the comments section is mostly comments about using proofs for formal verification of code. I agree the article is coming from a mathematics standpoint
Why would comments on an article about formal proofs about mathematics discuss formal proofs about code?
To be fair, it is surprising when HN comments stay on topic.
Do people really approach Lean proofs from the programming language level rather than assembly code level (x64, arm, etc)?
Yes, Verus (for Rust) is a good example of programming language level verification. The recent GitHub Next announcement indicates they are using it in their backend.
You're right that if you verify at the assembly language it can give you additional certainty that there are no additional errors in the compilation step. However, it can be easier for AI to reason about proofs written at the programming language level because (from what I can deduce) it matches the level that humans are operating at and it doesn't have additional details such as registers and memory operations to worry about.
AWS uses Verus for their verified layers of Nitro, too.
I have just begun working with it, but I prefer the convention of including the formalizations alongside application code, as opposed to maintaining a parallel codebase.
You don't need a model of your programming language in Lean, that's overkill.
It's already useful to isolate a DSL in your software and model that in Lean. Sure that only proves that the domain specific code is correct, but now you've factored your proof into proving high-level correctness and low-level correctness.
What are you proving in Lean if you don't have the semantics of the thing that you're writing a proof about?
You need to pick an abstraction level. I don't prove in lean the impact of my programs on the price of eggs at the local market. I don't prove in lean how much energy my programs consume, or even how much RAM they consume. I do prove that say a particular pair of functions commute, or that another function is idempotent, or that under a denotational semantics a collection of functions can be composed without error.
I think that is a valid level to use formal verification but I would argue it is much more powerful to also have a formal model of code. For example, if I have:
Yes it is very helpful to know some properties about `run_some_function` (maybe it is commutative? idempotent?), but it is much more valuable to have a model which can translate what ultimately is being passed to `run_some_action` at the end and how that derives from the return of `run_some_function`, grounding the LLM's reasoning in something deterministic it can't break from.For Elixir its author, Jose Valim, created Lynx: https://github.com/josevalim/lynx
--- start quote ---
You write laws in Erlang/Elixir, which are automatically translated to Lean alongside your Erlang/Elixir code, and you write proofs in Lean. The laws and proofs are assembled together and verified by Lean's kernel.
This technique is also known as shallow embedding, since it translates Erlang/Elixir to Lean and then use the features and properties of Lean for proofs.
--- end quote ---
I'm a little worried that the mountain of disclaimers against Lean proofs is going to slow something many industries desperately need: more formalization. (Buzzard's article is great by the way considering its target audience—my comment is more about the broader tech industry's perception of LLM-generated Lean proofs.)
One of the nice things about mathematical theories relevant to the real world (e.g. PA) is that confirming that a proof is correct is a purely mechanical procedure. Now Lean is built on the Calculus of Inductive Constructions (which can indirectly capture PA, but includes a lot more as well), but it's child's play to write a simple PA proof checker. It's probably a few hundred lines of Python, and even for something like a million line proof, confirming or refuting whether a proof is correct will take a few seconds tops (again, pure PA here, not Lean).
The caveats in the article are indeed important (e.g., "Did your formal sentence in the language of first-order arithmetic actually capture what you intended?", "Is there a bug in your proof checker?"), but I'd label these more as human factors than math factors. I'm afraid the general public is starting to get the impression that "Lean-checked" is just another quality improvement, like reviewing your code carefully or adding some unit tests. It's not—it's totally different, and I don't think we should make too many scary disclaimers as to dissuade further progress with formalization.
The problem with your PA proposal is that, while the kernel will be much small, the specifications/theorems will be much harder to read. So you are saving on a one time effort (writing a correct kernel) by sacrificing in many time efforts (writing a correct specification).
Pro's and cons. Personally I'm glad to have a short list of additional "check this too" items to improve the confidence in the accuracy of the rigor.
> "Lean-checked" is just another quality improvement, like reviewing your code carefully or adding some unit tests. It's not—it's totally different
I think for most things, this is actually a good way to see it: As a third leg on top of reviewing your code carefully and unit tests.
As you say, people will still make many mistakes in how they write their formal requirements, or miss things they care about.
For math this is not an issue, it's usually easy to check that the lean statement is exactly what you want. But for software, and many other real world applications, it's better thought of as a new orthogonal testing vector that everyone should be using.
> But for software, and many other real world applications, it's better thought of as a new orthogonal testing vector that everyone should be using.
Not sure about should, but as long as even modestly complex systems like the Signal communication protocol map into millions of lines of Lean, its pretty clear that not everyone will be able to use it. We still need better tools (tools better suited to our monkey brains).
I don't think the number of "lines of Lean" should be the issue. That's just logs.
The question is whether it's fast and cheap enough to do.
> Question 4: does the code exploit a bug in Lean’s official kernel?
My understanding is that this has generally been a non issue as one does not exploit a bug unintentionally (at least it hasn't happened yet), but the Huggingface hack shows agents are willing to go through great lengths to cheat to pass evals.
Given an impossible task I could see an agent finding and exploiting a Lean kernel bug and then obfuscating its method.
I suspect the easiest solution to this is to have adversarial agents looking for such cheating.
Already happened. An agent produced a Lean proof of the Collatz conjecture being incorrect by exploiting a bug in Lean: https://leodemoura.github.io/blog/2026-8-1-postmortem-for-ke...
That isn't what the parent comment described at all. In order to qualify, you would have to instruct an agent to prove the collatz conjecture, it find the bug, and then exploit it. What happened here is someone found the kernel bug, and then wrote a cute psuedo-proof with it so they could get a blog post out of it.
If only there was a way to prove the compiler worked exactly as intended in all cases...
https://xkcd.com/1821/
Point 3 in this article basically is just a case of bad defaults in Lean (Lean at this point should probably just make `lake comparator` a mandatory part of compilation and that immediately fixes the issue as Buzzard mentions).
Really there's only three things to check that the theorem is correct (leaving aside important questions of whether humans actually understand the proof and only caring about whether theorem is true):
1. Check that the formal statement of the theorem matches the informal statement of the theorem
2. Actually compile the proof (with `lake comparator`)
3. Try to make the Lean kernel as bug-free as possible (the hardest one but also the one with the most amount of attention focused on it) or run with multiple different kernels
Assuming those three, no need to read the potentially millions of lines of proof, if all you care about is correctness and not understanding.
>Only 3 steps to verify a proof
>step 1, verify the proof
No. Verify the statement not the proof (usually the statement is <<1% the lines of the proof, e.g. in the case of Fermat's Last Theorem it's approximately 0.00001%).
This mistake is amusing because one of the main goals of the article is to explain why the statement is different from the proof of the statement, and why this distinction is important.
The statement is a formal description of the thing you're trying to prove; the proof is the step-by-step reasoning that shows why it's true. Usually the first is much smaller than the second, which is good because it's the thing that you (or some entity you trust) needs to check manually. Checking that the proof proves the statement is what Lean compilation does for you automatically.
Loosely, the statement is like an interface definition (e.g., an abstract base class); the proof is like an implementation of that interface. For sufficiently simple contracts, even regular compiled languages like Java or C++ can check an implementation for "correctness", e.g., that all methods declared in the base class have been defined, that each of them either returns a value of the type declared in its signature or throws, etc. Lean is similar but allows a lot more conditions to be expressed.
> Put another way — given a Challenge.lean file with only trusted code imported and the statement of a theorem, and a Solution.lean file with millions of lines of untrusted code imported and the proof of a theorem which looks identical, how do we know for sure that somewhere in the millions of lines of code there is no hack which changes the meaning of the statement, changes the rules from just the axioms of mathematics.
As someone with a PhD in mathematics, but no experience with Lean, I found this pretty surprising. Shouldn't there be some kind of encapsulation, which insulates the statement from the proof?
I don't understand this at all, as someone with a PhD in this area, and who uses lean quite a bit.
If 'Challenge.lean' states the theorem, the solution can't change the theorem. The only thing it can do is do a 'sorry', just skip proving some part, but that's printed out very clearly when it happens.
They are not talking about changing the theorem. The solution can arbitrarily chose to override or ignore certain things to make sure the theorem passes.
As in the infamous 2=1 (https://en.wikipedia.org/wiki/Mathematical_fallacy#Division_...). We ignore that division by zero is undefined, and boom, we get a "proper" solution to the stated theorem.
That's what the article means by "how do we know for sure that somewhere in the millions of lines of code there is no hack which changes the meaning of the statement, changes the rules from just the axioms of mathematics".
As for the meaning of the statement, I don't know how it is easy in maths. But in code, I've had Claude try and contort the definition of "function accepts integers" to "anything that can look as an integer e.g. a string containing an integer is accepted" and twist the solution to that.
'The solution can arbitrarily chose to override or ignore certain things to make sure the theorem passes.' - No, to be clear, in Lean, it really can't (with the one exception of using 'sorry' to mark some bits as unproved, which is then always flagged clearly).
> No, to be clear, in Lean, it really can't
It can. For two reasons. One reason is that the proofs themselves depend on on unchecked and unverified definitions and statements:
--- start quote ---
An example of this right now is a PR to DeepMind’s Formal Conjectures repository containing a Lean formalisation of the statement of the Hodge conjecture. At the time of writing, the statement relies on over 10,000 lines of code sitting on top of Mathlib, because Mathlib does not yet have the definitions of things like the cycle class map needed to make the statement.
Another example is Axiom’s recent announcement of a formally-verified proof of the classification of finite simple groups. The statement which Axiom’s system proved is here in Tau Ceti, an AI-generated library of Lean code. The file containing the statement imports over 250K lines of Lean code in Tau Ceti (over 900 files). Of these, around 5600 lines of code are crucial because they contain all the relevant definitions
--- end quote ---
And the second reason is why `lake comparator` exists.
I think the deeper problem - an I am reading Computational Reflections by Cantwell Smith and that might be colouring my view - is that fundamentally it comes down to the interpretation of what the code or the math says.
The very meaning of interpretation in computer science is at odds with the use in philosophy of language and that is symptomatic.
We're still fundamentally trying to advance Hilbert's program. It's like Göedel's work hasn't sunk in to our collective reasoning about reasoning. It just doesn't seem possible to build a self contained correct and truthful system of symbolic logic. Truth is in the relationship of the system with other systems.
Automating mathematics, like automating software engineering, or automating automation, are all self contradictory propositions. Anything of what we build only ever means anything in how it interacts with us. I mean, people will go round and round in this forum discussing semantics and the meta implication of these unceasing discussions never seems to hit home. Namely, that there is no universal, truthful, correct, system of symbolic logic. It seems in fact impossible to have one. But the results that indicate this impossibility are built with a system of symbolic logic and the validity therein is contingent on people's understanding and, like all symbolic logic systems (including the ones that states the limits to universal symbolic logic systems) they are bootstrapped from axioms one must chose at the start or by default and they can never be fully proved by the very system built atop them.
As an ex- mathematician and ex- computer scientist who worked in programming language theory and computer algebra systems, I can assure you that we think about these problems deeply. In fact, outside of a few strongly normalizing toys, the first thing we do in PLT is drop completeness in favor of soundness. The article itself discusses this (the soundness issues that were addressed).
Although, now I'm charmed about the idea of a programming language -- possibly a Brainfuck descendent? -- that drops soundness for completeness.
Oh I know how deeply we think about these things in the academy. I also know how practice, not least of all in industry, falls short of the depth of the theoretical thought.
I hear and participate daily in discussions about automation, computation, programming - languages, environments. I have no illusions about the pinnacle of theory filtering down to how people spend their time, how organizations their resources, or how people talk about these problems every day in the context of real world project which tie up very limited resources pursuing things the theory should have made very clear to be out of reach for several decades now.
Hey! Your comment really made me think about what a language would even look like that was 'complete' rather than 'sound'. It's kind of an interesting problem. The issue is that our formalisms really aren't set up to handle this question "out of the box". I knocked together three different avenues of attack: (1) we could allow a complete but unsound concretization via abstract interpretation (the OG AI!); (2) we could allow complete but unsound extensions of a simply typed lambda calculus by shortcutting closure over µ-recursion, adding in untyped casts, or by unifying the cell model without providing access guarantees (effectless effect systems?); and (3) we could add the "it's about the same" operator to the core Siek-Taha gradual type system. All three results are pretty goofy. I especially like the fact that (2) is shockingly similar to Haskell's coercions and ... just plain old C (with a lot of hand waving).
Gödel incompleteness work has indeed an effect on Lean. Lean cannot prove that the kernel itself is valid; you would need a system "greater" than dependent-type theory, or bigger than Proof Calculus [1] [2], for this.
[1] https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...
[2] https://en.wikipedia.org/wiki/Proof_calculus
> It's like Göedel's work hasn't sunk in to our collective reasoning about reasoning.
There's nothing in Göedel's work that warrants any "sinking" or much attention for that matter.
> Namely, that there is no universal, truthful, correct, system of symbolic logic.
Except that "universal", in the sense of Göedel, is nether necessary nor important for anything of practical significance, it's just theorizing for the sake of theorizing. The rest is rock solid.
> like all symbolic logic systems (including the ones that states the limits to universal symbolic logic systems) they are bootstrapped from axioms one must chose at the start.
Of course, it's symbolic systems all the way down, but don't forget all real science without exceptions is based on them, including LLMs and what not. In practice, there's no problem whatsoever, you simply select the one that supports your case the best. Because... what'ya gonna do? There's no alternative (tm)
> self contained
This is a totally unnecessary requirement, noticed by Alfred Tarski roughly two seconds after Göedel's work was announced.
If you don't personally find formal reasoning useful in proving correctness, that's a you problem. The rest of us are progressing forward just fine.
In Lean, the proof is to the statement, as an object is to a class (in e.g. Python). Lean only compiles your proof if it is a valid "object" of your statement.
So the statement can exist without a proof. Thus, imports that are scoped just for the proof is valid.
> def FermatsLastTheorem : Prop := ∀ a b c n : ℕ, n ≥ 3 → a^n + b^n ≠ c^n
> This code imports Lean’s mathematics library Mathlib and then defines a mathematical statement which looks at first glance like it might be Fermat’s Last Theorem, but which actually is a trivially false statement because in Lean ℕ contains 0, so a = b = c = 0 and n = 3 gives a counterexample.
...
> The first question which one has to ask when judging a pdf which comes with some Lean code is: “what is the statement of the theorem in the Lean code, and does this correspond to the actual theorem statement in the pdf?” For a human Lean expert, this is typically (but not always) an easy question to answer. For example any Lean-competent human mathematician would be able to read the two lines of code above and observe that this was a mis-statement of Fermat’s Last Theorem.
To add to this, if the statement is wrong, the proof steps and other theorem statements that appear in the full proof are going to follow a path that goes with the wrong statement so there's further opportunities to see there's a mistake.
Often problems with the statement make it suspiciously easy to prove, the proof takes a path that doesn't make sense with the correct statement in mind, or it makes it impossible to complete the proof. In the above example, you would see from the easy counterexample found in the proof that the statement is wrong. The proofs aren't a black box.
And if the proof is taking advantage of a bug in Lean somewhere to prove some faulty intermediate theorem, you can still scrutinise the path that takes the proof, the same as you would to find a problem in a flawed hand-written proof. The flaw would ripple through the rest of the proof.
It's like if your high-level requirements for a computer program (the theorem statement) had a mistake and someone implemented it (the proof). The implementation is going to follow a path that implements that mistake. So by looking at individual function definitions that were implemented and what the code is doing, there's opportunities to spot mistakes in the requirements like missing assumptions or parts that don't make sense. This often happens when you're trying to code something based on flawed requirements, where the problem becomes more obvious as you get into the details of the code. It's the same with proofs.
This is exactly one reason why human written mechanized proofs are so much more trust worthy. AI isn't really good at judgment calls, and that includes whether it is mechanizing the right things.
“Beware of bugs in the above code; I have only proved it correct, not tried it." - Donald Knuth
The sizes of these proofs are mind boggling.
This raises the question, is there any value in a proof so complicated that no human could ever understand it?
Is there any purpose to mathematical proofs other than to increase our understanding?
The sizes of the total sum of software that enables you and me to write comments here is mind boggling. Probably no human could understand it all. But then why are we comfortable with it? Abstractions.
The purpose of software isn't primarily to understand computers. I wouldn't know the purpose of say a number theory proof other then gaining a deeper understanding.
The software that enables you and me to write comments here is held together with spit and duck tape (*). For mathematical proofs we should strive for higher standards.
*) no offense intended, don't mean to belittle the software of this particular website in particular, but software in general is written with different aim than perfection.
When I hear this question, I keep thinking of AI proving a consequence of e.g. quantum theories that allows for a new property we wouldn't have thought of. Even if we don't understand the proof yet, the "consequence" might still give rise to experiments to verify if it's valid. That seems like real-life value/purpose to me.
Asimov had this idea back then too. He wrote a short story on how AI could prove FTL travel and design a machine before humans totally understood how all of it worked.
Formal verification of software
> https://terrytao.wordpress.com/wp-content/uploads/2026/09/cr...
Many mathematicians are now saying that (human!) understanding is (and was) the true goal of pure math. Not proving theorems itself. But I don't think that's correct. Otherwise the mathematicians would be happy about the new math genius AIs (which are infinitely patient in explaining anything), rather than feeling the threat of AIs beating human mathematicians in proving various conjectures.
I rather think pure resesarch math was merely partially about understanding. The other part was a sort of game, where you win prestige if you are the first to prove a famous conjecture, and where you feel like a "researcher", a math scientist of sorts, who "discovers" the secrets of the platonic math universe.
Pure mathematicians are a bit like gamers "exploring" the procedurally generated worlds of Minecraft (or rather, solving procedurally generated puzzles in a puzzle game), while feeling like they thereby make serious scientific discoveries.
The difference is that no actual gamer would delude himself into thinking he is a researcher.
> Otherwise the mathematicians would be happy about the new math genius AIs (which are infinitely patient in explaining anything), rather than feeling the threat of AIs beating human mathematicians in proving various conjectures.
I am unhappy about the “math genius” AIs because they’re part of the ongoing trend of commercializing basic human interaction.
That's a more general LLM issue independent of math.
Thanks, this is really interesting and helpful context to us non-mathematicians.
> It is probably worth stressing again: do not run random Lean code on your local machine unless you sandbox it first. Lean’s mathematics library is safe to run and build upon because this is human-curated and checked by human experts.
I was surprised by this - does Lean need the ability to run arbitrary code? It feels like limiting Lean to not being Turing complete would be an advantage.
Lean needs the ability to call out to external solvers and import proof certificates generated by them. In theory you could have Turing-complete Lean without the ability to interact with other programs or the file system, but you would have to fully reimplement those external solvers in Lean and it would be terribly slow.
Leo De Moura's initial goal was for LEAN to be a formal verification tool for programs, before a theorem prover. Theorem proving just was a very convenient side effect, both theoretically (dependent-type theory) and for having mathematicians popularizing Lean.
It does. What if you want to use Lean to write say an HTTP server that contains several theorems about how it behaves?
(Judging from my very limited experience with Lean, and with Lean written by LLMs)
It also doesn't help that LLMs tend to generate a lot of unnecessary Lean code to prove even the simplest statements.
Proofs are really hard, good proofs are really really hard. Having any proof at all is a huge step forward from not having one.
2/5
This is a trivial example, but I asked Prover9 (which is a brute-force automated theorem prover) to prove a theorem, and then I asked GPT-6 Sol to explain the proof. Instead of doing that, it said "You don't need most of this" and gave a shorter proof.
You're implying that GPT-6 Sol can do better. That is not how LLMs work! They will write code, and then, in a separate invocation, find 100 problems with that code. Unlike humans, the immediate context massively impacts the immediate next response, and "knowing better" is nearly entirely irrelevant, because even with thinking and reasoning, these machines do not adequately reflect.
Instructing them to reflect also doesn't help; they pull whatever hallucinations make up the "reflection" into immediate context, and now they're even more confused.
I can believe that GPT provides shorter proof than automated theorem provers which are effectively brute force searching with far less guidance than an LLM.
But from what I’ve seen, it’a absolutely not true that GPT on average generates shorter proofs than humans. Humans have GPT beat handily on that. Sometimes GPT generates shorter proofs, like in cases where humans missed a connection that GPT found. But most of the time, GPT will generate >10X longer proofs, hundreds, even thousands of pages of inscrutable slop maths where it’s completely unclear what’s going on.
Funny how they always seem to do great on a trivial example. But not too trivial that it isn't worth sharing... Just trivial enough.
The problem here is that people want a simple fact to deal with many situations.
A) "Someone said Lean4 proof so it must be bullet-proof and fulfill all my hopes!" B) "Eh, lean4 is imperfect, so it means nothing"
Inconveniently, both of those caricatures are wrong.
Lean4 gives us a high level of confidence in the _theorem validated, under the assumptions allowed_.
This makes it high value to learn to read lean4 theorems (not proofs).
I remember some frustrating math courses back when I was studying CS. The experience was somewhat like this:
Imagine you know nothing about vi except that it's a famous command line utility for linux. One day, your supervisor is tasked to give you an introduction to it.
However, he doesn't tell you what vi is for or why it exists or how you use it - instead, he just sends you a tarball with the entire source code of it. Then he spends an hour with you going through the main function line by line, stopping whenever he finds a construct particularly clever or elegant. At the end, he runs "make" once to demonstrate that everything compiles.
He asks if you have any questions. You have none - you're still far away from being able to formulate meaningful questions. He is satisfied, declares you Knowledgeable in Vi and calls it a day.
It feels as if seasoned mathematicians now have a similar experience with AI generated proofs like the one students had in introductory math lectures like this one.
I'm hoping a bit that this could motivate people to develop better didactics for math.
> OpenAI never claimed that it did! They just said “here is some Lean code which goes with this paper”.
Maybe I'm misunderstanding this, but it feels like TFA is giving OpenAI entirely too much credibility here.
If I told you "My grandma makes great chicken enchiladas" and put a plate in front of you containing an enchilada filled with dog crap, you'd be rightfully confused and upset when you bit into it. Sure, I never specified what the enchilada in front of you was filled with, but come on!
What did OpenAI actually claim?
> It is worth noting that of the 290 OpenAI theorems which come with Lean code, around 38 of them have the property that the Lean code does not even state, let alone prove, the main theorem of the paper. And OpenAI never claimed that it did! They just said “here is some Lean code which goes with this paper”. Sometimes the code just states and proves some intermediate results — my impression is that OpenAI decided to go live when some of the formalizations were incomplete, so just dumped what they had. To give a concrete example: paper 194 claims to prove Lech’s multiplicity conjecture in commutative algebra, with no restriction on dimension, residue field or characteristic; this question has been open since 1960. The headline theorem in the corresponding Lean code can be checked to be Corollary 5.4 of the paper — already an impressive achievement, but not the headline result.
Both the Lean kernel and the so called independent Con Leche kernel have massive Claude contributions.
Asking the same entity to write the proof checkers as well as the proofs is not good practice. I'm very curious why Buzzard omits this fundamental fact.
This is just like an auditing company also writing the original financial statements.
Con Leche, okay.
But prove to me that the main Lean 4 kernel is Claude written, because I think it's not.
The issues in the article sound eerily familiar to anyone who has seen the passing unit tests that LLMs love to generate.
He doesn't tell it to make no mistakes. How can I trust this article!
https://xkcd.com/3201/ "Proof Without Content"
I am not looking forward to bugs in code becoming bugs in specifications instead.
It is far more pleasant to deal with bugs in code.
For complex things like compilers, we like to deliberately leave some parts of the spec weak to make the code pragmatic. Welp.
..."left as an exercise to the reader".
It all comes back to Searle's Chinese Room thought experiment, or a twist on it. Imagine creating a von Neumann-style self-replicating LLMs that do nothing but replicate and prove theorems. Well, then, what of it? You'd have a universe full of proved math theorems but does it all mean anything at all?
I have a concise proof of FLT.
96 cores and 500 GB RAM to ... verify FLT?
The huge size of the proof is also remarkable, since brute forcing a long sequence of steps is much harder than brute forcing a shorter sequence of steps.
I thought the whole "tactics" approach in Lean was supposed to permit shorter proofs? to skip the low level stuff?
If I formalize my short proof of FLT in MetaMath, it would just be a small file and any computer would be able to verify it.
This reflects so poorly on both OAI as well as the Lean project.
Would you be willing to share your concise proof?
Sure, in a few weeks or months I will publish it.
Just subscribe to the metamath mailing list and you will see it pass by when it does.
The margin is too narrow to contain it.
At this point I've seen so many Lean articles that I have to ask: who check that Lean is actually correct itself?
The article mentions that there are a number of independent lean kernels.
Thank you, I should have actually gone through the whole article before commenting.