Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.
Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice.
Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.
Take yourself out of that loop so you only find out once it's sure.
Love LLMs gonna keep using them. It feels like your suggested approach is expensive, in terms of tokens. I feel (second time I say this) that when I steer the process I get pretty good results vs my coworkers that let the LLMs run away. I do have data on our token usage, not much in terms of quality of the delivery.
I keep thinking about the c compiler implementation that anthropic shared earlier in the year that had all the requirements you mention and arguably wasn’t that great.
Thr thing is that both you and your agent should have a way to verify the solution.
OBVIOUSLY, the compiler experiment was just a cringe pr stunt. But it has a point: everything works better with a good testing loop, and compilers always have one by thr nature of the work they do
I think an interesting direction for benchmarking is to take inspiration from metamorphic testing. Metamorphic testing is a way of extending property-based testing (in which you ask the test framework itself to automatically generate many random (input, expected output) pairs to test for you, instead of manually writing individual tests yourself) to handle situations where (a) it's hard to independently come up with the right answer for a specific given input, but (b) relationships between inputs imply checkable relationships between outputs. For example, if you're trying to test your own implementation of sin(), it's hard to automatically generate random (input, expected output) test pairs without using a separate, trusted implementation of the sine function, which may not be available; but one thing you can easily do is check, for many different random x, that sin(x) == -sin(x+180).
How to apply this idea to benchmarks? Basically, look for simple transformations of the input instances that should yield simple transformations of the outputs -- in particular, outputs that, in a non-overfitted implementation, should take the same length of time to compute. For regexes, you could rotate a subset of non-magic characters in both the string and the regex (e.g., A -> B, B -> C, ..., Z -> A).
Another example would be to reverse both the string and the regex (taking care to handle parenthesised regex subexpressions correctly) -- unlike the previous one, it's not expected that the transformed instance will take the exact same length of time, but there should not be too much blow up.
Unfortunately, even a holdout set doesn’t protect you from overfitting, it just takes longer.
Of course having a holdout set is better than not having one. It’s just not a silver bullet.
Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.
Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice.
Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.
Take yourself out of that loop so you only find out once it's sure.
Love LLMs gonna keep using them. It feels like your suggested approach is expensive, in terms of tokens. I feel (second time I say this) that when I steer the process I get pretty good results vs my coworkers that let the LLMs run away. I do have data on our token usage, not much in terms of quality of the delivery.
I keep thinking about the c compiler implementation that anthropic shared earlier in the year that had all the requirements you mention and arguably wasn’t that great.
Thr thing is that both you and your agent should have a way to verify the solution.
OBVIOUSLY, the compiler experiment was just a cringe pr stunt. But it has a point: everything works better with a good testing loop, and compilers always have one by thr nature of the work they do
> expensive, in terms of tokens.
No amount of tokens can come close to my hourly rate.
I've gone in loops where it'll tell me twenty times in a row that it's definitely found the root cause of a bug, when in fact it definitely did not.
I think an interesting direction for benchmarking is to take inspiration from metamorphic testing. Metamorphic testing is a way of extending property-based testing (in which you ask the test framework itself to automatically generate many random (input, expected output) pairs to test for you, instead of manually writing individual tests yourself) to handle situations where (a) it's hard to independently come up with the right answer for a specific given input, but (b) relationships between inputs imply checkable relationships between outputs. For example, if you're trying to test your own implementation of sin(), it's hard to automatically generate random (input, expected output) test pairs without using a separate, trusted implementation of the sine function, which may not be available; but one thing you can easily do is check, for many different random x, that sin(x) == -sin(x+180).
How to apply this idea to benchmarks? Basically, look for simple transformations of the input instances that should yield simple transformations of the outputs -- in particular, outputs that, in a non-overfitted implementation, should take the same length of time to compute. For regexes, you could rotate a subset of non-magic characters in both the string and the regex (e.g., A -> B, B -> C, ..., Z -> A).
Another example would be to reverse both the string and the regex (taking care to handle parenthesised regex subexpressions correctly) -- unlike the previous one, it's not expected that the transformed instance will take the exact same length of time, but there should not be too much blow up.