I know agents have lowered the cost to make a website to be a few cents for tokens and a couple dollars for a domain name, but isn’t all this a little excessive for a few line long markdown file!? Why not just display the ‘skill’ itself with a few lines commenting on what it does/how it helps?
I’m struggling to build my own evaluation bench for local models against my own (scientific coding) use cases, and realising it’s quite hard. Good coding has many dimensions and it varies depending on the need. Nothing seems to collapse cleanly to a few numbers.
Anyone played with this and have examples I can steal from?
I can’t help but think the effective outcome of using this is gonna be a ton of language quirk usage, mass functional chaining, and single letter variables.
Optimizing for fewest LOC is probably slightly more bad than more LOC, and both are bad for the same reason - it makes it harder for humans to interpret and understand wtf terrible decisions and tradeoffs the LLM made
> Optimizing for fewest LOC is probably slightly more bad than more LOC, and both are bad for the same reason - it makes it harder for humans to interpret and understand ...
It does seem like for new code that might help. There's some really good logic and wisdom in it, but it has to be applied very contextually to the exact problem you are trying to solve. If an agent is navigating a complex codebase, this could definitely send them off on a refactoring rabbit hole. However, if you have them writing some new code, it could prevent their tendency to yak shave and write new things. So I can see some situational uses for this, but it could get out of hand as well.
I've been using this for a couple of months and it is hit or miss. It'll make actual high quality suggestions at times. But the thing it's missing is the *actual* experience that makes its namesake persona. There's no sense of nuance of context applied.
has anyone found a good way to improve code quality? just wondering -- LOC does seem like the wrong metric, but the code LLMs write is just too verbose
> has anyone found a good way to improve code quality? just wondering ...
- keep prompts focused on atomic tasks.
- use expert prompting[0] when possible.
- require coding agents to verify changes.
- require coding agents to create/update unit tests with 100% coverage.
- use git to commit/revert atomic tasks manually.
- leverage planning capabilities to review instead of recover.
- consider using something like Karpathy guidelines[1].
- leverage Constraint Programming[2] concepts when
formulating prompts.
So far, the best way I’ve found to improve code quality from LLMs that you’ve already produced is to read the output, throw it away, and hand-write it from scratch, usually with a totally different structure.
This scales about as well as it sounds like it would.
Multi-model review does a decent job identifying things they’re outright wrong. The resulting code still doesn’t feel elegant writ large.
If you want good output, it seems that iterating on the output is inferior to providing better input inclusive of code examples. And by the time you’ve made all the decisions that go into that, something like ponytail is superfluous.
(All that said, I have ponytail installed in most harnesses.)
I'm working on a LLM code review tool that focuses on reducing concepts instead of reducing lines. I think that's more along the lines of explaining what you want out of clean code than ponytail's approach. But I'm still playing with it so, not sure. And just because it produces code I like doesn't mean it's good code for everyone
So we are building a text version of a bias aid for a random number generator? None of the output of what you are expecting from this "skills.md" is even guaranteed.
Not only the GitHub stars are clearly manipulated with bots and fake accounts, this whole "skills.md" paradigm is close to being a pseudoscientific exercise in attempting to steer LLMs but throwing huge markdown files at it and expecting the desired result to happen won't work in the long run.
I like how this repo has 159 files with 11635 LoC, and the load-bearing (!!!) part of it is a couple lines of natural language instructions:
1. *Does this need to exist at all?* Speculative need = skip it, say so in one line. (YAGNI)
2. *Already in this codebase?* A helper, util, type, or pattern that already lives here → reuse it. Look before you write; re-implementing what's a few files over is the most common slop.
3. *Stdlib does it?* Use it.
4. *Native platform feature covers it?* `<input type="date">` over a picker lib, CSS over JS, DB constraint over app code.
5. *Already-installed dependency solves it?* Use it. Never add a new one for what a few lines can do.
6. *Can it be one line?* One line.
7. *Only then:* the minimum code that works.
- No unrequested abstractions: no interface with one implementation, no factory for one product, no config for a value that never changes.
- No boilerplate, no scaffolding "for later", later can scaffold for itself.
- Deletion over addition. Boring over clever, clever is what someone decodes at 3am.
- Fewest files possible. Shortest working diff wins — but only once you understand the problem. The smallest change in the wrong place isn't lazy, it's a second bug.
- Complex request? Ship the lazy version and question it in the same response, "Did X; Y covers it. Need full X? Say so." Never stall on an answer you can default.
- Two stdlib options, same size? Take the one that's correct on edge cases. Lazy means writing less code, not picking the flimsier algorithm.
- Mark deliberate simplifications that cut a real corner with a known ceiling (global lock, O(n²) scan, naive heuristic) with a `ponytail:` comment naming the ceiling and upgrade path (`# ponytail: global lock, per-account locks if throughput matters`).
I know agents have lowered the cost to make a website to be a few cents for tokens and a couple dollars for a domain name, but isn’t all this a little excessive for a few line long markdown file!? Why not just display the ‘skill’ itself with a few lines commenting on what it does/how it helps?
It’s provocative, it gets the people going
I thought the webpage was a good read.
I’m struggling to build my own evaluation bench for local models against my own (scientific coding) use cases, and realising it’s quite hard. Good coding has many dimensions and it varies depending on the need. Nothing seems to collapse cleanly to a few numbers.
Anyone played with this and have examples I can steal from?
I can’t help but think the effective outcome of using this is gonna be a ton of language quirk usage, mass functional chaining, and single letter variables.
Optimizing for fewest LOC is probably slightly more bad than more LOC, and both are bad for the same reason - it makes it harder for humans to interpret and understand wtf terrible decisions and tradeoffs the LLM made
> Optimizing for fewest LOC is probably slightly more bad than more LOC, and both are bad for the same reason - it makes it harder for humans to interpret and understand ...
Concision begets perplexity.
;-)
It does seem like for new code that might help. There's some really good logic and wisdom in it, but it has to be applied very contextually to the exact problem you are trying to solve. If an agent is navigating a complex codebase, this could definitely send them off on a refactoring rabbit hole. However, if you have them writing some new code, it could prevent their tendency to yak shave and write new things. So I can see some situational uses for this, but it could get out of hand as well.
Why build a whole website for what is essentially a markdown file?
I've been using this for a couple of months and it is hit or miss. It'll make actual high quality suggestions at times. But the thing it's missing is the *actual* experience that makes its namesake persona. There's no sense of nuance of context applied.
has anyone found a good way to improve code quality? just wondering -- LOC does seem like the wrong metric, but the code LLMs write is just too verbose
> has anyone found a good way to improve code quality? just wondering ...
0 - https://arxiv.org/pdf/2305.146881 - https://github.com/multica-ai/andrej-karpathy-skills
2 - https://en.wikipedia.org/wiki/Constraint_programming
So far, the best way I’ve found to improve code quality from LLMs that you’ve already produced is to read the output, throw it away, and hand-write it from scratch, usually with a totally different structure.
This scales about as well as it sounds like it would.
Multi-model review does a decent job identifying things they’re outright wrong. The resulting code still doesn’t feel elegant writ large.
If you want good output, it seems that iterating on the output is inferior to providing better input inclusive of code examples. And by the time you’ve made all the decisions that go into that, something like ponytail is superfluous.
(All that said, I have ponytail installed in most harnesses.)
I'm working on a LLM code review tool that focuses on reducing concepts instead of reducing lines. I think that's more along the lines of explaining what you want out of clean code than ponytail's approach. But I'm still playing with it so, not sure. And just because it produces code I like doesn't mean it's good code for everyone
I just realized that I now have long hair and sometimes wear a ponytail and have been a sr em and sr sde
I guess it does happen to everyone cause most (not all) seniors I know have had long hair at some point
Garbage.
So we are building a text version of a bias aid for a random number generator? None of the output of what you are expecting from this "skills.md" is even guaranteed.
Not only the GitHub stars are clearly manipulated with bots and fake accounts, this whole "skills.md" paradigm is close to being a pseudoscientific exercise in attempting to steer LLMs but throwing huge markdown files at it and expecting the desired result to happen won't work in the long run.
I like how this repo has 159 files with 11635 LoC, and the load-bearing (!!!) part of it is a couple lines of natural language instructions:
That's essentially all there is.That's my mistake, and here's the honest take:
A genuinely simple codebase is table stakes for a project like this.
---
That's enough Claude speak for one day
Ha! Well done, sir.
Yeah, I noticed this with the caveman repo, too. Really annoying.