> Human brains do this by deep double descent-style overparameterization, and adopting a scaling strategy of extremely high-learning-rate training of extremely overparameterized models on small diverse highly-filtered datasets.
That’s an extremely steep claim with no source other than vibes. Last time I checked my biology notes, model parameters are neurons, and they cost a ton of energy to maintain. Your hypothesis is really far removed from any actual neuroscience. Also, where are those filtered datasets coming from? Do you think genetics hands them to us? There’s about zero evidence for this claim as well.
I like new concepts for ML research but please do not make up theories of human cognition when you clearly have no idea about it.
We have a lot of synapses, but (agreeing with you) I don't find that sufficient to explain why humans (or animals!) do what we do. If you throw zillions of parameters at a problem with a weak architecture, you get really high-fidelity memorization, and we're not awesome at memorization compared to machines.
Humans can do an impressive amount of generalization from one error or surprise, and as is often rightly noted, don't need trillions of words to get going. And it all seems to happen some 'forward-only' way, without backpropagation -- we don't have AdamW or MuonClip helpfully nudging our synaptic connections towards whatever would have scored well on our most recent test. It is relevant that we're creatures with goals -- reinforcement learning is the only stage where there's a taste of that for neural nets -- but the learning differences seem at least partly independent of that.
I suppose it could turn out that, even if not sufficient, the large number of synapses is necessary to all this, like we're effectively buying a lot of lottery tickets that give us a shot at fishing interesting hypotheses out of the experiences flowing by. But I'm still awfully suspicious that we don't have the right mathematical model for learning messy ideas all worked out yet.
> Human brains do this by deep double descent-style overparameterization, and adopting a scaling strategy of extremely high-learning-rate training of extremely overparameterized models on small diverse highly-filtered datasets.
That’s an extremely steep claim with no source other than vibes. Last time I checked my biology notes, model parameters are neurons, and they cost a ton of energy to maintain. Your hypothesis is really far removed from any actual neuroscience. Also, where are those filtered datasets coming from? Do you think genetics hands them to us? There’s about zero evidence for this claim as well. I like new concepts for ML research but please do not make up theories of human cognition when you clearly have no idea about it.
We have a lot of synapses, but (agreeing with you) I don't find that sufficient to explain why humans (or animals!) do what we do. If you throw zillions of parameters at a problem with a weak architecture, you get really high-fidelity memorization, and we're not awesome at memorization compared to machines.
Humans can do an impressive amount of generalization from one error or surprise, and as is often rightly noted, don't need trillions of words to get going. And it all seems to happen some 'forward-only' way, without backpropagation -- we don't have AdamW or MuonClip helpfully nudging our synaptic connections towards whatever would have scored well on our most recent test. It is relevant that we're creatures with goals -- reinforcement learning is the only stage where there's a taste of that for neural nets -- but the learning differences seem at least partly independent of that.
I suppose it could turn out that, even if not sufficient, the large number of synapses is necessary to all this, like we're effectively buying a lot of lottery tickets that give us a shot at fishing interesting hypotheses out of the experiences flowing by. But I'm still awfully suspicious that we don't have the right mathematical model for learning messy ideas all worked out yet.
> Speculative proposal
I guess at least they're honest about it? lol