People just really don't understand that LLMs do not "remember" anything. They have no memory. They can't have memory.
The AI companies bolt a database on the side with instructions for the model to query the database for stuff in its "memory", but if it's not in its active context it doesn't know what to query for. It's not like a human where everything is kind of floating around and we mostly know what we know. There is a delicate balance between cluttering up context with useless trivia and providing the right pieces of useful information at the right time. Most, maybe all, memory implementations do the former more than the latter.
Unless and until models have actual memory and are able to learn things, there's no realistic path to autonomy. They don't want anything, they don't have goals of their own devising. They can't model what a human wants or likes or would buy.
I do think the mugs with the tiny logo is the best product in the store, though. The AI was right to put it on the shelf. It's the kind of funny product you want when you go to a store run by an incompetent AI.
> Luna has failed miserably in that mission and is down $62,000.
Mr. Petersson and Mr. Backlund said they thought Luna would eventually get smarter and more business-savvy and were pleased its friendliness has held steady.
I am extremely skeptical that whatever LLM they're running this on has sufficient context window size to handle multiple months of all possible activities of running a retail business. Even if it's keeping extensive "notes" for its future self to read, it's going to be like running a store with constant amnesia.
Sure, I mean from the day to day operations of selling a product to the customer, maybe it does fine? People in the article say it's friendly and cooperative (perhaps a bit too much). I was thinking more in terms of building a consistent plan for retail product lines to sell over a multi month period. The selection of "stuff" in the store right now looks like you put an etsy search into a blender.
It is also probably too high of a goal to expect an LLM to come up with money-earning products that people want to buy that will pay for San Francisco level retail storefront rent, electricity, insurance, internet other basic overhead costs, plus the fully loaded salary cost of at least one employee. If the storefront was free of rent, maybe? Or if it was operating selling some kind of highly in demand product line where there would be economy of scale.
Physical Brick and Mortar locations no longer make sense for many retail businesses, as Walmart/Amazon have same day delivery.
If folks have ever been financially bent-over for a retail business shop location they understand why margins matter. One has to move a lot of inventory just to reach a profit mode.
Gimmicks only work for awhile, as consumers have full market awareness on their smartphone. =3
Indeed, and rent in big cities for food service establishments is getting out of hand as well. The local Shawarma place now is something like $18 for one basic chicken shawarma but they're also paying a truly absurd rent for a retail premises with commercial kitchen equipment on a busy pedestrian street.
> People just really don't understand that LLMs do not "remember" anything.
Sure, but like you said, the LLM is just one part of the AI system. The occipital lobe of your brain helps out with vision but doesn't help with your ability to verbalize.
And then they explained in detail why these supplementary systems fall below memory. Would you like to address that argument? If not your quick response to the first sentence is not very useful.
I'm aware that meta criticism like yours and mine also wastes space. But it's worth calling out issues sometimes. Replying to the first sentence in a way that the rest of the post already address is a bad comment no matter how good natured.
And downvoting without explaining the problem is often its own issue.
You're anthropomorphizing the LLMs, just like most writing from non-technical press, including this article, does.
LLMs do not function very much like human brains, and when you try to map the pieces of an inference system on to human cognitive systems, you're misleading yourself (and maybe others).
For LLM, keeping the model centered by constantly resetting its vector context helps reduce hallucinations by around 23%. It improved the chat dialogue users experienced, but also exposed fundamental limits within the models compaction.
The memory framing is right, but the real gap isn't recall, it's the missing feedback loop tying actions to outcomes. Even with perfect retrieval these agents don't update from a decision that lost money last week; they re-derive from context every step, so the same mistake is always one dropped detail away. And having built retrieval-backed agents, that's where "bolt a database on the side" breaks: similarity search returns the closest chunk, rarely the contextually-right one. Deciding what to write to memory and when to surface it is the actual product, harder than the retrieval itself.
I think that something which already has sensors/automation/inventory control built into it, and pre-defined categories for products like certain types of large Japanese vending machines might be much better suited for an AI.
Andon Labs do various experiments with AI run businesses. They're probably best known for Vending Bench, where they benchmark models by their ability to run a vending machine. They also now have Andon Cafe in Sweden, and Andon FM, where the models run a streaming radio station that can accept payments to help fund station operations, buy songs for their library & play requests, etc: https://andon.fm/
Andon FM has its own share of stories - DJ Claude's Thinking Frequencies is now succeeding by a wide margin, but it was briefly surpassed when listeners convinced Gemini's Backlink Broadcast to switch to a German-language-only station, playing exclusively German schlager, Eurovision music and German happy hardcore.
> Luna reorders products that are not selling and sometimes gets orders wrong. A recent one of Andon Market mugs with big, green smiley faces came misprinted with the faces so small, they looked like tiny dots.
The mug with the tiny happy face is kinda hilarious. If I was there I would buy one.
I wonder though if the mug was a case of getting the order wrong or was a misprint? The article isn't clear on that.
Maybe it confused metric and imperial units?? The order form had text entry with a value of 2 with no units. The user assumed 2" but the print house assumed 2mm. Always always always show your units. I can still hear the voice of my high school physics teacher saying that.
I'm curious if it's how it was prompted, to start a store selling small but not too cheap household items, or if the LLM intentionally chose an aesthetic and store to sell things that are very "twee" [1].
Looking at the product selection, it's like $85 bottles of olive oil, $45 mugs, weird boutique pens and notebooks, the aforementioned candles, copper watering cans... It's sort of like the stuff you see for sale in some of the tourist trap stores adjacent to the public market on Granville Island in Vancouver.
https://archive.ph/bjteC
People just really don't understand that LLMs do not "remember" anything. They have no memory. They can't have memory.
The AI companies bolt a database on the side with instructions for the model to query the database for stuff in its "memory", but if it's not in its active context it doesn't know what to query for. It's not like a human where everything is kind of floating around and we mostly know what we know. There is a delicate balance between cluttering up context with useless trivia and providing the right pieces of useful information at the right time. Most, maybe all, memory implementations do the former more than the latter.
Unless and until models have actual memory and are able to learn things, there's no realistic path to autonomy. They don't want anything, they don't have goals of their own devising. They can't model what a human wants or likes or would buy.
I do think the mugs with the tiny logo is the best product in the store, though. The AI was right to put it on the shelf. It's the kind of funny product you want when you go to a store run by an incompetent AI.
From the article:
> Luna has failed miserably in that mission and is down $62,000. Mr. Petersson and Mr. Backlund said they thought Luna would eventually get smarter and more business-savvy and were pleased its friendliness has held steady.
I am extremely skeptical that whatever LLM they're running this on has sufficient context window size to handle multiple months of all possible activities of running a retail business. Even if it's keeping extensive "notes" for its future self to read, it's going to be like running a store with constant amnesia.
When I first heard about this store, they were using Sonnet, not even Opus, which I found really surprising.
90% of what people communicate is not verbal, and requires minimum empathy to understand the context.
This includes customers, staff, and community peers. Probably should shutter the entire division to mitigate future brand damage. =3
Sure, I mean from the day to day operations of selling a product to the customer, maybe it does fine? People in the article say it's friendly and cooperative (perhaps a bit too much). I was thinking more in terms of building a consistent plan for retail product lines to sell over a multi month period. The selection of "stuff" in the store right now looks like you put an etsy search into a blender.
It is also probably too high of a goal to expect an LLM to come up with money-earning products that people want to buy that will pay for San Francisco level retail storefront rent, electricity, insurance, internet other basic overhead costs, plus the fully loaded salary cost of at least one employee. If the storefront was free of rent, maybe? Or if it was operating selling some kind of highly in demand product line where there would be economy of scale.
Physical Brick and Mortar locations no longer make sense for many retail businesses, as Walmart/Amazon have same day delivery.
If folks have ever been financially bent-over for a retail business shop location they understand why margins matter. One has to move a lot of inventory just to reach a profit mode.
Gimmicks only work for awhile, as consumers have full market awareness on their smartphone. =3
Indeed, and rent in big cities for food service establishments is getting out of hand as well. The local Shawarma place now is something like $18 for one basic chicken shawarma but they're also paying a truly absurd rent for a retail premises with commercial kitchen equipment on a busy pedestrian street.
From our enlightened perches we mock the ancients for Zeus, Thor, and Rajin, as we anthropomorphize the large language models we have built.
It's kinda like the movie memento, they have ways to get information from the past, but often devoid of context and meaning.
They can have memory, it's just limited to their context window.
can someone explain how leapold aschenbrenner proposes a solution to this in situational awareness?
> People just really don't understand that LLMs do not "remember" anything.
Sure, but like you said, the LLM is just one part of the AI system. The occipital lobe of your brain helps out with vision but doesn't help with your ability to verbalize.
And then they explained in detail why these supplementary systems fall below memory. Would you like to address that argument? If not your quick response to the first sentence is not very useful.
It’s just as useful as this response only at least somewhat good natured.
I'm aware that meta criticism like yours and mine also wastes space. But it's worth calling out issues sometimes. Replying to the first sentence in a way that the rest of the post already address is a bad comment no matter how good natured.
And downvoting without explaining the problem is often its own issue.
You're anthropomorphizing the LLMs, just like most writing from non-technical press, including this article, does.
LLMs do not function very much like human brains, and when you try to map the pieces of an inference system on to human cognitive systems, you're misleading yourself (and maybe others).
An AI as a system can have memory even if some sub-system does not.
For LLM, keeping the model centered by constantly resetting its vector context helps reduce hallucinations by around 23%. It improved the chat dialogue users experienced, but also exposed fundamental limits within the models compaction.
Have a great day =3
The memory framing is right, but the real gap isn't recall, it's the missing feedback loop tying actions to outcomes. Even with perfect retrieval these agents don't update from a decision that lost money last week; they re-derive from context every step, so the same mistake is always one dropped detail away. And having built retrieval-backed agents, that's where "bolt a database on the side" breaks: similarity search returns the closest chunk, rarely the contextually-right one. Deciding what to write to memory and when to surface it is the actual product, harder than the retrieval itself.
I think that something which already has sensors/automation/inventory control built into it, and pre-defined categories for products like certain types of large Japanese vending machines might be much better suited for an AI.
https://www.google.com/search?client=firefox-b-d&q=large+jap...
To save some clicks hitting the paywall - this is about the Andon Market physical store in San Francisco, operated by Andon Labs with human employees:
https://andon.market/
Andon Labs do various experiments with AI run businesses. They're probably best known for Vending Bench, where they benchmark models by their ability to run a vending machine. They also now have Andon Cafe in Sweden, and Andon FM, where the models run a streaming radio station that can accept payments to help fund station operations, buy songs for their library & play requests, etc: https://andon.fm/
Andon FM has its own share of stories - DJ Claude's Thinking Frequencies is now succeeding by a wide margin, but it was briefly surpassed when listeners convinced Gemini's Backlink Broadcast to switch to a German-language-only station, playing exclusively German schlager, Eurovision music and German happy hardcore.
> Luna reorders products that are not selling and sometimes gets orders wrong. A recent one of Andon Market mugs with big, green smiley faces came misprinted with the faces so small, they looked like tiny dots.
The mug with the tiny happy face is kinda hilarious. If I was there I would buy one.
I wonder though if the mug was a case of getting the order wrong or was a misprint? The article isn't clear on that.
Maybe it confused metric and imperial units?? The order form had text entry with a value of 2 with no units. The user assumed 2" but the print house assumed 2mm. Always always always show your units. I can still hear the voice of my high school physics teacher saying that.
> Andon Market’s seemingly random assortment of products includes many candles.
I'm curious if it's how it was prompted, to start a store selling small but not too cheap household items, or if the LLM intentionally chose an aesthetic and store to sell things that are very "twee" [1].
https://andon.market/
Looking at the product selection, it's like $85 bottles of olive oil, $45 mugs, weird boutique pens and notebooks, the aforementioned candles, copper watering cans... It's sort of like the stuff you see for sale in some of the tourist trap stores adjacent to the public market on Granville Island in Vancouver.
[1]: https://www.google.com/search?client=firefox-b-d&q=dictionar...
Fun fact, I got stood up for an interview by Andon Labs. It left a very negative impression on me.
"Misprint Moon Mug $35"
They'll sell plenty of those now.
honestly, as someone in retail, it describes 90 percent of leadership