> Technical: I started this project when the first LLMs came out. I've built extensive internal evals to understand how LLMs are performing. The hallucinations at the time were simply too frequent to passthrough this data to visitors. However, I recently re-ran my evals with Opus 4.5 and was very impressed. I am running out of scenarios that I can think/find where LLMs are bad at interpreting data.
It's nice to see an AI-centric Show HN product that uses proper evals and cares about data quality.
How did you build your initial data set that you're using for the evals? Bootstrapping a high quality data set is one of the hardest parts of really knowing how an AI product is performing.
Sort've related, but here in Australia pet food manufactures are not required to list the nutritional content of their foods, whereas in the US as I understand it they do.
They do but the nutritional information guides you to feeding your dog 20,000kcal a day. The suggested serving size on every brand I've seen is about 5 cups for a 70lb dog, whereas my dog gains weight on more than one cup.
At least the "grain free" labels appear to be accurate.
I'd love to see a project that actually analyzes every supplement on the market to make sure it actually contains what it claims to, contains it at the listed dosage, and to show anything else found (heavy metals for example). That's not something AI can do for us though since it'd involve physically collecting and testing samples.
At the moment, my focus is on what I can do by aggregating and analyzing the available data. Extracting ingredients, normalizing them, normalizing quantities across every supplement, etc. This has already proven to be a lot bigger undertaking than I could have imagined at the beginning of this journey. I had learn a lot about databases, scraping, LLMs, and evals. But it has been a tremendously fun (if sometimes overwhelming) journey.
First, I need to figure out how to monetize what I've built so far. And maybe I cannot, in which case I will start over. But I am trying to find my niche that people uniquely value. So far I found that there are is a pull from people chasing deals (e.g. finding products containing specific ingredients with the highest price per mcg) and people seeking for niche ingredients. This is not exactly the target audience I had in mind when building this, but I am glad they are finding value.
Evolving into performing actual lab tests and producing our own high-quality supplements is my dream.
I did something like this, pre LLM, except with food additives.While I dont have the technical expertise to fully evaulate what these substances are, and do, in many cases they are banned in some juristictions, and often have exceptionaly dubious reasons for bieng put in food, I did notice one very stark thing, in that not one single food aditive has ever been held up as " WOW! this stuff is so good!, a boon and benifit, etc, whatever" nothing, zero substances acclaimed as unequivical benifit to humanity.
I eat only food now.
Oh, and one more thing, many of these things are "white crystaline" substances, which means they are 100% concentrated things that are never found in actual foods,
think sugar.
Doesn’t that kind of exist already across a couple sites? Thinking consumer labs does this across lots of supplement categories. I have seen a few others too.
That's the plan. I don't intend to own any stock. I want to focus on covering the broadest range of supplements across all of the marketplaces, having the richest data about them, and then focus on the affiliate revenue.
The affiliate revenue can be anywhere from 5% to 10% depending on the affiliate partner. Considering no overhead of support, inventory, or logistics, it's a pretty good deal for me, especially for now, while I'm still a solo founder.
I just searched the most popular/researched supplement of all time: creatine. There is a mistake in the data there: the Wellnesss Code Whey Protein indeed contains creatine, but not 2g per container, but 2g per serving (that is correctly reflected). Error is easily spotted due to the price per gram being an extreme outlier. That is perhaps something you can look for when evaluating the data gotten from the LLM.
There is interesting context here. The first version of Pillser was focused on very narrow area of supplements, and specifically, supplements sold in the shape of pills (therefore the name "Pillser" [pill search]). However, it kinda snowballed from there, and powder substances were one of the last things I've added. The evals need to be expanded to have more examples specifically around powder substances. However, that's a fix I know how to implement and will prioritize.
Thank you for sharing the example that you've found!
Failed to execute 'removeChild' on 'Node': The node to be removed is not a child of this node.
Something broke. We're working on it, but in the meantime, try reloading the page. If that doesn't work, come back later.
Error ID: b846e2e2ba3b483ab93f10e72ef76820
NotFoundError: Failed to execute 'removeChild' on 'Node': The node to be removed is not a child of this node.
at ds (https://pillser.com/assets/entry.client-DWgmqxdv.js:1:112074)
at gs (https://pillser.com/assets/entry.client-DWgmqxdv.js:1:113602)
at ys (https://pillser.com/assets/entry.client-DWgmqxdv.js:1:113850)
at gs (https://pillser.com/assets/entry.client-DWgmqxdv.js:1:113728)
I appreciate you letting me know. I can see it in Sentry as well. It's odd though, it looks like it's coming directly from React. There are no traces in my own codebase. I'll try to isolate it.
I’d love to hear your domain expertise—specifically which supplements you’d recommend avoiding, which ones are generally worth taking, and whether there are any brands or products you trust. I imagine you’ve picked up a lot of insights beyond the technical side as well.
I think it will be nice if there is a summary of that somewhere on the website. It was the first thing I was looking for.
Great resource for categories and research on supplements. Because supplements in U.S. are in the "wild west" of regulatory oversite, its important to know ingredient efficacy and ingredient purity. Consumer Labs (pay for subscription) offers some testing on selected products. Need to tweek "purchase from other market" -not all markets are "Out" as indicated.
Please seriously consider *not* using ads to generate revenue. We need to return to other models rather than just slapping ads on everything. Be the change you want to see in the world. Do you really want to see more ads?
Have a look at https://examine.com as the closest site I can think of in this space, and check out their revenue models.
Without ads, you are just gatekeeping it from poor people. You can make the premium ad-free, but making the free version ad-free is basically asking for the price of the premium version to be raised to subsidize everyone else.
That is odd. When I searched for Vitamin D, it showed me a list (which I could sort by $/mg). That list included Vitamin D, Vitamin D2 and Vitamin D3. I assumed (incorrectly it seems) that this meant that it was doing some sort of prefix match. It seems that the result table highlights the "Ingredient" column, so I assumed (incorrectly) that it was searching that column. However, it doesn't seem to search the "Supplement" column, nor is it doing a prefix match (or substring match) on the Ingredient column. This is just confusing.
There is even a Google Sheet that is generated based on the data that we have, although I just noticed that it's not updating properly, so I need to fix that.
I would love to get your feedback on what data to prioritize and how to make it easier to browse.
I've learned from my last sprint that this project can get overwhelming. So now I am making it my priority to build a community of people I can reach out to for feedback and direction. If supplements is your thing, I would greatly appreciate being able to chat with you from time to time (even if it just a quick email!)
Hi, not into supplements, but really appreciating your approach to building a community and the reasoning behind it (and that you're being so transparent about it!). I'll give this a go too, thanks for the inspiration :).
I am more of a geek than a business shark, but when I was doing market research, I couldn't believe just how big some of the supplement marketplaces are. iHerb ($2nb+), Amazong ($8bn+), GNC ($2bn+), Vitamin Shoppe ($1bn+). However, it's a market that requires a lot of investment to enter (logistic/storage/support). I'm hoping I'm going to get my foot across the door by providing the most comprehensive solution for education, as well as discovering and comparing supplements across all of these marketplaces.
I don't want to, and I don't plan to. At a time when this happened, I was deep in my studies preparing for exams, and I just remember thinking to myself, "I cannot afford to pull myself into what could become a legal matter" I am now in better position to allocate time and attention should such claims be made again.
That said, the couple of brands that were removed were not brands that I would have wanted anyone to buy anyway. Not much is lost by not having them on the website from the perspective of fulfilling customers' journeys of finding a good product.
I do like the suggestion made in one of the comments informing customers why some brands are not visible on the website. That warning might on its own deter others from making such claims.
Maybe, have a page in which you show that all the details of that product have been redacted as the producer requested. Something like a DMCA strike list.
This leaves a great say about how muche the company wants to manage the narrative of their product.
U GOT A WAYK00L BB0ARD D00DZ!
I borowed my brothers vic-2O to read ur bboard!!1!
Suplements r important 2 me after mom fed us nothing but enfamil
It looks like my doc says i got "Irritable Biff Syndrome"
So I gonna order some of these (do they accept Minecraft or RuneScape money??
UGOT WAYK00L BB0ARDZ, D00DZ!!
1. dangerously unwise area to apply "AI" given all its known problems, and whats at stake
2. I tried using that site and immediately saw problems and broken behavior. The "Ask AI" feature (sounds pretty key?) literally did nothing. and doing a search for say aspirin yielded results that... had no aspirin! no brainer to bail out fast
I've fucked myself up so much over the years with supplements. Wish I could go back 30 years and tell myself to just eat real food. B6 toxicity for instance is crippling. That being said these days I just take two things: vit K2 and magnesium.
That's solid advice for most people in the Western world. K2 and magnesium are sensible choices–genuinely hard to get enough of through diet alone. The bigger problem is that most supplement sales are marketing-led. That's what pushed me to at least try to make the whole process more scientific.
> Technical: I started this project when the first LLMs came out. I've built extensive internal evals to understand how LLMs are performing. The hallucinations at the time were simply too frequent to passthrough this data to visitors. However, I recently re-ran my evals with Opus 4.5 and was very impressed. I am running out of scenarios that I can think/find where LLMs are bad at interpreting data.
It's nice to see an AI-centric Show HN product that uses proper evals and cares about data quality.
How did you build your initial data set that you're using for the evals? Bootstrapping a high quality data set is one of the hardest parts of really knowing how an AI product is performing.
Unfortunately, none of these data are usable because (in the US, at least) there is no oversight on labeling accuracy for nutritional supplements.
That means I can dump woodchips into capsules and sell them as Multivitamins with 12 vitamins & minerals, and nobody would be the wiser.
There is more rigorous testing being done in underground steroid + peptide communities than in legal nutritional supplements.
Crazy world where you can trust vialed peptides from China more than something you bought on Amazon...
That's true. US is a wild wild west in that regard. However, I am working next to clearly label which supplements have COA vs which are unverified.
Sort've related, but here in Australia pet food manufactures are not required to list the nutritional content of their foods, whereas in the US as I understand it they do.
They do but the nutritional information guides you to feeding your dog 20,000kcal a day. The suggested serving size on every brand I've seen is about 5 cups for a 70lb dog, whereas my dog gains weight on more than one cup.
At least the "grain free" labels appear to be accurate.
Supplements should at least be regulated like food. List of ingredients and tests that contains ingredients and doesn't contain anything harmful.
This seems more like cataloguing than analyzing
I'd love to see a project that actually analyzes every supplement on the market to make sure it actually contains what it claims to, contains it at the listed dosage, and to show anything else found (heavy metals for example). That's not something AI can do for us though since it'd involve physically collecting and testing samples.
That's my dream!
That's where I want to take this project.
At the moment, my focus is on what I can do by aggregating and analyzing the available data. Extracting ingredients, normalizing them, normalizing quantities across every supplement, etc. This has already proven to be a lot bigger undertaking than I could have imagined at the beginning of this journey. I had learn a lot about databases, scraping, LLMs, and evals. But it has been a tremendously fun (if sometimes overwhelming) journey.
First, I need to figure out how to monetize what I've built so far. And maybe I cannot, in which case I will start over. But I am trying to find my niche that people uniquely value. So far I found that there are is a pull from people chasing deals (e.g. finding products containing specific ingredients with the highest price per mcg) and people seeking for niche ingredients. This is not exactly the target audience I had in mind when building this, but I am glad they are finding value.
Evolving into performing actual lab tests and producing our own high-quality supplements is my dream.
I did something like this, pre LLM, except with food additives.While I dont have the technical expertise to fully evaulate what these substances are, and do, in many cases they are banned in some juristictions, and often have exceptionaly dubious reasons for bieng put in food, I did notice one very stark thing, in that not one single food aditive has ever been held up as " WOW! this stuff is so good!, a boon and benifit, etc, whatever" nothing, zero substances acclaimed as unequivical benifit to humanity. I eat only food now. Oh, and one more thing, many of these things are "white crystaline" substances, which means they are 100% concentrated things that are never found in actual foods, think sugar.
Doesn’t that kind of exist already across a couple sites? Thinking consumer labs does this across lots of supplement categories. I have seen a few others too.
Consumer Labs is definitely the furthest ahead compared to everyone else. I've been using them for many years now.
Sounds expensive to test every podunk brand on Amazon. Maybe just the big brands that claim to be accurate like Bryan Johnson's Blueprint.
Monetize it with Amazon or other affiliate links, and provide dollar per effective dose for a given set of desired supplements.
That's the plan. I don't intend to own any stock. I want to focus on covering the broadest range of supplements across all of the marketplaces, having the richest data about them, and then focus on the affiliate revenue.
The affiliate revenue can be anywhere from 5% to 10% depending on the affiliate partner. Considering no overhead of support, inventory, or logistics, it's a pretty good deal for me, especially for now, while I'm still a solo founder.
I just searched the most popular/researched supplement of all time: creatine. There is a mistake in the data there: the Wellnesss Code Whey Protein indeed contains creatine, but not 2g per container, but 2g per serving (that is correctly reflected). Error is easily spotted due to the price per gram being an extreme outlier. That is perhaps something you can look for when evaluating the data gotten from the LLM.
There is interesting context here. The first version of Pillser was focused on very narrow area of supplements, and specifically, supplements sold in the shape of pills (therefore the name "Pillser" [pill search]). However, it kinda snowballed from there, and powder substances were one of the last things I've added. The evals need to be expanded to have more examples specifically around powder substances. However, that's a fix I know how to implement and will prioritize.
Thank you for sharing the example that you've found!
found a bug searching for "collagen":
I appreciate you letting me know. I can see it in Sentry as well. It's odd though, it looks like it's coming directly from React. There are no traces in my own codebase. I'll try to isolate it.
Might be a browser extension fucking with the dom. Just guessing.
I’d love to hear your domain expertise—specifically which supplements you’d recommend avoiding, which ones are generally worth taking, and whether there are any brands or products you trust. I imagine you’ve picked up a lot of insights beyond the technical side as well.
I think it will be nice if there is a summary of that somewhere on the website. It was the first thing I was looking for.
Great resource for categories and research on supplements. Because supplements in U.S. are in the "wild west" of regulatory oversite, its important to know ingredient efficacy and ingredient purity. Consumer Labs (pay for subscription) offers some testing on selected products. Need to tweek "purchase from other market" -not all markets are "Out" as indicated.
Please seriously consider *not* using ads to generate revenue. We need to return to other models rather than just slapping ads on everything. Be the change you want to see in the world. Do you really want to see more ads?
Have a look at https://examine.com as the closest site I can think of in this space, and check out their revenue models.
I know Examine.com, and I've spoken to the founder a few times. I really appreciate what they are doing.
I am very open about how the site is planning to be monetized https://pillser.com/affiliate-disclosure
Without ads, you are just gatekeeping it from poor people. You can make the premium ad-free, but making the free version ad-free is basically asking for the price of the premium version to be raised to subsidize everyone else.
See also rtings.com, with 4,359 products bought and tested.
https://www.rtings.com/company/how-we-make-money
I picked Vitamin D as it was an option on the main page. The cheapest offered was $1.1/mg.
Costco (https://www.costco.com/p/-/kirkland-signature-extra-strength...) sells Vitamin D at less than half that price. On Amazon, the two pack of those is even cheaper.
Just an observation.
I think you've mistaken Vitamin D and Vitamin D3
https://pillser.com/search?s=dfbtbc9110&q=%22Vitamin+D3%22
The cheapest price on Pillser for D3 is $0.26/mg
That is odd. When I searched for Vitamin D, it showed me a list (which I could sort by $/mg). That list included Vitamin D, Vitamin D2 and Vitamin D3. I assumed (incorrectly it seems) that this meant that it was doing some sort of prefix match. It seems that the result table highlights the "Ingredient" column, so I assumed (incorrectly) that it was searching that column. However, it doesn't seem to search the "Supplement" column, nor is it doing a prefix match (or substring match) on the Ingredient column. This is just confusing.
https://pillser.com/search?q=%22Vitamin+D%22
When you're typing things into search, you're using a freeform search which is going to match the ingredients based on the partial match.
If you want to look for the specific ingredients, you can do so by clicking on the "Ingredients" of the product example: Here are the links for the Vitamin D and D3. https://pillser.com/vitamins/vitamin-d and https://pillser.com/vitamins/vitamin-d3
There is even a Google Sheet that is generated based on the data that we have, although I just noticed that it's not updating properly, so I need to fix that.
This is fantastic! I've been doing this manually for years. My main focus is always what does the research show about xyz supplement on gut bacteria.
I would love to get your feedback on what data to prioritize and how to make it easier to browse.
I've learned from my last sprint that this project can get overwhelming. So now I am making it my priority to build a community of people I can reach out to for feedback and direction. If supplements is your thing, I would greatly appreciate being able to chat with you from time to time (even if it just a quick email!)
I am at lilouartz@gmail.com
Hi, not into supplements, but really appreciating your approach to building a community and the reasoning behind it (and that you're being so transparent about it!). I'll give this a go too, thanks for the inspiration :).
There is big money in supplements, so I don't think it will be difficult to monetize.
I am more of a geek than a business shark, but when I was doing market research, I couldn't believe just how big some of the supplement marketplaces are. iHerb ($2nb+), Amazong ($8bn+), GNC ($2bn+), Vitamin Shoppe ($1bn+). However, it's a market that requires a lot of investment to enter (logistic/storage/support). I'm hoping I'm going to get my foot across the door by providing the most comprehensive solution for education, as well as discovering and comparing supplements across all of these marketplaces.
> Shady companies are sending C&Ds letters demanding their products are taken down from the website.
That's surprising because it seems like you just have a review site? What's the issue? Or is it just bs threats?
Please don't take any products down due to shady legal threats! Won't this compromise nearly the entirety of the value proposition?
Cripes.
I don't want to, and I don't plan to. At a time when this happened, I was deep in my studies preparing for exams, and I just remember thinking to myself, "I cannot afford to pull myself into what could become a legal matter" I am now in better position to allocate time and attention should such claims be made again.
That said, the couple of brands that were removed were not brands that I would have wanted anyone to buy anyway. Not much is lost by not having them on the website from the perspective of fulfilling customers' journeys of finding a good product.
I do like the suggestion made in one of the comments informing customers why some brands are not visible on the website. That warning might on its own deter others from making such claims.
Maybe, have a page in which you show that all the details of that product have been redacted as the producer requested. Something like a DMCA strike list. This leaves a great say about how muche the company wants to manage the narrative of their product.
1. dangerously unwise area to apply "AI" given all its known problems, and whats at stake
2. I tried using that site and immediately saw problems and broken behavior. The "Ask AI" feature (sounds pretty key?) literally did nothing. and doing a search for say aspirin yielded results that... had no aspirin! no brainer to bail out fast
Aspirin is not a supplement; it's a medication.
idk if you invented this but "add a ? to the end to get an AI response" seems like a convention that should catch on.
Well done!
OMG you've noticed!
I was so proud when I came up with it, but I don't think anyone ever commented on it until now.
and I agree. It needs to catch on!
How did you get around the legal issues?
Do you include the EU market as well?
I've fucked myself up so much over the years with supplements. Wish I could go back 30 years and tell myself to just eat real food. B6 toxicity for instance is crippling. That being said these days I just take two things: vit K2 and magnesium.
That's solid advice for most people in the Western world. K2 and magnesium are sensible choices–genuinely hard to get enough of through diet alone. The bigger problem is that most supplement sales are marketing-led. That's what pushed me to at least try to make the whole process more scientific.