I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.
that's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.
I was thinking you scrape your own website every day in the middle of the night when traffic is low, and make that available. They can come and collect it every day if they want to.
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!
I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
There is a large community of people that poison scrapers.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.
People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue.
The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.
exactly i took a look at that subreddit and doesnt look like theres any professionals just bunch of anti-AI users who thinks they are smarter
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
What a pity. Mostly I just want personal archives of things so that I can search them much faster than commercial solutions and the like.
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
Feels like it would be a good time for freenet and the like to catch on.
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.
that's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.
I was thinking you scrape your own website every day in the middle of the night when traffic is low, and make that available. They can come and collect it every day if they want to.
Yeah. Though I guess the point I thought of was like a deals site. That would have infinite pages and content
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
One article mentioned in the OP was discussed here:
Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)
How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!
Yes I've seen it. ClaudeBot will gleefully announce itself when it hammers my niche website a million times a day.
I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
I think they also have to operate in countries that don't mind shady things like this.
TIL:
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
WTF. That's just botnets.
Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...
Thank god for residential proxies.
Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet
i know a few very large startups that used it to fake their way into an exit
unethical yes but really raises the question as to what we see is real or not
Money is real. DAU that don't pay subscriptions, or don't lead to paid conversions on hosted ads, are worthless.
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
> widespread scraping of web sites in search of training data for large language models and related projects
This is a good thing, thanks to this we have powerful open source LLMs.
> This activity overwhelms sites with traffic.
When LLMs get good enough, we won't need those sites anymore :)
[not satire, this is what I think, without self-censorship]
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
What exactly should be illegal here? Scraping websites? AI agents? Not following robots.txt?
Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
"He who has the gold makes the rules" is older than the pyramids.
> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.
There is a large community of people that poison scrapers.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...
I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
https://news.ycombinator.com/newsguidelines.html
Seriously, dang?
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.
Just curious dang, did you warn them before banning?
Im not against the ban perse (single purpose accounts are bad), just curious if they had a chance to change their contribution style.
This is a strong positive sign that poison fountain works.
I wasn't aware of this project. Thanks for the heads up.
People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue. The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.
Yeah. A fun thing to do is to try and actually read common crawl!
Really makes you think, what we're feeding them...
exactly i took a look at that subreddit and doesnt look like theres any professionals just bunch of anti-AI users who thinks they are smarter
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
Again, why do we allow China on the Internet?
Backbone operators should not be allowed to knowingly maintain connections to networks that allow connections from China or Russia.