AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW.
I was hit with a pretty substantial botnet "distributed scraping" attack yesterday.
- About 400,000 different IP addresses over about 3 hours
- Mostly residential IP addresses
- Valid and unique user agents and referrers
- Each IP address would make only a few requests with a long delay in between requests
It would hit the server hard until the server became slow to respond, then it would back off for about 30 seconds, then hit hard again. I was able to block most of the requests with a combination of user agent and referrer patterns, though some legit users may be blocked.
The attack was annoying, but, the even bigger problem is that the data on this website is under license - we have to pay for it, and it's not cheap. We are able to pay for it (barely) with advertising revenue and some subscriptions.
If everyone is getting this data from their "agent" and scrapers, that means no advertising revenue, and soon enough no more website to scrape, jobs lost, nowhere for scrapers to scrape for the data, nowhere for legit users to get the data for free, etc.
Thanks for sharing the perspective here. I think a lot of folks on HN have rightly said that a lot of the problems with the modern internet are due to the ad-supported business model. I don't think you were ever going to move away from it voluntarily -- too many people support it, even if they grumble about it.
But maybe (and likely for worse) LLMs will finally kill this model.
I would love for the ad-supported model to die. I hate ads, and I hate having to serve ads. We get some subscription users but nowhere near enough to cover costs.
Unfortunately, what I think will happen - and indeed already is - is that the AI companies themselves will replace much of the WWW. Sites like the one I am talking about will cease to exist. AI companies, once they can no longer scrape (steal) the data will end up licensing the data themselves and replace us as the distributor to end users. Perhaps as a subscription add-on or also with an ad based model.
Which to some may be fine. Personally, I don't want a few centralized AI companies replacing the hundreds of thousands of independent websites online. Way too much centralized power there.
I much prefer having my thoughts distilled down into easily digestable and agreeable idioms that I can push around with absolute faith that they weren't just lies written by some PERSON on the internet.
Do you not run Anubis or have strict fail2ban rules? I just straight up ban IPs forever if they lookup files that will never exist on my servers. That plus Anubis with the strictest settings.
Fail2ban doesn't scale well to these volumes of traffic and request patterns.
Just like fail2ban is not very useful against a DDOS attack where each unique IP only makes a few requests with a large (hour+) delay in between requests. There is no clear "fail" in these requests, and the fail2ban database becomes huge and far too slow.
- 400,000 Unique IP addresses
- 1 to 3 requests per hour per IP addresses - with delays of over 60 minutes between each request.
- Legit request URLs, legit UA & referrer
Maybe Anubis would help, but it's also a risk for various reasons.
The more sophisticated bots run real headless browsers that anubis can't touch, and they only follow links that are actually visible on the page, so they wouldn't hit fail2ban.
They even sell access to proxy servers that successfully evade cloudflare captchas automatically.
At some point there needs to be a check if it's a real human... But it's a cat and mouse game - any way we create to keep bots off gets a work around by clever engineers.
Hard disagree, it's very easy for a bot to use a credit card. And not only are card numbers often stolen, they're even given to teenagers these days, and can also be owned by businesses and exist entirely virtually... so I don't think you can assume the use of a credit card can always be tied to legitimate use by a single person.
Companies would offer all-you-can-DDoS plans at $20/bot per month if they could. Bots are only a problem to them because they prevent legitimate customers from handing over their credit card.
Knew it was getting bad, but Meta's facebookexternalhit bot changed their behavior recently.
In addition to pulling responses with huge amplification (40x, at least, for posting a single Facebook post to an empty audience), it's sending us traffic with fbclids in the mix. No idea why.
They're also sending tons of masked traffic from their ASN (and EC2), with a fully deceptive UserAgent.
The weirdest part though is that it's scraping mobile-app APIs associated with the site in high volume. We see a ton of other AI-training focused crawlers do this, but was surprised to see the sudden change in behavior on facebookexternalhit ... happened in the last week or so.
Everyone is nuts these days. Got DoSed by Amazonbot this month too. They refuse to tell me what happened, citing the competitive environment.
> it's sending us traffic with fbclids in the mix. No idea why.
The click IDs are likely to make the traffic look more like a human who has clicked a link rather than a bot? That way it gets past simple filters that explicitly let such requests in before bothering to check that the source address of the request seems to be a DC rather than a residential IP.
> citing the competitive environment
All the companies are competing to be the biggest inconvenience to everyone else while scraping as much stuff as they can.
> The click IDs are likely to make the traffic look more like a human who has clicked a link rather than a bot?
It's certainly possible. However, the traffic is still coming from Facebook's network with a FB proxy PTR record in DNS. Seems much more likely to fool your typical site owner than a bad actor.
> Seems much more likely to fool your typical site owner than a bad actor.
Sorry, I wasn't clear. That is exactly what I think they would be doing. While it won't fool blocks that check source first it will fool those who allow requests with those IDs through because they can't afford to block their content appearing on the distraction media sites.
After 2 minutes at 150 kHashes on mobile, I finally see the first pixel of the progress bar filling up. Seems like it will take hours or a day to finish. Some estimate would have been nice.
Ironically I used a LLM to write a bypass for this ridiculous tool, doing hashing in a browser makes no sense, Claude's very bad implementation of it in C does tens of megahash a second and passes all of the challenges nearly instantly. It took about 5 minutes for Claude to write that, and it's not even a particularly fast implementation, but it beats the pants off doing string comparisons for every loop in JavaScript which is what the Anubis tool does.
for (; ;) {
const hashBuffer = await calculateSHA256(data + nonce);
const hashArray = new Uint8Array(hashBuffer);
let isValid = true;
for (let i = 0; i < requiredZeroBytes; i++) {
if (hashArray[i] !== 0) {
isValid = false;
break;
}
}
It's less proof of work and just annoying to users, and feel good to whoever added it to their site, I can't wait for it to go away. As a bonus, it's based on a misunderstanding of hashcash, because it is only testing zero bytes comparison with a floating point target (as in Bitcoin for example), the difficulty isn't granular enough to make sense, only a couple of the lower ones are reasonably solvable in JavaScript and the gaps between "wait for 90 minutes" and "instantly solved" are 2 values apart.
>It's less proof of work and just annoying to users, and feel good to whoever added it to their site,
this is being disproved in the article posted:
>And so Anubis was enabled in the tar pit at difficulty 1 (lowest setting) when requests were pouring in 24/7. Before it was enabled, it was getting several hundred-thousand requests each day. As soon as Anubis became active in there, it decreased to about 11 requests after 24 hours, most just from curious humans.
apparently it does more than annoying users and making the site owner feel good (well, i suppose effective bot blocking would make the site owner feel quite good)
> After 2 minutes at 150 kHashes on mobile, I finally see the first pixel of the progress bar filling up. Seems like it will take hours or a day to finish. Some estimate would have been nice.
Literally the grandparent of the comment chain you're responding to.
Yes, Anubis is just non standard and obscure, the proof of work bit is completely irrelevant (except for getting people on their phone to not visit your website).
It doesn't matter if your hottest loop is using string comparisons, as another poster pointed out in C you aren't even doing the majority of the second hash because you know the result (or enough of it) before finishing it. The JavaScript version just does whole hashes and turns them into a Uint8Array, then iterates through it.
I dont understand what you mean. Are you saying that in C you only calculate the first few digits of the hash? That's not how sha256 works.
Edit: oh i think you mean in c the string comparison short curcuits. I would expect the same to be true in javascript too. Its true in most languages.
Maybe you are just worried about general language overhead, which is a fair point. Is the anubis check even using multiple threads? For the c case, the real benefit wouldn't be if you can use C, but if you can use the GPU.
The whole thing is kind of silly though. SHA256 is a terrible choice of hash for PoW. They should be using argon2 or something memory heavy.
The language matters, but your original guess was actually correct, you can do tricks with sha256 where you only end up calculating a fraction of the total double hash in order to get a pass or fail.
Modern bitcoin miners do a double sha256 hash and increment in just a little bit more than a single hash of work. The input is 80 bytes, which is two compression rounds of 64 bytes in sha256, only the data in the second round has changed (the appended nonce), so you don’t bother doing the first compression round again. With other quirks you can end up doing multiple hashes at once “asicboost” due to partial collisions within the input too.
Oh good point. It looks like anubis is using 64 byte random plus a nonce. Unless i'm missing something it seems like using 56 bytes random plus 8 byte nonce would be a better design or maybe hmac'ing the nonce & data together.
But then again using sha256 doesn't make sense at all.
Maybe post your brilliant solution to commercial companies with hundreds of millions in funding unrestrained bot scraping the Internet for AI training instead of complaining about people desperate to rein it in as individuals.
Anybody can prompt Claude to implement this, which was my point, it doesn't stop bots because a bot can literally write the bypass! My prompt was the proof of work function from the repository, asked it to make an implementation in C that could solve it faster, and that was about it.
This is fallacious and extremely disrespectful (or even malicious?). You don't have to propose a way to fix a broken thing to point out that it's broken.
Normal and sane people understand this intuitively. If someone goes to a mechanic because their car is broken and the mechanic says "well, if you can tell that you car is broken, then you should be able to figure out how to fix it" - that mechanic would be universally hated and go out of business in months. Same thing for a customer complaining about a dish made for them in a restaurant, or a user pointing out a bug in a piece of software.
> Anubis uses a Proof-of-Work scheme in the vein of Hashcash
And if you look up Hashcash on Wikipedia you get https://en.wikipedia.org/wiki/Hashcash which explains how Hashcash works in a fairly straightforward manner (unlike most math pages).
Oh fun so now we're effectively draining users' phone and laptop batteries now just to prove that they have batteries and somehow that's a proxy for them being human
Given that you're clearly completely ignorant of everything in the anti-spam space, you should probably do some research before making uninformed comments like this.
I'm surprised at the effectiveness of simple PoW to stop practically all activity.
I'll implement Anubis at low difficulty for all my projects and leave a decent llms.txt referenced in my sitemap and robots.txt so LLMs can still get relevant data for my site while.keeping bad bots out. I'm getting thousands of requests from China that have really increased costs, glad it seems the fix is rather easy.
The default is to allow non-Mozilla user agents so that existing (good) automation continues to work and so that people stopped threatening to burn my house down. Lovely people in the privacy community.
I'm just saying, making visitors wait at least a minute while making their device turn red hot is going to stop 99,9% of your visitors. So at that point what's the point in trying to serve the content?
> These bots are almost certainly scraping data for AI training; normal bad actors don't have funding for millions of unique IPs thrown at a page. They probably belong to several different companies. Perhaps they sell their scraped data to AI companies, or they are AI companies themselves. We can't tell, but we can guess since there aren't all that many large AI corporations out there.
Is the theory here that OpenAI, Anthropic, Gemini, xAI, Qwen, Z.ai etc are all either running bad scrapers via domestic proxies in Indonesia, or are buying data from companies that run those scrapers?
I want to know for sure. Who is paying for this activity? What does the marketplace for scraped data look like?
I agree it's a more than a bit handwavy. The common consensus seems to be that AI companies are driving this, but it's really hard to conclusively prove who or what is behind the attacks.
Weird part #1 is that the traffic isn't for the most part shaped like crawler traffic. It's incredibly bursty, and heavily redundant, missing even the most obvious low hanging fruit optimizations.
Could be someone is using residential proxies to wrap AI agents' web traffic, but even so, there's a lot of pieces that don't really make sense, like why the traffic pattern is like being hit by a shotgun. It isn't just one request, but anywhere between 40 and 100 redundant requests.
A popular theory is that this is because of sloppy coding, AI companies are too rich to care, but then again that doesn't really add up. This isn't just a minor inefficiency, if it is "just" bad coding, they stand to gain monumental efficiency improvements by fixing the issues, in the sense of getting the data much faster, a clear competitive edge.
Really weird.
My unsubstantiated guess is the residential proxy/botnet is very unreliable, and that's why they fire so many request. Makes sense if it's sold as a service.
I suspect the redundant requests are primarily designed to weed out poisoned data served on otherwise valid URLs. I've also seen the redundant requests increase massively the more sources I blocked at the firewall level, so it feels like they're pre-emptively overcompensating for some percentage of requests being blocked.
My website contains ~6000 unique data points in effectively infinite combinations on effectively infinite pages. Some of those combinations are useful for humans, but the AI-scrapers could gain a near-infinite efficiency improvement by just identifying as a bot and heeding my robots.txt and/or rel="nofollow" hints to access the ~500 top level pages which contain close to everything which is unique. They just don't care. All their efficiency attempts are directed solely toward bypassing blocks. (Today I saw them varying the numbers in their user agent strings: X15 rather than X11, Chrome/532 rather than Chrome/132, and so on...)
> A popular theory is that this is because of sloppy coding, AI companies are too rich to care, but then again that doesn't really add up
I can substantiate this a bit. Verified traffic from Amazonbot is too dumb to do anything with 429s. They will happily slam your site with more traffic than you can handle, and will completely ignore the fact that over half the responses are useless rate limits.
They say they honor REP, but Amazonbot will still hit you pretty persistently even with a full disallow directive in robots.txt
The root sources of the traffic from residential proxies gets murky very quickly.
It's easy to follow the chain partway for some traffic, eg "Why are we receiving all this traffic from Digital Ocean? ... oh, it's their hero client Firecrawl, using a deceptive UserAgent" ... but it still leaves the obvious question about who the Firecrawl client is.
Res proxy traffic is insane these days. There is also plenty of grey-market snowshoe IPs available for the right price, from a handful of ASNs. I regularly see unified crawling missions by unknown agents using 1000+ "clean" IP addresses an hour.
> Before it was enabled, it was getting several hundred-thousand requests each day. As soon as Anubis became active in there, it decreased to about 11 requests after 24 hours
I love experimental data like this. So much better than gut reaction that was spammed when anubis was just introduced
I'm getting this patern a lot on Prestashop websites, where thousand, to not say hundreds of thousand, of request are coming from bots not announcing themselves in the User-agent, and coming from different IP's
Very annoying. And you can't filter them because they look like legitimate trafic.
On a page with differents options (such as color, size, etc...) they'll try all the combinaisons, eating all the ressources.
We started building out a set of spam/fraud/bot management tooling. If you have any decent infrastructure in place already, this is a pretty manageable task with a mismash of techniques. ASN based blocking (ip lookup databases can be self hosted and contain ASN) for the obvious ones like alibaba etc, subnet blocking for the less obvious (see pattern, block subnet, alleviates but doesn't solve problems).
If you have a logging stack, you can easily find crawler/bot patterns, then flag candidate IP subnets for blocking.
It's definitely whackamole though. We are experimenting with blocking based on risk databases, which run between $2k and $10k a year depending on provider. These map IP ranges to booleans like is_vpn, is_tor, etc, and also contain ASN information. Slightly suspicious crawling behavior or keyword flagging combined with a hit in that DB, and you have a high confidence block.
All this stuff is now easy to homeroll with claude. Before it would have been a major PITA.
What is the point of these anti bot measures if organic HN traffic can nuke your site regardless? If this is about protecting information from being acquired by undesirable parties, then this site is currently operating in the most ideal way possible.
The information will eventually be ripped out. You cannot defeat an army with direct access to TSMC's wafer start budget and Microsoft's cloud infrastructure. I would find a different hill to die on. This is exactly like the cookie banners. No one is winning anything here. Publishing information to the public internet is a binary decision. If you need to control access, you do what Netflix and countless others have done. You can't have it both ways.
I think that must be the point they're trying to make, yes
It also drives home that Anubis needs a time estimate for sites that don't use Anubis as a "can you run javascript" wall but as an actual proof of work mechanism that it purports to be its main mechanism
It shows a difficulty of "8" with "794 kilohashes per second", but what does that mean? I understand the 8 must be exponential (not literally that 8 hashes are expected to find 1 solution on average), but even as a power of 2, 2^8=256 I happen to know by heart, so thousands of hashes per second would then find an answer in a fraction of a second. Or if it's 8 bytes instead of bits, then you expect to find a solution after like 8 million hashes, which at ~800k is about ten seconds. There is no way to figure out how long the expected wait is even if you understand all the text on the page (which most people wouldn't) and know some shortcuts to do the mental math (how many people know small powers of 2 by heart)
So the elephant in the room: How much of HN is bot generated? Those who know have every incentive not to share and those who don't have no way to figure it out. At this point i have to assume that every new account is a bot
The commenters here don't care what the article is about when they can't access the article and the much more concerning question not about web scraping is.
I felt a vibe change, some are obvious and some not, but it does feel different, the main change i've seen is in downvotes, I don't say very controversial things and have had many things very quickly downvoted, and then slowly upvoted, I think hn was very slow to downvote in the past (except obvious trolls/spam). So for me the main worry is not even the comments, but the invisible bias generated by voting.
That's a good point ... it's trivial to have an agent post something onto HN on your behalf, so even old accounts are not immune. It's just the nature of things now, until we get better technology to assign some sort of "ALIVENESS" attestation to folks without revealing identity too much.
I've thought about this a bit and I can't really see why someone would want to write AI content here other than to spam ads but they are handled quickly. Does anyone see AI content with a clear motivation or agenda here? There are very few rep based privileges right so that seems like an unlikely motivation as well.
Most of the HN bot accounts I see have a link-to-vibecoded-product in bio, and/or are trying to build up "organic" activity before a Show HN post for the same.
A less publicly-visible motive would be if they were building up accounts to use for paid-upvote schemes.
This is a venture capitalist driven community that attracts the sleaziest kind of spammers you could think of under the badge of growth hacking and networking. Besides this very obvious motivation to spam you have all kinds of nerds here eager to do it just because they can (on one of the most famous tech places where registration is made as easy as possible)
I think HN is one of the better ones these days. I have no data to back this up, but the comments aren't like reddit comments. Go into any reddit post on the main subs, and you won't have to scroll very far to get a comment about Trump derailing the whole thing.
Digg's recent shutdown message talked about how bad and aggressive bots were. I'd love to see Kevin and Alex post in depth about lessons learned, Dead Internet, and call out social sites.
> let webWorkerURL = `${options.basePrefix}/.within.website/x/cmd/anubis/static/js/worker/sha256-${workerMethod}.mjs?cacheBuster=${options.version}`;
It looks like it's computing sha256 hashes.
Such an ASIC friendly PoW has the downside that someone with ASICs would be able to either overwhelm the site or drive up the difficulty so high that CPUs can never get through.
Looks like they've gone ahead and implemented the easiest fool-proof method of preventing scraping as the site is currently not loading across mutliple devices.
> The IPs of these bots here actually do not come from datacenters or VPNs most of the time; the overwhelming majority come from residential and mobile networks.
So I started searching for what these residential proxy networks actually are.
An interesting and sad aspect of the war on bots and scraping that is being waged is that we are hurting ourselves in the process, too. Many tasks I'm trying to get my AI assistant to do cannot be done quickly, because sites defensively prohibit access to their content. I'm not scraping: it's my agent trying to fetch a page or two to perform a task for me (such as check pricing or availability).
You aren’t scraping for the sake of training a model, but scraping the prices and availability is still scraping, right?
I think some of the folks running sites would rather have you go to the site and view the items “suggested based on your shopping history” (I consider these ads, the vendors might disagree), etc.
I’m more sympathetic to the people running sites than the LLM training scrapers, but these are two parties in a many-party game and neither one is perfectly aligned with users.
> scraping the prices and availability is still scraping
Web browsing is scraping, too.
I am not doing anything that I myself wouldn't do, it would just take me longer. I'm not mass-scraping, training new models, etc etc. I'm just using a helper tool to do some work for me.
If you prevent that, you are effectively saying: humans have to perform the manual labor of clicking and browsing through our site, they are not allowed to be helped in any way. I don't think this is the right answer.
I would assume most sites that block access to your AI assistant do so because they want to show a human ads, i.e. not run at a loss. Seems reasonable.
My grad school research was on computational models of human/machine cognition, and I'm now commercializing it as a 'proof-of-human API' for bot detection, spam reduction, and identity verification.
One of the mistakes people assume is that AI capability means humanness. If you know exactly where to look, you can start to identify differences between improving frontier models and human cognition.
One concrete example from a forthcoming blog post of mine:
[begin]
In fact, CAPTCHAs can still be effective if you know where to look.
We ran 75 trials -- 388 total attempts -- benchmarking three frontier AI agents against reCAPTCHA v2 image challenges. We looked across two categories: static, where each image grid is an individual target, and cross-tile challenges, where an object spans multiple tiles.
On static challenges, the agents performed respectably. Claude Sonnet 4.5 solved 47%. Gemini 2.5 Pro: 56%. GPT-5: 23%.
On cross-tile challenges: Claude scored 0%. Gemini: 2%. GPT-5: 1%.
In contrast, humans find cross-tile challenges easier than static ones. If you spot one tile that matches the target, your visual system follows the object into adjacent tiles automatically.
Agents find them nearly impossible. They evaluate each tile independently, produce perfectly rectangular selections, and fail on partial occlusion and boundary-spanning objects. They process the grid as nine separate classification problems. Humans process it as one scene.
The challenges hardest for humans -- ambiguous static grids where the target is small or unclear -- are easiest for agents. The challenges easiest for humans -- follow the object across tiles -- are hardest for agents. The difficulty curves are inverted. Not because agents are dumb, but because the two systems solve the problem with fundamentally different architectures.
Faking an output means producing the right answer. Faking a process means reverse-engineering the computational dynamics of a biological brain and reproducing them in real time. The first problem can be reduced to a machine learning classifier. The second is an unsolved scientific problem.
The standard objection is that any test can be defeated with sufficient incentive. But fraudsters weren't the ones who built the visual neural networks that defeated text CAPTCHAs -- researchers were. And they aren't solving quantum computing to undermine cryptography. The cost of spoofing an iris scan is an engineering problem. The cost of reproducing human cognition is a scientific one. These are not the same category of difficulty.
>The first problem can be reduced to a machine learning classifier. The second is an unsolved scientific problem.
I can't believe people are still using this as a generic anti-AI argument even though a decade ago people were insisting that there's no way AI can have the capabilities that frontier LLMs have today. Moreover it's unclear whether the gap even exists. Even if we take the claim that the grid pattern is some sort of fundamental constraint that AI models can't surpass, it doesn't seem too hard to work around by infilling the grids pattern and presenting the 9 images to LLMs as one image.
> “…reverse-engineering the computational dynamics of a biological brain and reproducing them in real time…”
Is not an anti-AI argument, it’s an open and unsolved question. Your optimism is appreciated, but the dismissal and assumption this is already solved is foolish and naive.
> The IPs of these bots here actually do not come from datacenters or VPNs most of the time; the overwhelming majority come from residential and mobile networks. Asian and Indonesian countries are where nearly all of them reside.
It's really awful as an indonesian, my indonesian isp regularly got blocked by HN as well :(
My blog gets this degree of scraping too. I have some 5 million requests over the same period as they say they got 7 million over and I barely noticed before I put Cloudflare in front to cache and now I don’t notice at all. I have the Cloudflare AI stuff turned off and mostly use it through the tunnel so I don’t have to expose my local IP.
Is this actually a problem? Most of my requests claim to be Amazonbot but someone showed me they weren’t and I’ve forgotten how.
TLS fingerprinting is not sufficient to stop residential proxies, the proxy acts as a transparent pass-through at the TLS layer making it trivial to use something like curl_cffi to mimic a real browser TLS fingerprint.
However residential proxies do have a weakness, since they need to maintain 2 separate TCP conenctions you can exploit RTT differences between layers 3 and 7 to detect if the connection to your server is being terminated somewhere along the path. Solutions exist that can reliably detect and block residential proxies, for example: https://layer3intel.com/tripwire
The user agents in that screenshot are fake, nobody would be running Chrome 106 on windows 10... run a php script on every page that checks for valid combinations and 400 the rest.
Worse than I could imagine? I imagine that bots might destroy the internet. Not just the internet as we know it; I mean make the internet completely unusable to any human being.
Taking a 2024 report on bot loads on the Internet is like taking a 1950s Car & Driver article for modern vehicle stats.
That’s how fast the landscape is changing.
And remember: while the report might have been released in 2024, it takes time to conduct research and publish. A good chunk of its data was likely from 2023 and earlier.
> NOTE: Use a VPN on these pages if you don't want your IP shown in the logs, but it won't be significant amongst the millions of others anyways
Is this supposed to be a joke? Is the author expecting users to travel back in time and use a VPN so their IP is scrubbed from logs that will get published at any time, because that's something the author just obviously has the right to do?
> The EDPB explicitly identifies IP addresses as being personal data due to their ability to identify individual data subjects.[1]
Everybody says that bots put websites down, while marketing oriented folks start practicing AO (agent optimization) - to make their offerings even more available and penetrating.
This is why I see (well managed) government digital IDs as sensible moves. Apart from DDOS attacks, if bots have to “prove” who they are on each request it seems like a win-win.
If you want “papers, please” every time you back out of your driveway or go beyond your government-assigned oblast, then your suggestion is the digital version of the physical authoritarian nightmare that was imposed by totalitarianist regimes throughout history.
People have a right to complete anonymity, and should be able to go across the majority of the Internet just as they can go across most of the country.
That’s what you are missing.
Don’t get me wrong, I am also in favour of a single government ID, but in terms of combatting identity fraud, accessing public resources like single-payer healthcare, and making it easier for a person to prove their identity to authorities or employers.
It should not be used as a pass card for fundamental rights that normally would have zero government involvement.
I walk out my front door in the UK and I am not anonymous. Every transaction I make either identifies me through bank, railway or other id, or quite simply by my face standing in front of the coffee seller. My walk down the road is observed by neighbours and postmen.
Should my government arrest me without cause or trample on my free speech rights, I get that’s a problem but I am not sure why being anonymous helps. Having rights upheld by the courts helps, well trained police who respect the law helps.
I am honestly open to debate on this but I do find the “what if Hitler took over government where would we be” to be a problematic argument not a final answer.
> Should my government arrest me without cause or trample on my free speech rights, I get that’s a problem but I am not sure why being anonymous helps. Having rights upheld by the courts helps, well trained police who respect the law helps.
You're suggesting the same government that would violate your rights would then help prevent it? I don't follow. Any power structure (tiered or not) was wiped away by authoritarians, historically. They will not be helping in the worst case. Ideological capture (corruption) has already started eroding at UK rights and that took a much less overt effort. America has had a robust 3-branch system (executive, legislative, judiciary) corrupted by a singular cult of personality. THAT was highly unlikely to happen, but here we are.
With this being said, I do predict that anonymity on the web is going to be phased out. It will result in all sorts of changes to cultural norms across western nations that largely will curtail rights. I dread it.
Shouldn't we try tracing IP addresses and fining organizations for letting the traffic through or originating the traffic first? Seems a lot simpler.
Sorry maybe I should be clearer - the problem of tyrannical governments is not solved by being anonymous online, or indeed any technology that makes it hard for government to do the tyrannical things. Safety lies in an engaged citizenry that reacts to fundamental threats. The protests against the ICE in Minnesota being an example.
>>> It will result in all sorts of changes to cultural norms across western nations
I quite agree - but I (hope / think) that the benefits can outweighs the downsides if done well. Those nations that do it well will I believe find a rocket like boost to society and industry perhaps akin to post 1945 world. Those who don’t will fall behind.
> Every transaction I make either identifies me through bank, railway or other id, or quite simply by my face standing in front of the coffee seller. My walk down the road is observed by neighbours and postmen.
Are these the government? Is the bank the government? Is the rail company the government?
No? Then you have answered your own question.
A silo of identification between you and a service provider that uses the provider’s own tooling is still anonymity from government authoritarianism.
The fact that nearly all of these silos are leaky IRL - with the government eager to punch howitzer-sized holes through them for even more access - is not the point. It is a citizen-hostile flaw that needs patching through loophole-proof legislation, not an ID system that would violently eradicate any remaining separation of government from capitalism.
Remember: when government and capitalism rides in the same cart, it is called corporatism, and is the basis of Fascism. Which is what is happening to America.
The problem here is that pretty much every part of modern life has been government and capitalism riding in the same cart - from cities installing electric power stations 100 years ago, to roads and inventions like the transistor and internet itself was government and private capital working towards common goals.
The issue is we want “good” government and “good” corporate behaviour but not the bad. And knowing the difference especially ahead of time requires engaged citizenry, lots of feedback mechanisms that are not overwritten by corruption and noise in the mechanism (ie primaries materringnmore than elections is a feedback mechanism fail in my book)
> The problem here is that pretty much every part of modern life has been government and capitalism riding in the same cart - from cities installing electric power stations 100 years ago, to roads and inventions like the transistor and internet itself was government and private capital working towards common goals.
And here is your argument’s fatal flaw: mistaking public works, that benefits the commons and society as a whole, with government tracking.
Just a sec - you said:
“”” Remember: when government and capitalism rides in the same cart, it is called corporatism, and is the basis of Fascism. ”””
The implication being when gov and big business get together bad things do happen. My point is they can happen yes, but also same players can make good things happen - it depends on the players (and regulatory and reporting and voting and and and )
Government tracking can be good - tracking covid cases good, tracking criminals planning robberies good. It it can also be bad.
This is imo why this is such hard problem - there is no clear answers only something we want sensible courts to (quickly) decide upon based on good laws.
You may be missing that it's easy and free for website owners to fix the problem. But it's hacker news after all. If somebody is bothered by a leaf falling on them on their walk to the corner store, the suggested solution here will be to have a full communist revolution.
Millions upon millions of people use Cloudflare to stop bots for free, and there are other alternatives as well. It's incredibly easy. So no, there's just as much need for government intervention, as there is a need for the government to stop leaves from falling on you when you're walking to the corner store.
AI companies and notably AI scrapers are a cancer that is destroying what's left of the WWW.
I was hit with a pretty substantial botnet "distributed scraping" attack yesterday.
- About 400,000 different IP addresses over about 3 hours
- Mostly residential IP addresses
- Valid and unique user agents and referrers
- Each IP address would make only a few requests with a long delay in between requests
It would hit the server hard until the server became slow to respond, then it would back off for about 30 seconds, then hit hard again. I was able to block most of the requests with a combination of user agent and referrer patterns, though some legit users may be blocked.
The attack was annoying, but, the even bigger problem is that the data on this website is under license - we have to pay for it, and it's not cheap. We are able to pay for it (barely) with advertising revenue and some subscriptions.
If everyone is getting this data from their "agent" and scrapers, that means no advertising revenue, and soon enough no more website to scrape, jobs lost, nowhere for scrapers to scrape for the data, nowhere for legit users to get the data for free, etc.
Thanks for sharing the perspective here. I think a lot of folks on HN have rightly said that a lot of the problems with the modern internet are due to the ad-supported business model. I don't think you were ever going to move away from it voluntarily -- too many people support it, even if they grumble about it.
But maybe (and likely for worse) LLMs will finally kill this model.
I would love for the ad-supported model to die. I hate ads, and I hate having to serve ads. We get some subscription users but nowhere near enough to cover costs.
Unfortunately, what I think will happen - and indeed already is - is that the AI companies themselves will replace much of the WWW. Sites like the one I am talking about will cease to exist. AI companies, once they can no longer scrape (steal) the data will end up licensing the data themselves and replace us as the distributor to end users. Perhaps as a subscription add-on or also with an ad based model.
Which to some may be fine. Personally, I don't want a few centralized AI companies replacing the hundreds of thousands of independent websites online. Way too much centralized power there.
Evidently, users and customers like not having to sift through hundreds or thousands of independent websites.
I much prefer having my thoughts distilled down into easily digestable and agreeable idioms that I can push around with absolute faith that they weren't just lies written by some PERSON on the internet.
Absolutely.
It's so much easier to know the truth when someone else tells me what it is and what to think about it.
How refreshing.
> I hate ads, and I hate having to serve ads. We get some subscription users but nowhere near enough to cover costs.
I hate ads and I hate having to use an ad blocker to be able to not go crazy in order to use the Internet.
You merely hate "having" to serve ads because it denies you profit from the people you're exploiting with those ads.
Why is your business more deserving to exist on the Internet than my usage??
Ad-free premium has shown itself again and again to devolve into ad protection rackets.
The minute the internet dies for good, the chat bots will run half-locally and request payments to stop recommending VRAM enlarging pills.
Do you not run Anubis or have strict fail2ban rules? I just straight up ban IPs forever if they lookup files that will never exist on my servers. That plus Anubis with the strictest settings.
https://anubis.techaro.lol/
Fail2ban doesn't scale well to these volumes of traffic and request patterns.
Just like fail2ban is not very useful against a DDOS attack where each unique IP only makes a few requests with a large (hour+) delay in between requests. There is no clear "fail" in these requests, and the fail2ban database becomes huge and far too slow.
- 400,000 Unique IP addresses
- 1 to 3 requests per hour per IP addresses - with delays of over 60 minutes between each request.
- Legit request URLs, legit UA & referrer
Maybe Anubis would help, but it's also a risk for various reasons.
The more sophisticated bots run real headless browsers that anubis can't touch, and they only follow links that are actually visible on the page, so they wouldn't hit fail2ban.
They even sell access to proxy servers that successfully evade cloudflare captchas automatically.
If you don't mind me asking, what sort of data are you licensing? I noticed that you explicitly don't mention it.
And self-ddos via HN advertising (a la slashdotted?:)
At some point there needs to be a check if it's a real human... But it's a cat and mouse game - any way we create to keep bots off gets a work around by clever engineers.
What makes a real human?
CC payment. This is the ultimate test.
Hard disagree, it's very easy for a bot to use a credit card. And not only are card numbers often stolen, they're even given to teenagers these days, and can also be owned by businesses and exist entirely virtually... so I don't think you can assume the use of a credit card can always be tied to legitimate use by a single person.
Companies would offer all-you-can-DDoS plans at $20/bot per month if they could. Bots are only a problem to them because they prevent legitimate customers from handing over their credit card.
Don’t worry, man, once AGI is here you’ll get your allowance (or whatever the hyperscalers plan is).
You’ll enjoy painting or some other art even if you aren’t interested in the arts. That’s what I’ve seen written about it.
Or they'll let you starve to death which is way way easier and way faster for "them"
Well yeah, art will be valueless in a flooded market. Starving is implied, “starving artist”.
Unfortunately nobody cares about destroying the internet if it gets them a Lambo.
Greed and ignorance have taken over the tech industry.
Knew it was getting bad, but Meta's facebookexternalhit bot changed their behavior recently.
In addition to pulling responses with huge amplification (40x, at least, for posting a single Facebook post to an empty audience), it's sending us traffic with fbclids in the mix. No idea why.
They're also sending tons of masked traffic from their ASN (and EC2), with a fully deceptive UserAgent.
The weirdest part though is that it's scraping mobile-app APIs associated with the site in high volume. We see a ton of other AI-training focused crawlers do this, but was surprised to see the sudden change in behavior on facebookexternalhit ... happened in the last week or so.
Everyone is nuts these days. Got DoSed by Amazonbot this month too. They refuse to tell me what happened, citing the competitive environment.
> it's sending us traffic with fbclids in the mix. No idea why.
The click IDs are likely to make the traffic look more like a human who has clicked a link rather than a bot? That way it gets past simple filters that explicitly let such requests in before bothering to check that the source address of the request seems to be a DC rather than a residential IP.
> citing the competitive environment
All the companies are competing to be the biggest inconvenience to everyone else while scraping as much stuff as they can.
> All the companies are competing to be the biggest inconvenience to everyone else while scraping as much stuff as they can.
So the invisible hand is just a middle finger?
> The click IDs are likely to make the traffic look more like a human who has clicked a link rather than a bot?
It's certainly possible. However, the traffic is still coming from Facebook's network with a FB proxy PTR record in DNS. Seems much more likely to fool your typical site owner than a bad actor.
> Seems much more likely to fool your typical site owner than a bad actor.
Sorry, I wasn't clear. That is exactly what I think they would be doing. While it won't fool blocks that check source first it will fool those who allow requests with those IDs through because they can't afford to block their content appearing on the distraction media sites.
Ahh gotcha
I’ve been sitting on this page for two minutes and it’s still not sure whether I’m a bot lol. What did I do in a past life to deserve this :(
After 2 minutes at 150 kHashes on mobile, I finally see the first pixel of the progress bar filling up. Seems like it will take hours or a day to finish. Some estimate would have been nice.
Ironically I used a LLM to write a bypass for this ridiculous tool, doing hashing in a browser makes no sense, Claude's very bad implementation of it in C does tens of megahash a second and passes all of the challenges nearly instantly. It took about 5 minutes for Claude to write that, and it's not even a particularly fast implementation, but it beats the pants off doing string comparisons for every loop in JavaScript which is what the Anubis tool does.
It's less proof of work and just annoying to users, and feel good to whoever added it to their site, I can't wait for it to go away. As a bonus, it's based on a misunderstanding of hashcash, because it is only testing zero bytes comparison with a floating point target (as in Bitcoin for example), the difficulty isn't granular enough to make sense, only a couple of the lower ones are reasonably solvable in JavaScript and the gaps between "wait for 90 minutes" and "instantly solved" are 2 values apart.I wrote one that uses opencl: https://github.com/DavidBuchanan314/anubis_offload
Bravo, you even implemented the midstate speedup from Bitcoin, that's way more impressive.
It's not exactly rocket science heh, just baffling that the original anubis impl left an order-of-magnitude speedup on the table.
>It's less proof of work and just annoying to users, and feel good to whoever added it to their site,
this is being disproved in the article posted:
>And so Anubis was enabled in the tar pit at difficulty 1 (lowest setting) when requests were pouring in 24/7. Before it was enabled, it was getting several hundred-thousand requests each day. As soon as Anubis became active in there, it decreased to about 11 requests after 24 hours, most just from curious humans.
apparently it does more than annoying users and making the site owner feel good (well, i suppose effective bot blocking would make the site owner feel quite good)
The Anubis difficulty setting is (or was) so high that nobody could visit the site without leaving it open for minutes or hours.
I think that only happens when you use an old browser without native sha256 support (so it fallback to pure js implementation of sha256).
>Anubis was enabled in the tar pit at difficulty 1 (lowest setting) when requests were pouring in 24/7
>difficulty 1 (lowest setting)
literally in the comment you're responding to
> After 2 minutes at 150 kHashes on mobile, I finally see the first pixel of the progress bar filling up. Seems like it will take hours or a day to finish. Some estimate would have been nice.
Literally the grandparent of the comment chain you're responding to.
The actual site did not have it at difficulty 1.
Literally discussed throughout the whole comment section.
That doesnt mean the PoW is doing anything, it might just mean bots have js disabled.
I dont think the person was claiming annubis doesnt work, they were disputing PoW is the reason it actually works.
Yes, Anubis is just non standard and obscure, the proof of work bit is completely irrelevant (except for getting people on their phone to not visit your website).
Shouldnt browser also have it implemented in c? Like i assume crypto.subtle isnt written in js.
It doesn't matter if your hottest loop is using string comparisons, as another poster pointed out in C you aren't even doing the majority of the second hash because you know the result (or enough of it) before finishing it. The JavaScript version just does whole hashes and turns them into a Uint8Array, then iterates through it.
I dont understand what you mean. Are you saying that in C you only calculate the first few digits of the hash? That's not how sha256 works.
Edit: oh i think you mean in c the string comparison short curcuits. I would expect the same to be true in javascript too. Its true in most languages.
Maybe you are just worried about general language overhead, which is a fair point. Is the anubis check even using multiple threads? For the c case, the real benefit wouldn't be if you can use C, but if you can use the GPU.
The whole thing is kind of silly though. SHA256 is a terrible choice of hash for PoW. They should be using argon2 or something memory heavy.
The language matters, but your original guess was actually correct, you can do tricks with sha256 where you only end up calculating a fraction of the total double hash in order to get a pass or fail.
Modern bitcoin miners do a double sha256 hash and increment in just a little bit more than a single hash of work. The input is 80 bytes, which is two compression rounds of 64 bytes in sha256, only the data in the second round has changed (the appended nonce), so you don’t bother doing the first compression round again. With other quirks you can end up doing multiple hashes at once “asicboost” due to partial collisions within the input too.
Oh good point. It looks like anubis is using 64 byte random plus a nonce. Unless i'm missing something it seems like using 56 bytes random plus 8 byte nonce would be a better design or maybe hmac'ing the nonce & data together.
But then again using sha256 doesn't make sense at all.
Maybe post your brilliant solution to commercial companies with hundreds of millions in funding unrestrained bot scraping the Internet for AI training instead of complaining about people desperate to rein it in as individuals.
Anybody can prompt Claude to implement this, which was my point, it doesn't stop bots because a bot can literally write the bypass! My prompt was the proof of work function from the repository, asked it to make an implementation in C that could solve it faster, and that was about it.
This is fallacious and extremely disrespectful (or even malicious?). You don't have to propose a way to fix a broken thing to point out that it's broken.
Normal and sane people understand this intuitively. If someone goes to a mechanic because their car is broken and the mechanic says "well, if you can tell that you car is broken, then you should be able to figure out how to fix it" - that mechanic would be universally hated and go out of business in months. Same thing for a customer complaining about a dish made for them in a restaurant, or a user pointing out a bug in a piece of software.
At this point I wonder if you can post a crypto miner page on HN and people will fall for it.
I don't get this kHash thing. Do we have captchas mining bitcoin in a distributed fashion for free now?
The page says
> Anubis uses a Proof-of-Work scheme in the vein of Hashcash
And if you look up Hashcash on Wikipedia you get https://en.wikipedia.org/wiki/Hashcash which explains how Hashcash works in a fairly straightforward manner (unlike most math pages).
Oh fun so now we're effectively draining users' phone and laptop batteries now just to prove that they have batteries and somehow that's a proxy for them being human
Given that you're clearly completely ignorant of everything in the anti-spam space, you should probably do some research before making uninformed comments like this.
On what page? https://gladeart.com/blog/the-bot-situation-on-the-internet-... loaded effectively instantly for me.
The cynic in me thinks that they’re mining bitcoin on our phones… And after completing, it claimed the page was misconfigured.
I think we got honeybotted.
I'm surprised at the effectiveness of simple PoW to stop practically all activity.
I'll implement Anubis at low difficulty for all my projects and leave a decent llms.txt referenced in my sitemap and robots.txt so LLMs can still get relevant data for my site while.keeping bad bots out. I'm getting thousands of requests from China that have really increased costs, glad it seems the fix is rather easy.
>I'm surprised at the effectiveness of simple PoW to stop practically all activity.
It's even dumber than that, because by default anubis whitelists the curl user agent.
vsshhhh don't tell the bots !
Anubis' white lists and block rules are configurable though. The defaults are a bit silly.
The default is to allow non-Mozilla user agents so that existing (good) automation continues to work and so that people stopped threatening to burn my house down. Lovely people in the privacy community.
Sorry xe :(
It's definitely more than enough to stop me as a human wanting to visit the site, so yeah.
In that case a better solution would be to take the site down altogether.
Take down the site entirely because a couple humans get into a fit about it?
I'm just saying, making visitors wait at least a minute while making their device turn red hot is going to stop 99,9% of your visitors. So at that point what's the point in trying to serve the content?
The site's down entirely anyway. The silly "proof of work" finishes only to tell me the site is down.
What a waste of time.
> These bots are almost certainly scraping data for AI training; normal bad actors don't have funding for millions of unique IPs thrown at a page. They probably belong to several different companies. Perhaps they sell their scraped data to AI companies, or they are AI companies themselves. We can't tell, but we can guess since there aren't all that many large AI corporations out there.
Is the theory here that OpenAI, Anthropic, Gemini, xAI, Qwen, Z.ai etc are all either running bad scrapers via domestic proxies in Indonesia, or are buying data from companies that run those scrapers?
I want to know for sure. Who is paying for this activity? What does the marketplace for scraped data look like?
I agree it's a more than a bit handwavy. The common consensus seems to be that AI companies are driving this, but it's really hard to conclusively prove who or what is behind the attacks.
Weird part #1 is that the traffic isn't for the most part shaped like crawler traffic. It's incredibly bursty, and heavily redundant, missing even the most obvious low hanging fruit optimizations.
Could be someone is using residential proxies to wrap AI agents' web traffic, but even so, there's a lot of pieces that don't really make sense, like why the traffic pattern is like being hit by a shotgun. It isn't just one request, but anywhere between 40 and 100 redundant requests.
A popular theory is that this is because of sloppy coding, AI companies are too rich to care, but then again that doesn't really add up. This isn't just a minor inefficiency, if it is "just" bad coding, they stand to gain monumental efficiency improvements by fixing the issues, in the sense of getting the data much faster, a clear competitive edge.
Really weird.
My unsubstantiated guess is the residential proxy/botnet is very unreliable, and that's why they fire so many request. Makes sense if it's sold as a service.
I suspect the redundant requests are primarily designed to weed out poisoned data served on otherwise valid URLs. I've also seen the redundant requests increase massively the more sources I blocked at the firewall level, so it feels like they're pre-emptively overcompensating for some percentage of requests being blocked.
My website contains ~6000 unique data points in effectively infinite combinations on effectively infinite pages. Some of those combinations are useful for humans, but the AI-scrapers could gain a near-infinite efficiency improvement by just identifying as a bot and heeding my robots.txt and/or rel="nofollow" hints to access the ~500 top level pages which contain close to everything which is unique. They just don't care. All their efficiency attempts are directed solely toward bypassing blocks. (Today I saw them varying the numbers in their user agent strings: X15 rather than X11, Chrome/532 rather than Chrome/132, and so on...)
> A popular theory is that this is because of sloppy coding, AI companies are too rich to care, but then again that doesn't really add up
I can substantiate this a bit. Verified traffic from Amazonbot is too dumb to do anything with 429s. They will happily slam your site with more traffic than you can handle, and will completely ignore the fact that over half the responses are useless rate limits.
They say they honor REP, but Amazonbot will still hit you pretty persistently even with a full disallow directive in robots.txt
How do you know it's Amazonbot?
User Agent, SWIPed IP space, and the PTR records resolving to an Amazon-controlled crawl zone.
I want more data too.
The root sources of the traffic from residential proxies gets murky very quickly.
It's easy to follow the chain partway for some traffic, eg "Why are we receiving all this traffic from Digital Ocean? ... oh, it's their hero client Firecrawl, using a deceptive UserAgent" ... but it still leaves the obvious question about who the Firecrawl client is.
Res proxy traffic is insane these days. There is also plenty of grey-market snowshoe IPs available for the right price, from a handful of ASNs. I regularly see unified crawling missions by unknown agents using 1000+ "clean" IP addresses an hour.
https://parallel.ai/
I bet lot of companies want to provide search results to AI agents.
> Before it was enabled, it was getting several hundred-thousand requests each day. As soon as Anubis became active in there, it decreased to about 11 requests after 24 hours
I love experimental data like this. So much better than gut reaction that was spammed when anubis was just introduced
Well yeah, but I also didn't make it through to the actual site. That can't be the idea, right? After 5 seconds of 100% CPU and no progress I gave up.
The idea is to scare off bots and not normal humans.
May be you are a bot /s
Yes my name is Anna
On my computer, with Firefox it uses 14 CPU cores, consumes an extra 35 Watts, and the progress bar barely moves. Is this site mining cryptocurrency?
On Safari or Orion it is merely extremely slow to load.
I definitely wouldn't use any of this on a site that you don't want delisted for cryptojacking.
I'm getting this patern a lot on Prestashop websites, where thousand, to not say hundreds of thousand, of request are coming from bots not announcing themselves in the User-agent, and coming from different IP's
Very annoying. And you can't filter them because they look like legitimate trafic.
On a page with differents options (such as color, size, etc...) they'll try all the combinaisons, eating all the ressources.
We started building out a set of spam/fraud/bot management tooling. If you have any decent infrastructure in place already, this is a pretty manageable task with a mismash of techniques. ASN based blocking (ip lookup databases can be self hosted and contain ASN) for the obvious ones like alibaba etc, subnet blocking for the less obvious (see pattern, block subnet, alleviates but doesn't solve problems).
If you have a logging stack, you can easily find crawler/bot patterns, then flag candidate IP subnets for blocking.
It's definitely whackamole though. We are experimenting with blocking based on risk databases, which run between $2k and $10k a year depending on provider. These map IP ranges to booleans like is_vpn, is_tor, etc, and also contain ASN information. Slightly suspicious crawling behavior or keyword flagging combined with a hit in that DB, and you have a high confidence block.
All this stuff is now easy to homeroll with claude. Before it would have been a major PITA.
> safari can't open the page
What is the point of these anti bot measures if organic HN traffic can nuke your site regardless? If this is about protecting information from being acquired by undesirable parties, then this site is currently operating in the most ideal way possible.
The information will eventually be ripped out. You cannot defeat an army with direct access to TSMC's wafer start budget and Microsoft's cloud infrastructure. I would find a different hill to die on. This is exactly like the cookie banners. No one is winning anything here. Publishing information to the public internet is a binary decision. If you need to control access, you do what Netflix and countless others have done. You can't have it both ways.
Is Anubus being set to difficulty 8 on this page supposed to be a joke? I gave up after about 20 seconds.
I think that must be the point they're trying to make, yes
It also drives home that Anubis needs a time estimate for sites that don't use Anubis as a "can you run javascript" wall but as an actual proof of work mechanism that it purports to be its main mechanism
It shows a difficulty of "8" with "794 kilohashes per second", but what does that mean? I understand the 8 must be exponential (not literally that 8 hashes are expected to find 1 solution on average), but even as a power of 2, 2^8=256 I happen to know by heart, so thousands of hashes per second would then find an answer in a fraction of a second. Or if it's 8 bytes instead of bits, then you expect to find a solution after like 8 million hashes, which at ~800k is about ten seconds. There is no way to figure out how long the expected wait is even if you understand all the text on the page (which most people wouldn't) and know some shortcuts to do the mental math (how many people know small powers of 2 by heart)
I waited a minute until my phone got hot.
So the elephant in the room: How much of HN is bot generated? Those who know have every incentive not to share and those who don't have no way to figure it out. At this point i have to assume that every new account is a bot
The article is about automated web scraping, not bots writing content.
The commenters here don't care what the article is about when they can't access the article and the much more concerning question not about web scraping is.
I felt a vibe change, some are obvious and some not, but it does feel different, the main change i've seen is in downvotes, I don't say very controversial things and have had many things very quickly downvoted, and then slowly upvoted, I think hn was very slow to downvote in the past (except obvious trolls/spam). So for me the main worry is not even the comments, but the invisible bias generated by voting.
I've noticed that too. I thought I was just posting particularly bad takes.
That's a good point ... it's trivial to have an agent post something onto HN on your behalf, so even old accounts are not immune. It's just the nature of things now, until we get better technology to assign some sort of "ALIVENESS" attestation to folks without revealing identity too much.
I've thought about this a bit and I can't really see why someone would want to write AI content here other than to spam ads but they are handled quickly. Does anyone see AI content with a clear motivation or agenda here? There are very few rep based privileges right so that seems like an unlikely motivation as well.
Most of the HN bot accounts I see have a link-to-vibecoded-product in bio, and/or are trying to build up "organic" activity before a Show HN post for the same.
A less publicly-visible motive would be if they were building up accounts to use for paid-upvote schemes.
You can automate shilling to drive or at least influence opinion.
This is a venture capitalist driven community that attracts the sleaziest kind of spammers you could think of under the badge of growth hacking and networking. Besides this very obvious motivation to spam you have all kinds of nerds here eager to do it just because they can (on one of the most famous tech places where registration is made as easy as possible)
> Those who know have every incentive not to share
Why do you say that?
I think HN is one of the better ones these days. I have no data to back this up, but the comments aren't like reddit comments. Go into any reddit post on the main subs, and you won't have to scroll very far to get a comment about Trump derailing the whole thing.
Digg's recent shutdown message talked about how bad and aggressive bots were. I'd love to see Kevin and Alex post in depth about lessons learned, Dead Internet, and call out social sites.
> let webWorkerURL = `${options.basePrefix}/.within.website/x/cmd/anubis/static/js/worker/sha256-${workerMethod}.mjs?cacheBuster=${options.version}`;
It looks like it's computing sha256 hashes. Such an ASIC friendly PoW has the downside that someone with ASICs would be able to either overwhelm the site or drive up the difficulty so high that CPUs can never get through.
Hugged to death?
https://web.archive.org/web/20260329052632/https://gladeart....
Thanks! I was wondering if there was an actual site behind it of if it was just a joke.
honestly this should be updated to the main link, the Anubis at difficulty 8 is astonishingly hostile
Added above. Thanks!
> we tend to take anti-bot measures very seriously
Should have have maybe prioritized differently...
Maybe my imagination is just too accurate but this didn't tell me anything I didn't expect to hear.
> Here is a massive log file for some activity in the Data Export tar pit:
A bit of a privacy faux pas, no? Some visitors may be legitimate.
Looks like they've gone ahead and implemented the easiest fool-proof method of preventing scraping as the site is currently not loading across mutliple devices.
Not even a 404, just not available at all.
Not sure what they are doing, but they don't seem to do it well.
> The IPs of these bots here actually do not come from datacenters or VPNs most of the time; the overwhelming majority come from residential and mobile networks.
So I started searching for what these residential proxy networks actually are.
https://datadome.co/bot-management-protection/how-proxy-prov...
An interesting and sad aspect of the war on bots and scraping that is being waged is that we are hurting ourselves in the process, too. Many tasks I'm trying to get my AI assistant to do cannot be done quickly, because sites defensively prohibit access to their content. I'm not scraping: it's my agent trying to fetch a page or two to perform a task for me (such as check pricing or availability).
We need a better solution.
You aren’t scraping for the sake of training a model, but scraping the prices and availability is still scraping, right?
I think some of the folks running sites would rather have you go to the site and view the items “suggested based on your shopping history” (I consider these ads, the vendors might disagree), etc.
I’m more sympathetic to the people running sites than the LLM training scrapers, but these are two parties in a many-party game and neither one is perfectly aligned with users.
> scraping the prices and availability is still scraping
Web browsing is scraping, too.
I am not doing anything that I myself wouldn't do, it would just take me longer. I'm not mass-scraping, training new models, etc etc. I'm just using a helper tool to do some work for me.
If you prevent that, you are effectively saying: humans have to perform the manual labor of clicking and browsing through our site, they are not allowed to be helped in any way. I don't think this is the right answer.
I would assume most sites that block access to your AI assistant do so because they want to show a human ads, i.e. not run at a loss. Seems reasonable.
My grad school research was on computational models of human/machine cognition, and I'm now commercializing it as a 'proof-of-human API' for bot detection, spam reduction, and identity verification.
One of the mistakes people assume is that AI capability means humanness. If you know exactly where to look, you can start to identify differences between improving frontier models and human cognition.
One concrete example from a forthcoming blog post of mine:
[begin]
In fact, CAPTCHAs can still be effective if you know where to look.
We ran 75 trials -- 388 total attempts -- benchmarking three frontier AI agents against reCAPTCHA v2 image challenges. We looked across two categories: static, where each image grid is an individual target, and cross-tile challenges, where an object spans multiple tiles.
On static challenges, the agents performed respectably. Claude Sonnet 4.5 solved 47%. Gemini 2.5 Pro: 56%. GPT-5: 23%.
On cross-tile challenges: Claude scored 0%. Gemini: 2%. GPT-5: 1%.
In contrast, humans find cross-tile challenges easier than static ones. If you spot one tile that matches the target, your visual system follows the object into adjacent tiles automatically.
Agents find them nearly impossible. They evaluate each tile independently, produce perfectly rectangular selections, and fail on partial occlusion and boundary-spanning objects. They process the grid as nine separate classification problems. Humans process it as one scene.
The challenges hardest for humans -- ambiguous static grids where the target is small or unclear -- are easiest for agents. The challenges easiest for humans -- follow the object across tiles -- are hardest for agents. The difficulty curves are inverted. Not because agents are dumb, but because the two systems solve the problem with fundamentally different architectures.
Faking an output means producing the right answer. Faking a process means reverse-engineering the computational dynamics of a biological brain and reproducing them in real time. The first problem can be reduced to a machine learning classifier. The second is an unsolved scientific problem.
The standard objection is that any test can be defeated with sufficient incentive. But fraudsters weren't the ones who built the visual neural networks that defeated text CAPTCHAs -- researchers were. And they aren't solving quantum computing to undermine cryptography. The cost of spoofing an iris scan is an engineering problem. The cost of reproducing human cognition is a scientific one. These are not the same category of difficulty.
[end]
How does your software work with blind people like me who use screen readers?
Your key finding is that humans process the grid as one visual scene — but that's a finding about sighted cognition.
Isn't this, like most things, a sensitivity specificity tradeoff?
How many real humans should be blocked from your system to keep the bots out?
What is the Blackstone ratio of accessibility?
>The first problem can be reduced to a machine learning classifier. The second is an unsolved scientific problem.
I can't believe people are still using this as a generic anti-AI argument even though a decade ago people were insisting that there's no way AI can have the capabilities that frontier LLMs have today. Moreover it's unclear whether the gap even exists. Even if we take the claim that the grid pattern is some sort of fundamental constraint that AI models can't surpass, it doesn't seem too hard to work around by infilling the grids pattern and presenting the 9 images to LLMs as one image.
> “…reverse-engineering the computational dynamics of a biological brain and reproducing them in real time…”
Is not an anti-AI argument, it’s an open and unsolved question. Your optimism is appreciated, but the dismissal and assumption this is already solved is foolish and naive.
> The IPs of these bots here actually do not come from datacenters or VPNs most of the time; the overwhelming majority come from residential and mobile networks. Asian and Indonesian countries are where nearly all of them reside.
It's really awful as an indonesian, my indonesian isp regularly got blocked by HN as well :(
My blog gets this degree of scraping too. I have some 5 million requests over the same period as they say they got 7 million over and I barely noticed before I put Cloudflare in front to cache and now I don’t notice at all. I have the Cloudflare AI stuff turned off and mostly use it through the tunnel so I don’t have to expose my local IP.
Is this actually a problem? Most of my requests claim to be Amazonbot but someone showed me they weren’t and I’ve forgotten how.
You can build some great anti-bot mechanisms with simple https://github.com/abrahamjuliot/creepjs logic. A normal user will often appear 31% or lower 'like headless score', mobile is a bit different. You'll still have trouble against sophisticated infra: https://x.com/_alexspring/status/2037968450753335617
Looks like Anubis is also blocking robots.txt which seems to defeat the point of having robots.txt in the first place.
Update: looks like this has been fixed now.
At first glance this seems like a crypto miner.
Maybe I’m a bot, I gave up waiting before the progress bar was even 1% done.
>How can you protect your sites from these bots?
JA4 fingerprinting works decently for the residential proxies.
TLS fingerprinting is not sufficient to stop residential proxies, the proxy acts as a transparent pass-through at the TLS layer making it trivial to use something like curl_cffi to mimic a real browser TLS fingerprint.
However residential proxies do have a weakness, since they need to maintain 2 separate TCP conenctions you can exploit RTT differences between layers 3 and 7 to detect if the connection to your server is being terminated somewhere along the path. Solutions exist that can reliably detect and block residential proxies, for example: https://layer3intel.com/tripwire
Cool link thanks for sharing. Too bad it is so expensive. Maybe I’ll find a way to DIY.
The user agents in that screenshot are fake, nobody would be running Chrome 106 on windows 10... run a php script on every page that checks for valid combinations and 400 the rest.
Worse than I could imagine? I imagine that bots might destroy the internet. Not just the internet as we know it; I mean make the internet completely unusable to any human being.
So many Anbuis comments here. There are various Anubis bypass extensions available. I haven't seen Anubis prompts in years.
I'm willing to wait a while for pages to load if it's an effective means of taking back the web from the bots.
I don't know if they have issue with my ff+ubo, but it is almost a minute that anubis is blocking me. screw them.
As soon as I see that anime bot thing which this website is using I close the tab. More annoying than Cloudflare.
Quote:
> "The idea is that at individual scales the additional load is ignorable, ..."
Three minutes, one pixel of progress bar, 2 CPUs at 100%, load average 4.3 ...
The site is not protected by Anubis, it's blocked by it.
Closed.
noticed that firefox gives 2x kHashes/s more than chrome (1000 vs 500)
Taking a 2024 report on bot loads on the Internet is like taking a 1950s Car & Driver article for modern vehicle stats.
That’s how fast the landscape is changing.
And remember: while the report might have been released in 2024, it takes time to conduct research and publish. A good chunk of its data was likely from 2023 and earlier.
Yea it's pretty bad
I cannot get past the bot check (190kH/s), is it mining crypto on my laptop?
> NOTE: Use a VPN on these pages if you don't want your IP shown in the logs, but it won't be significant amongst the millions of others anyways
Is this supposed to be a joke? Is the author expecting users to travel back in time and use a VPN so their IP is scrubbed from logs that will get published at any time, because that's something the author just obviously has the right to do?
> The EDPB explicitly identifies IP addresses as being personal data due to their ability to identify individual data subjects.[1]
Dickhead.
[1]: https://techgdpr.com/blog/is-an-ip-address-considered-person...
What a great way to not get any traffic at all.
Everybody says that bots put websites down, while marketing oriented folks start practicing AO (agent optimization) - to make their offerings even more available and penetrating.
Good luck banning yourself from the future.
This sounds like something a bot would say.
Employ constant faceID can deter it
Please drink verification can.
Ok. So I get a page saying it’s verifying I’m not a bot with some kink of measurements per second and I don’t get through. Is that the point?
She's definitely a bot with some kink!
I don't get what it is or whether it's a satire or not.
If a webstie takes so long to verify me I'll bounce. That's it.
I very very very much doubt that lol
I know / we know lol
The final Eternal September
This is why I see (well managed) government digital IDs as sensible moves. Apart from DDOS attacks, if bots have to “prove” who they are on each request it seems like a win-win.
I may be missing something of course
If you want “papers, please” every time you back out of your driveway or go beyond your government-assigned oblast, then your suggestion is the digital version of the physical authoritarian nightmare that was imposed by totalitarianist regimes throughout history.
People have a right to complete anonymity, and should be able to go across the majority of the Internet just as they can go across most of the country.
That’s what you are missing.
Don’t get me wrong, I am also in favour of a single government ID, but in terms of combatting identity fraud, accessing public resources like single-payer healthcare, and making it easier for a person to prove their identity to authorities or employers.
It should not be used as a pass card for fundamental rights that normally would have zero government involvement.
>>> People have a right to complete anonymity
Why? (Am not trolling. Genuinely interested)
I walk out my front door in the UK and I am not anonymous. Every transaction I make either identifies me through bank, railway or other id, or quite simply by my face standing in front of the coffee seller. My walk down the road is observed by neighbours and postmen.
Should my government arrest me without cause or trample on my free speech rights, I get that’s a problem but I am not sure why being anonymous helps. Having rights upheld by the courts helps, well trained police who respect the law helps.
I am honestly open to debate on this but I do find the “what if Hitler took over government where would we be” to be a problematic argument not a final answer.
> Should my government arrest me without cause or trample on my free speech rights, I get that’s a problem but I am not sure why being anonymous helps. Having rights upheld by the courts helps, well trained police who respect the law helps.
You're suggesting the same government that would violate your rights would then help prevent it? I don't follow. Any power structure (tiered or not) was wiped away by authoritarians, historically. They will not be helping in the worst case. Ideological capture (corruption) has already started eroding at UK rights and that took a much less overt effort. America has had a robust 3-branch system (executive, legislative, judiciary) corrupted by a singular cult of personality. THAT was highly unlikely to happen, but here we are.
With this being said, I do predict that anonymity on the web is going to be phased out. It will result in all sorts of changes to cultural norms across western nations that largely will curtail rights. I dread it.
Shouldn't we try tracing IP addresses and fining organizations for letting the traffic through or originating the traffic first? Seems a lot simpler.
Sorry maybe I should be clearer - the problem of tyrannical governments is not solved by being anonymous online, or indeed any technology that makes it hard for government to do the tyrannical things. Safety lies in an engaged citizenry that reacts to fundamental threats. The protests against the ICE in Minnesota being an example.
>>> It will result in all sorts of changes to cultural norms across western nations
I quite agree - but I (hope / think) that the benefits can outweighs the downsides if done well. Those nations that do it well will I believe find a rocket like boost to society and industry perhaps akin to post 1945 world. Those who don’t will fall behind.
> Every transaction I make either identifies me through bank, railway or other id, or quite simply by my face standing in front of the coffee seller. My walk down the road is observed by neighbours and postmen.
Are these the government? Is the bank the government? Is the rail company the government?
No? Then you have answered your own question.
A silo of identification between you and a service provider that uses the provider’s own tooling is still anonymity from government authoritarianism.
The fact that nearly all of these silos are leaky IRL - with the government eager to punch howitzer-sized holes through them for even more access - is not the point. It is a citizen-hostile flaw that needs patching through loophole-proof legislation, not an ID system that would violently eradicate any remaining separation of government from capitalism.
Remember: when government and capitalism rides in the same cart, it is called corporatism, and is the basis of Fascism. Which is what is happening to America.
The problem here is that pretty much every part of modern life has been government and capitalism riding in the same cart - from cities installing electric power stations 100 years ago, to roads and inventions like the transistor and internet itself was government and private capital working towards common goals.
The issue is we want “good” government and “good” corporate behaviour but not the bad. And knowing the difference especially ahead of time requires engaged citizenry, lots of feedback mechanisms that are not overwritten by corruption and noise in the mechanism (ie primaries materringnmore than elections is a feedback mechanism fail in my book)
> The problem here is that pretty much every part of modern life has been government and capitalism riding in the same cart - from cities installing electric power stations 100 years ago, to roads and inventions like the transistor and internet itself was government and private capital working towards common goals.
And here is your argument’s fatal flaw: mistaking public works, that benefits the commons and society as a whole, with government tracking.
The two are NOT the same.
Just a sec - you said: “”” Remember: when government and capitalism rides in the same cart, it is called corporatism, and is the basis of Fascism. ”””
The implication being when gov and big business get together bad things do happen. My point is they can happen yes, but also same players can make good things happen - it depends on the players (and regulatory and reporting and voting and and and )
Government tracking can be good - tracking covid cases good, tracking criminals planning robberies good. It it can also be bad.
This is imo why this is such hard problem - there is no clear answers only something we want sensible courts to (quickly) decide upon based on good laws.
/s?
You may be missing that it's easy and free for website owners to fix the problem. But it's hacker news after all. If somebody is bothered by a leaf falling on them on their walk to the corner store, the suggested solution here will be to have a full communist revolution.
It is absolutely not free or easy to stop bots.
Millions upon millions of people use Cloudflare to stop bots for free, and there are other alternatives as well. It's incredibly easy. So no, there's just as much need for government intervention, as there is a need for the government to stop leaves from falling on you when you're walking to the corner store.