This is really unacceptable folks. There are those of us that have to keep these sites up, and it's seriously been a few years of nightmare scrapers and botnets, and stupid things like this that you are trying to legitimize that will make this worse. If a site doesn't want you, you should go away. There's a reason for it. Not every website is backed by a billion/trillion dollar company with the resources to absorb things.
> Bot detectors flag automation by reading the browser fingerprint; Fortress corrects that fingerprint inside Chromium's C++, so the browser presents as an ordinary Chrome install.
This does not seem like it would work against anything but the most basic bot protection.
Man, this sucks. I doubt there’s anything that I can say to get people to stop doing things like this, but the eventual outcome here isn’t going to be freedom for you to scrape sites that are trying to avoid being DDoSed by bots, but instead that we all end up in a world where device attestation is required to do practically anything online. And for what?
Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them down because the bots are relentlessly hammering their sites and driving up hosting costs.
I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.
There actually is a well defined demarcation point where the site owners wishes do apply. It's called... A demarc point! Everything that happens past this point is by definition entirely in the control of the admin. Which includes closing the door on whoever they wish. This is a technical reality, not something you can effect with opinions or wanting things more than other people want other things. It might seem contradictory but this is actually the main thing that makes the internet as free as it is.
When we're talking about ethics, I think we should give the owner more than that. If they don't want to be DDoSed we shouldn't say "too bad, the attackers are outside your network, your wishes don't apply, get good".
I think more like mimicking the perfect resident. Then again it’s Theseus’s ship no, if you are able to do something so perfectly what is the difference between you and the clone? A Hollywood movie in the making.
Since my grade-school mascot/team was "The Trojans" and my adopted hometown is firmly rooted in pre-Christian Greek culture and mythology, I am currently studying the Trojan War, and I may observe that "Trojan" is an extremely perspicacious brand-name for condoms, more than the average guy would expect.
This is really unacceptable folks. There are those of us that have to keep these sites up, and it's seriously been a few years of nightmare scrapers and botnets, and stupid things like this that you are trying to legitimize that will make this worse. If a site doesn't want you, you should go away. There's a reason for it. Not every website is backed by a billion/trillion dollar company with the resources to absorb things.
Things like this make me wish that we have to pass ethics courses to work in tech.
> Bot detectors flag automation by reading the browser fingerprint; Fortress corrects that fingerprint inside Chromium's C++, so the browser presents as an ordinary Chrome install.
This does not seem like it would work against anything but the most basic bot protection.
Man, this sucks. I doubt there’s anything that I can say to get people to stop doing things like this, but the eventual outcome here isn’t going to be freedom for you to scrape sites that are trying to avoid being DDoSed by bots, but instead that we all end up in a world where device attestation is required to do practically anything online. And for what?
Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them down because the bots are relentlessly hammering their sites and driving up hosting costs.
I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.
Why do you feel entitled to navigate to a website you neither own nor pay for via a method the owner expressly forbids?
Why do you feel entitled to dictate what user agent I use, if it’s well-behaved?
Why do your opinions about what a "well-behaved" agent is override the wishes of a site's owner?
Site owners should not have full control over what users do. Their wishes do not apply past certain boundaries.
For a non-AI example I run into occasionally, any wish to stop time-shifting should not be relevant to what I choose to do as a media consumer.
There actually is a well defined demarcation point where the site owners wishes do apply. It's called... A demarc point! Everything that happens past this point is by definition entirely in the control of the admin. Which includes closing the door on whoever they wish. This is a technical reality, not something you can effect with opinions or wanting things more than other people want other things. It might seem contradictory but this is actually the main thing that makes the internet as free as it is.
When we're talking about ethics, I think we should give the owner more than that. If they don't want to be DDoSed we shouldn't say "too bad, the attackers are outside your network, your wishes don't apply, get good".
> Is it ethical to scrape when a site has explicitly blocked bots
Ethical? ppl get jailed in China for this. "Breaching computer systems" is a felony.
It's most definitely unethical.
odd name; shouldn't it be named after something that stealthily infiltrates fortresses?
fellow contributor on this.
we thought of it the other way round: your automation is the fortress, and every bot-detector trying to fingerprint it is the siege.
also most good burglar names were taken on PyPI
in my defence, i did lobby hard for "GuyWhoWavesYouThroughTheGate"
"Doorman"
I think more like mimicking the perfect resident. Then again it’s Theseus’s ship no, if you are able to do something so perfectly what is the difference between you and the clone? A Hollywood movie in the making.
"Of course... Why didn't I think of that?"
Trojan Horse probably wouldn't make a very good name for a browser.
Since my grade-school mascot/team was "The Trojans" and my adopted hometown is firmly rooted in pre-Christian Greek culture and mythology, I am currently studying the Trojan War, and I may observe that "Trojan" is an extremely perspicacious brand-name for condoms, more than the average guy would expect.
Trojan Condoms: We will “trick” the defenses 100% of the time. Wait wrong take.
which is exactly the problem, a condom is famous for stopping things getting through and we do the literal opposite.