If you read the article, you'd find they admit to that, their position is nuanced, and this is not a "gotcha".
> We'll be honest about where we stand. We don't like copyright as it exists. We'd like to see it radically reformed, and we wouldn't mourn it if it were abolished. Our community lives in its grey areas every day: many of the models on our sites are fan ports of characters from commercial games, made out of love and shared for free. When a studio sends us a takedown notice, we comply.
> While we as small artists and fan communities get copyright used against us, the world's largest companies are committing copyright infringement at a scale unseen in human history. This has been the case since the invention of the internet of course; digital media make duplication, remixing and resharing trivial, but copyright law never caught up. Filesharing stabbed copyright in the gut. AI is just twisting the knife, and completes the creation an unequal system where copyright infringement is only a crime if you're a low net worth individual.
Rich companies are infringing copyright at industrial scale in order to build their LLMs, and even if caught and challenged, they don't suffer any catastrophic punishment. The infringed party gets enough money to shut them up and everyone moves on. A fine is a price. They'll keep doing it with impunity. Meanwhile the small guy who doesn't have copyright lawyers on retainer is getting shafted, time and time again.
Considering you link something that has a non comnercial license CC-by-NC, there might be others with permissive licenses. There might be things in their TOS of the site,
There's more with different licenses. If you scroll through the pages, there are a lot of assets ripped from games, with a variety of licenses CC-by-NC, or just "beerware":
Scroll through the site, there are A LOT of assets from games, packaged by users. It's a nice subtrefuge, in practice it is a place to get ripped assets and the platform washes it's hands.
Again, I don't have any particularly strong issue with this, just with the pearl clutching regarding roblox downloading the data.
Scientists say this is the first occurance of the word "model" in the title of a post on HN that does not refer to a neural network of some sort since 2023
Only tangentially related, but I've recently realized that unless you're some sort of business with a valid requirement of that, you can just block incoming traffic from all of AWS, DigitalOcean, Azure, etc.
I think the spark came from some other HN comment I saw? So thank you for that, whoever wrote that.
That alone is not a sufficient strategy of course, but you can get rid of a lot of bots by just refusing to talk to where they tend to live.
Of course, if everyone were to do that, they'd eventually pivot to residential proxies. But I don't see that happening.
I initially thought this was a good idea too until another comment pointed out you'll lose most business or personal customers in offices because they connect through VPN gateways running on those cloud services (especially Azure)
I would argue that more often than not it might not. Really depends on the specifics though.
A lot of stuff on the web actually isn't really relevant for all audiences everywhere I'd say. It just happened to be accessible by all, because academics back in the day dreamt of a better open world that didn't really manifest :(
Let me tell you, at the business I work at, some of our customers work at a company that uses a commercial VDI service for all workstations couldn’t access our service because the VDI egress was in GCP and we had an AWS WAF rule banning datacenter IPs.
Obviously, we removed the rule because it was doing nothing useful but plenty of harm. Real bots come from residential IPs.
Same, I just checked and this week the vast majority of the bot traffic I get is random residential IPs that all pretend to be the exact same Chrome 155 on Linux.
Then you are not looking. Already have to switch to resi proxies from time to time when encountering yet another smartass who decided to play security theater and block datacenter ips on his shitpile he calls website, because "OUR USERS MUST BE GOOD BOYS WHO LOVE TO LIVE UNDER SURVILANCE AND ALLOW ISP TO TRACK THEM 24/7" (and before you say something like "others can track you too!", your ISP has the ability to knock on your door, they should be given as little information as possible, let them do metadata analysis on per-person basis if they really want it, but no 1-click mass surveillance shit).
If you are valuable, you will be be scraped. Turning user hostile in a hopeless endeavor to "protect yourself" is not the solution.
I can see that you're having strong emotions about this, but that rant is just hard to read and serves primarily as venting and not really to convince anyone of anything.
Unless one zooms out and thinks "okay, these guys seem to react negatively if we do exactly that. We should do more of it", which is probably not what you want.
Nobody cares what you do, you are not smart enough. If you even entertain the idea of "let's just block datacenter ips!", all you accomplish by doing that is showcasing your incompetence and lose all respect from legitimate users who are good at tech. The scraper will just turn the resi proxy and will continue as is.
But maybe selecting for tech illiterate cattle as your userbase is what you want, then I guess more power to you. Don't be surprised by the scorn though.
I trust my ISP far more than I trust bigtech. The downside of connecting from my ISP is that the site I connect to can fingerprint me trivially from the IP range
I'm sympatic to open3dlab but I think the focus on adult-content is just trying to spark outcry for cheap.
It has been shown times and times against that removing adult content from the training source (at least for diffusion models), results in generally lower-quality output, even outside of adult-content, particularly in terms of representing human bodies.
would the implied intent (from the inclusion of adult content) of training models on open3dlab content, without asking them, render Roblox's actions _less_ antisocial???
The current AI Ethos is morality is just another obstacle on the way to their IPO. ...Lets just hope after they get rich, they stay away, and dont bother us with books, videos and conference speeches on entrepreneurship mentoring...
> The crawler did not respect our robots.txt, (which it did retrieve), ignored our Terms of Service, (which forbid scraping), and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only.
The news is that people download it so much that they saturate your server’s connection and/or peg its CPU or disk (whichever is the weakest link in the chain). We used to call this a Denial of Service attack but now it’s apparently just a cost of doing business, even for people who just want to run a non-commercial hobby site.
Man if only there was such a thing as good manners, good behaviour, accepted practices from a community to not massively hammer servers. If only these websites had been up for a decade with very few problems, before a notoriously manipulative, lying, deceiving, abusive company that makes money off of child abusers came in and just went hurr durr free data in the WORST possible way instead of just politely asking for a dump.
we've spent the entire 2010s with lots of talk whether creating a scraper for google or linkedin and packaging that data to sell is illegal just for the entire thing to turn around where large companies are basically burning small servers down with requests to do the exact same thing
maybe stupid question but how is it possible to not be protected against supercrawlers like this, when you own large amounts of data, specially if all requests are coming from the same server ? Aren't cyberprotection services like CloudFlare work against this kind of "attacks" ?
Yeah, I can imagine it being a problem if it was from different IP addresses, but surely putting a limit on requests from a single IP would be a straightforward fix? Although maybe until it happens, you don't think about it.
I mean, it's not the size of the download, it's the unauthorised scraping of community assets strictly against terms and ignoring robots.txt by an American corporation. Resource of a kind that used to get kids threatened with spending most of their life in prison.
But I agree that this kind of theft by US corporations is no longer news.
(Yes, it's theft: they were aware they may not do it and according to the article attempted to get around further restrictions. I mean it's literally even theft of the $3 you mention — you don't have to be in possession the $3 to illegally deprive its owner of it under common theft statutes)
The internet must understand one simple rule. What’s downloadable will always be downloaded. There are plenty of people who don’t give a flying fk about your LICENSE.md, robots.txt, ToS and whatnot. It’s like leaving your bicycle unlocked in public. Even if there is a giant cardboard next to it saying: “please, don’t steal” - nobody cares.
> The crawler did not respect our robots.txt, (which it did retrieve), ignored our Terms of Service, (which forbid scraping), and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only.
It's a community site, first model I looked at (Tanya by Shrimport) is under CC-BY-ND-NC licence (Creative Commons, By attribution, No-derivatives, Non-commercial).
AI training by definition is derivative, and Roblox not using it for commercial purposes if used in production is practically nil.
This is a huge part of why AI upsets people, we can't have nice things such as communities where people are sharing 3d models without it being feeding ground for commercial companies that has little intention to ever honor any licences or compensate authors.
I know it is an unpopular opinion, but if you do not want your content to be scraped, you have to put it behind payment walls and copyright it. In Germany we have decisions from the highest courts that deemed websracping perfectly legal (it was in the context of a competitor scraping airline ticket fares).
And how were those license conditions violated by downloading? Their are permissive in the first place, and violating without passing on to a third party is almost impossible.
> Can I scrape all GPL source code from github. Then use the code commercially and then say so what, they made their source code available to download?
Yes, that's the whole point of the GPL. Use it however you want, but don't restrict the next person
I think technically such a model should abide by the licence of the inputs, hence all models would probably be GPL if the FSF had resources to fight it and the legal profession had an inkling of the processes involved.
Or maybe the FSF is just waiting for everyone using AI code to become hooked and then subpoena every Antrophic/OAI/Codex customer and force the most massive source release in history (If Adobe/MS would file lawsuits vs the Craft apps and with their resources create legal precedent it could create chaos, maybe the only part stopping them right now from filing lawsuits is them realizing it could backfire since they're heavy AI users themselves).
Of course not. But there’s no indication whatsoever that any license has been violated. Downloading a file from a free access server cannot violate license terms.
Fear that something will happen is not the same thing as evidence it has happened.
Who actually cares? Isn't the whole premise of the AI revolution just gorging ourselves with other people's data so we can have AGI? I was under the impression that we have accepted this sacrifice with OAI and friends leading the way.
Crypto, big data, AI. It's all just another middle man hoping to scrape some joy out of everyone's lives to they can put a price tag on it and sell it to rubes and the government(and rubes in the government).
I'm just to ignorant to this in context of how the reality just works.
Downloading content on the web is what the web is for. Using the data for training is something lawers need to figure out.
open3dlab should spend the time and energy, as unfortunate it is, to sue them to get this resolved.
Ablog post about this is basically 0 information. The stories about soooo many bots now crawling the internet and creating a lot more traffic is going through news and hn for month.
How can you start an outcry like this, but host stuff like this, tagged with a permissive license:
https://open3dlab.com/project/e2a0830d-f166-4b1b-aabf-8ca735...
Just an example, there's plenty more on the frontpage.
I have nothing against these guys, but I just can't understand the moral high ground of the complaint.
If you read the article, you'd find they admit to that, their position is nuanced, and this is not a "gotcha".
> We'll be honest about where we stand. We don't like copyright as it exists. We'd like to see it radically reformed, and we wouldn't mourn it if it were abolished. Our community lives in its grey areas every day: many of the models on our sites are fan ports of characters from commercial games, made out of love and shared for free. When a studio sends us a takedown notice, we comply.
> While we as small artists and fan communities get copyright used against us, the world's largest companies are committing copyright infringement at a scale unseen in human history. This has been the case since the invention of the internet of course; digital media make duplication, remixing and resharing trivial, but copyright law never caught up. Filesharing stabbed copyright in the gut. AI is just twisting the knife, and completes the creation an unequal system where copyright infringement is only a crime if you're a low net worth individual.
Rich companies are infringing copyright at industrial scale in order to build their LLMs, and even if caught and challenged, they don't suffer any catastrophic punishment. The infringed party gets enough money to shut them up and everyone moves on. A fine is a price. They'll keep doing it with impunity. Meanwhile the small guy who doesn't have copyright lawyers on retainer is getting shafted, time and time again.
[delayed]
Oddly, this makes their argument even stronger. Why would Roblox download this?
The game at the moment is capture as much content as possible.
Considering you link something that has a non comnercial license CC-by-NC, there might be others with permissive licenses. There might be things in their TOS of the site,
There's more with different licenses. If you scroll through the pages, there are a lot of assets ripped from games, with a variety of licenses CC-by-NC, or just "beerware":
https://open3dlab.com/project/0a4538ba-00df-4f6d-b3e3-14d978...
DMCA safe harbor? They're not responsible for user uploads until they are made aware of violations.
Scroll through the site, there are A LOT of assets from games, packaged by users. It's a nice subtrefuge, in practice it is a place to get ripped assets and the platform washes it's hands.
Again, I don't have any particularly strong issue with this, just with the pearl clutching regarding roblox downloading the data.
Party A: hosts largely copyrighted material, says they are opposed to copyright in its current form
Party B: massive IP business , very aggressive about everything from brand to software, defends and monetizes current copyright regime
I think it’s fair for party A to call out party B for downloading a massive trove of known-copyrighted material.
Then Roblox trains on it and voila…
well it kinda worked? over on reddit r/gaming thinks the data has been stolen
more radicalization towards AI
Moral high grounds don't mean much these days. They are weaponized as a way to get an advantage in an argument.
Scientists say this is the first occurance of the word "model" in the title of a post on HN that does not refer to a neural network of some sort since 2023
Only tangentially related, but I've recently realized that unless you're some sort of business with a valid requirement of that, you can just block incoming traffic from all of AWS, DigitalOcean, Azure, etc.
I think the spark came from some other HN comment I saw? So thank you for that, whoever wrote that.
That alone is not a sufficient strategy of course, but you can get rid of a lot of bots by just refusing to talk to where they tend to live.
Of course, if everyone were to do that, they'd eventually pivot to residential proxies. But I don't see that happening.
There have been a number of articles on the frontpage of HN about the residential proxy traffic. It is happening. This was one of the most popular:
https://news.ycombinator.com/item?id=49491791
I know, but it's an extra step, more expensive, and less legally solid.
None of these measures alone are a full solution, but if you stack a bunch of cheese slices, eventually, no holes line up anymore.
I initially thought this was a good idea too until another comment pointed out you'll lose most business or personal customers in offices because they connect through VPN gateways running on those cloud services (especially Azure)
True but the question is if that matters.
I would argue that more often than not it might not. Really depends on the specifics though.
A lot of stuff on the web actually isn't really relevant for all audiences everywhere I'd say. It just happened to be accessible by all, because academics back in the day dreamt of a better open world that didn't really manifest :(
> day dreamt of a better open world that didn't really manifest :(
it is open and better indeed. open and better for capital - the true, living, flesh-eating god.
This is a website about starting businesses.
Let me tell you, at the business I work at, some of our customers work at a company that uses a commercial VDI service for all workstations couldn’t access our service because the VDI egress was in GCP and we had an AWS WAF rule banning datacenter IPs.
Obviously, we removed the rule because it was doing nothing useful but plenty of harm. Real bots come from residential IPs.
All my business traffic comes from either our own AS, or from ZScaler's AS
Do many companies really use azure etc rather than dedicated services like zscaler?
Allmost all the bottraffic I see is from random residential proxies though.
Same, I just checked and this week the vast majority of the bot traffic I get is random residential IPs that all pretend to be the exact same Chrome 155 on Linux.
Probably all people that fell for the “superbox” TV streamer scam.
Won't this harm SEO? What about agentic software suites where authorized users should be able to drop an app passkey into grokbot or such?
true! but how does one find all those ips?
That’s pretty easy - all major providers have public lists of IP ranges, usually as json files.
Just ask RIPE
>But I don't see that happening.
Then you are not looking. Already have to switch to resi proxies from time to time when encountering yet another smartass who decided to play security theater and block datacenter ips on his shitpile he calls website, because "OUR USERS MUST BE GOOD BOYS WHO LOVE TO LIVE UNDER SURVILANCE AND ALLOW ISP TO TRACK THEM 24/7" (and before you say something like "others can track you too!", your ISP has the ability to knock on your door, they should be given as little information as possible, let them do metadata analysis on per-person basis if they really want it, but no 1-click mass surveillance shit).
If you are valuable, you will be be scraped. Turning user hostile in a hopeless endeavor to "protect yourself" is not the solution.
I can see that you're having strong emotions about this, but that rant is just hard to read and serves primarily as venting and not really to convince anyone of anything.
Unless one zooms out and thinks "okay, these guys seem to react negatively if we do exactly that. We should do more of it", which is probably not what you want.
Nobody cares what you do, you are not smart enough. If you even entertain the idea of "let's just block datacenter ips!", all you accomplish by doing that is showcasing your incompetence and lose all respect from legitimate users who are good at tech. The scraper will just turn the resi proxy and will continue as is.
But maybe selecting for tech illiterate cattle as your userbase is what you want, then I guess more power to you. Don't be surprised by the scorn though.
It's hostile to you personally, but I wouldn't call it "user hostile".
Blocking "scrapping [sic]" is also not security theater, you misunderstand the intent of GP.
Anyone who thinks blocking datacenter ips helps with blocking scraping is a clown.
I trust my ISP far more than I trust bigtech. The downside of connecting from my ISP is that the site I connect to can fingerprint me trivially from the IP range
I'm sympatic to open3dlab but I think the focus on adult-content is just trying to spark outcry for cheap.
It has been shown times and times against that removing adult content from the training source (at least for diffusion models), results in generally lower-quality output, even outside of adult-content, particularly in terms of representing human bodies.
They should change their name to openish3dlab.
would the implied intent (from the inclusion of adult content) of training models on open3dlab content, without asking them, render Roblox's actions _less_ antisocial???
The current AI Ethos is morality is just another obstacle on the way to their IPO. ...Lets just hope after they get rich, they stay away, and dont bother us with books, videos and conference speeches on entrepreneurship mentoring...
Couldn't the adult file download just as well be used to train a model that rejects adult uploads to roblox? I don't see the outrage.
if you put stuff available to download, dont be surprised that someone will download it
What is the actual news here, can anyone explain ????
They didn't do it in the intended manner.
> The crawler did not respect our robots.txt, (which it did retrieve), ignored our Terms of Service, (which forbid scraping), and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only.
https://therecord.media/linkedin-wins-court-order-blocking-m...
Which is it then? Megacorps can enforce their TOS through the law but small companies cannot?
The news is that people download it so much that they saturate your server’s connection and/or peg its CPU or disk (whichever is the weakest link in the chain). We used to call this a Denial of Service attack but now it’s apparently just a cost of doing business, even for people who just want to run a non-commercial hobby site.
Man if only there was such a thing as good manners, good behaviour, accepted practices from a community to not massively hammer servers. If only these websites had been up for a decade with very few problems, before a notoriously manipulative, lying, deceiving, abusive company that makes money off of child abusers came in and just went hurr durr free data in the WORST possible way instead of just politely asking for a dump.
honestly for me is the hypocrisy
we've spent the entire 2010s with lots of talk whether creating a scraper for google or linkedin and packaging that data to sell is illegal just for the entire thing to turn around where large companies are basically burning small servers down with requests to do the exact same thing
"Leopard ate MY face" ?
maybe stupid question but how is it possible to not be protected against supercrawlers like this, when you own large amounts of data, specially if all requests are coming from the same server ? Aren't cyberprotection services like CloudFlare work against this kind of "attacks" ?
Yeah, I can imagine it being a problem if it was from different IP addresses, but surely putting a limit on requests from a single IP would be a straightforward fix? Although maybe until it happens, you don't think about it.
3$ in outbound traffic on a public service, how is this news?
I mean, it's not the size of the download, it's the unauthorised scraping of community assets strictly against terms and ignoring robots.txt by an American corporation. Resource of a kind that used to get kids threatened with spending most of their life in prison.
But I agree that this kind of theft by US corporations is no longer news.
(Yes, it's theft: they were aware they may not do it and according to the article attempted to get around further restrictions. I mean it's literally even theft of the $3 you mention — you don't have to be in possession the $3 to illegally deprive its owner of it under common theft statutes)
Aaron Swartz pulled down by the legal system for trying to get things accessible for people instead of being blocked by profit requirements.
Companies copying things to make profit for themselves not being held accountable.
Progress is one thing, injustice is another.
The internet must understand one simple rule. What’s downloadable will always be downloaded. There are plenty of people who don’t give a flying fk about your LICENSE.md, robots.txt, ToS and whatnot. It’s like leaving your bicycle unlocked in public. Even if there is a giant cardboard next to it saying: “please, don’t steal” - nobody cares.
If that data is at all important, why is is downloadable?
This is quite simple put in the article.
> The crawler did not respect our robots.txt, (which it did retrieve), ignored our Terms of Service, (which forbid scraping), and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only.
A robots.txt file is not anything, not enforceable, it's politeness. Of course it was ignored.
That fight was already lost back in 2024.
Sue them. They'd sue you for ignoring their ToS.
It's a community site, first model I looked at (Tanya by Shrimport) is under CC-BY-ND-NC licence (Creative Commons, By attribution, No-derivatives, Non-commercial).
AI training by definition is derivative, and Roblox not using it for commercial purposes if used in production is practically nil.
This is a huge part of why AI upsets people, we can't have nice things such as communities where people are sharing 3d models without it being feeding ground for commercial companies that has little intention to ever honor any licences or compensate authors.
If courts say that training models is fair use then copyright and license terms are moot.
It’s not fully played out in the courts yet, but so far doesn’t look like it’s going to be enforced how you want it to be.
I wonder at what point the Roblox kids all reach 18 years old, and Roblox can pivot them all to a separate adult version thereof...
Never, it gets new players constantly. This is like saying I wonder when YouTube is going to turn into Pornhub
I assume they need the adult content in order to detect adult content generated by their users.
I know it is an unpopular opinion, but if you do not want your content to be scraped, you have to put it behind payment walls and copyright it. In Germany we have decisions from the highest courts that deemed websracping perfectly legal (it was in the context of a competitor scraping airline ticket fares).
They ignored the licenses, so kinda moot point you're making.
And how were those license conditions violated by downloading? Their are permissive in the first place, and violating without passing on to a third party is almost impossible.
Were they used in a commercial context when a commercial entity downloaded it?
If yes, they were in violation of the license. Fyi, that's explicitly stated in the article we're commenting on.
If it’s fair use, the license doesn’t matter.
145GB isn’t much. It fits in RAM in some computers.
145gb of ram? Check out Mr. Moneybags over here!
So what? You made your content available and someone downloaded it. Why is this even notable?
Can I scrape all GPL source code from github. Then use the code commercially and then say so what, they made their source code available to download?
Oh, it would be so nice to force the Great Thieves like "open"AI to release their weights on GPL license...
> Can I scrape all GPL source code from github. Then use the code commercially and then say so what, they made their source code available to download?
Yes, that's the whole point of the GPL. Use it however you want, but don't restrict the next person
Right now, its legal to use this data to train a model.
Which is different than using the data 1 to 1.
I think technically such a model should abide by the licence of the inputs, hence all models would probably be GPL if the FSF had resources to fight it and the legal profession had an inkling of the processes involved.
Or maybe the FSF is just waiting for everyone using AI code to become hooked and then subpoena every Antrophic/OAI/Codex customer and force the most massive source release in history (If Adobe/MS would file lawsuits vs the Craft apps and with their resources create legal precedent it could create chaos, maybe the only part stopping them right now from filing lawsuits is them realizing it could backfire since they're heavy AI users themselves).
Would you also say that general licensing is irrelevant for everyone?
Of course not. But there’s no indication whatsoever that any license has been violated. Downloading a file from a free access server cannot violate license terms.
Fear that something will happen is not the same thing as evidence it has happened.
Can't believe we have corporate kiss ass in here
> Downloading a file from a free access server cannot violate license terms.
Sure it can. If you're downloading it for commercial purposes (clearly the case) you're violating the license.
If only you could read the article and see that the vast majority of the assets are licensed under a non-commercial clause.
I guess we'll never know until you learn to read anything other than sheet music.
I'm not even impressed..
Who actually cares? Isn't the whole premise of the AI revolution just gorging ourselves with other people's data so we can have AGI? I was under the impression that we have accepted this sacrifice with OAI and friends leading the way.
People really still buy into that framing? I think it's pretty clear that it's generally just about money as it has always been.
Tech is full of rubes that eat up the lies.
Crypto, big data, AI. It's all just another middle man hoping to scrape some joy out of everyone's lives to they can put a price tag on it and sell it to rubes and the government(and rubes in the government).
If the authors of the data are paid...ok. and not just the biggest players - every single one. But they are not.
Who's "we"?
"we" in this sentence is really a handful of encloistered silicon valley weirdos. Most people do not like it and have no real say in the matter.
I'm just to ignorant to this in context of how the reality just works.
Downloading content on the web is what the web is for. Using the data for training is something lawers need to figure out.
open3dlab should spend the time and energy, as unfortunate it is, to sue them to get this resolved.
Ablog post about this is basically 0 information. The stories about soooo many bots now crawling the internet and creating a lot more traffic is going through news and hn for month.