Am I correct that this has come about because archive.org respects robots.txt and these sites have blocked their crawler from indexing their sites?
I'm not sure how to articulate my thoughts on this exactly, other than to say it's disappointing that doing the right thing (i.e. respecting robots.txt) is rewarded with the burden of soliciting responses to a petition while at the same time others are rewarded with profit for ignoring those same directives.
Don't know if it helps your musings at all, but there's a good chance that if a high-profile crawler like archive.org disrespected their robots.txt, that archive.org would be faced with lawsuits (or some other form of pressure). This is not merely the most moral move; rather it is the only sensible move.
The only reason "others are rewarded with profit" in cases like these are because pinkie-promise-style obligations don't affect players too small or shadowy to bother litigating.
LLMs have other ways of accessing the content, they don’t need the Web Archive.
Every LLM company can afford to spin up a new subscriber account every day, proxying to appear different IPs from all sorts of ASNs, do some crawling until the account gets banned, and then do it again, and again, and again.
A bunch of people who have haven't ever loaded an ad or paid a subscription to those organizations are going to make a stand to demand they leave their backdoor open?
I know a little about this debate on the Times and Atlantic sides. I’ll get some grief for this, but I asked a senior person at the former what they thought about the paywall workarounds that are frequent on HN—I was genuinely shocked to learn they hadn’t heard about it.
In the end, we settled on agreeing that making such stuff available after 30 days, and possibly with access restrictions (can’t be pulled more than N times a day, in case it becomes relevant in the future) struck the right balance.
To my knowledge, the Internet Archive hasn’t done any outreach on this issue. In addition to pressuring the publications, I’d put some pressure on them to negotiate.
> can’t be pulled more than N times a day, in case it becomes relevant in the future
In case it "becomes relevant." Wouldn't that benefit you either way? It makes you wonder if they have a dashboard of unfortunate digital statistics on display somewhere and worship of these numbers have replaced the underlying spirit of journalism.
The title freaked me out. I thought this was about the Wayback Machine going away but no, it's just news publications blocking being archived.
I guess I don't really care. As soon as it becomes unworkable to view these publications through archivers I'll just stop viewing them altogether. I don't see this helping their bottom line though.
As long as other people are reading them, they're important for understanding what's happening in the world and what information the public is getting, which is why we need an accessible archive of their content.
Exactly. Libraries have kept microfiche archives of newspapers for forever, and they're an essential part of historical research.
They also preserved old books. But now I guess they're becoming middlemen for access to limited ebook platforms that ensure books disappear when publishers lose interest.
The "Information Age" is proving to be the setup for a dark age, when nonprofitable things are just thrown out and efforts to preserve them are actively fought.
I think part of this is important too because online news articles might have corrections, or certain paragraphs might get deleted in some rare situations. It's good to have a way of tracking those. Sometimes, the edits made to an article are very irrelevant to the actual message. I'm thinking stuff like typos, or even embarrassing gaffes like the recent time that a headline implied that the NATO acronym had the word "American" in it.
Can we just go back to ads and normalize blocking people who ad-block?
I'm grown up now, I understand how things work, and I'd rather see Tide and Coke ads than pay $20/mo to 8 different orgs, while maintaining that ad free option for those who want it.
The children of the internet probably won't sign a truce, so let's just cut them out and let intellectually honest people have a decent internet.
I'm a paying NYT subscriber for years. NYT has a ton of ads, even for subscribers. They don't offer an ad free version despite it being totally viable at a few more bucks a month based on their finances. Their ads are super disruptive to reading and their privacy policy appears to indicate they buy and sell your data.
I dunno. That seems like a pretty big fuck you to a paying customer already when all they have to do is provide a sub for a few more bucks a month. But I guess I'm a child of the Internet.
How about we go back to the era of humanity where modern marketing didn't exist?
How much faster would consumer software be if adware was made illegal? How much faster would our devices be if we didn't have half the code base supporting malware?
Acting like an ad enabled internet was the only option is extremely foolish, especially when the ad enabled internet was fully chosen and pushed onto the public by very specific people (thanks Newt Gingrich!).
The petition should be to ban the AI theft. If it is on wayback, the bots could as well scrape the NYT directly.
The NYT is of course guilty itself. It did not investigate the possible murder of its star witness Suchir Balaji and is too reserved in examining the consequences of AI in general.
If they don't fulfill their journalistic and societal obligations, soon its own journalists will be replaced by AI bullet point slop like Axios.
After many years of these media outlets circling the drain, this is likely the clearest signal of their irrelevance. It's not like anyone is committing these rags to microfiche anymore.
Wouldn’t it be better to let these legacy news orgs (which aren’t really anything beyond advertising and data harvesting firms) block archive.org and thus no one will read their articles and they can go under? I’m struggling to think of a reason I need NY Times. I’ve never had a subscription and never seen writing that I thought benefited me as a citizen (they’re Very pro-war of any kind).
Media influence and authority has historically depended on getting cited by writing that is more directly relevant to the reader's concern (i.e. the topic of research).
The paywalls were one thing, but disallowing archival is practically suicide.
Plenty to criticize about the NYT, but I see many people complaining about "legacy media" are often the same people getting "news" from random Twitter/X accounts.
if people are reading the articles through wayback, then they aren't making any money because no data is harvested and no click-thrus or impressions or whatever the metric is are registered.
Am I correct that this has come about because archive.org respects robots.txt and these sites have blocked their crawler from indexing their sites?
I'm not sure how to articulate my thoughts on this exactly, other than to say it's disappointing that doing the right thing (i.e. respecting robots.txt) is rewarded with the burden of soliciting responses to a petition while at the same time others are rewarded with profit for ignoring those same directives.
Don't know if it helps your musings at all, but there's a good chance that if a high-profile crawler like archive.org disrespected their robots.txt, that archive.org would be faced with lawsuits (or some other form of pressure). This is not merely the most moral move; rather it is the only sensible move.
The only reason "others are rewarded with profit" in cases like these are because pinkie-promise-style obligations don't affect players too small or shadowy to bother litigating.
Correct. Example snippet from the nytimes.com robots.txt:
It's because they want to restrict AI companies from stealing content, but they can't do it if internet archive proxies it all for them.
All of the LLMs would be massively less useful if it wasn't for scraping the latest news.
LLMs have other ways of accessing the content, they don’t need the Web Archive.
Every LLM company can afford to spin up a new subscriber account every day, proxying to appear different IPs from all sorts of ASNs, do some crawling until the account gets banned, and then do it again, and again, and again.
Idea: allow scraping but can’t publish for 1 year?
I am looking forward to this (https://news.ycombinator.com/item?id=48070516)
Maybe they should have an escrow like Financial Times is available on NewsBank service with a 30 day escrow
A bunch of people who have haven't ever loaded an ad or paid a subscription to those organizations are going to make a stand to demand they leave their backdoor open?
I know a little about this debate on the Times and Atlantic sides. I’ll get some grief for this, but I asked a senior person at the former what they thought about the paywall workarounds that are frequent on HN—I was genuinely shocked to learn they hadn’t heard about it.
In the end, we settled on agreeing that making such stuff available after 30 days, and possibly with access restrictions (can’t be pulled more than N times a day, in case it becomes relevant in the future) struck the right balance.
To my knowledge, the Internet Archive hasn’t done any outreach on this issue. In addition to pressuring the publications, I’d put some pressure on them to negotiate.
> can’t be pulled more than N times a day, in case it becomes relevant in the future
In case it "becomes relevant." Wouldn't that benefit you either way? It makes you wonder if they have a dashboard of unfortunate digital statistics on display somewhere and worship of these numbers have replaced the underlying spirit of journalism.
The title freaked me out. I thought this was about the Wayback Machine going away but no, it's just news publications blocking being archived.
I guess I don't really care. As soon as it becomes unworkable to view these publications through archivers I'll just stop viewing them altogether. I don't see this helping their bottom line though.
As long as other people are reading them, they're important for understanding what's happening in the world and what information the public is getting, which is why we need an accessible archive of their content.
Exactly. Libraries have kept microfiche archives of newspapers for forever, and they're an essential part of historical research.
They also preserved old books. But now I guess they're becoming middlemen for access to limited ebook platforms that ensure books disappear when publishers lose interest.
The "Information Age" is proving to be the setup for a dark age, when nonprofitable things are just thrown out and efforts to preserve them are actively fought.
I think part of this is important too because online news articles might have corrections, or certain paragraphs might get deleted in some rare situations. It's good to have a way of tracking those. Sometimes, the edits made to an article are very irrelevant to the actual message. I'm thinking stuff like typos, or even embarrassing gaffes like the recent time that a headline implied that the NATO acronym had the word "American" in it.
Archives protect truth
Can we just go back to ads and normalize blocking people who ad-block?
I'm grown up now, I understand how things work, and I'd rather see Tide and Coke ads than pay $20/mo to 8 different orgs, while maintaining that ad free option for those who want it.
The children of the internet probably won't sign a truce, so let's just cut them out and let intellectually honest people have a decent internet.
I'm a paying NYT subscriber for years. NYT has a ton of ads, even for subscribers. They don't offer an ad free version despite it being totally viable at a few more bucks a month based on their finances. Their ads are super disruptive to reading and their privacy policy appears to indicate they buy and sell your data.
I dunno. That seems like a pretty big fuck you to a paying customer already when all they have to do is provide a sub for a few more bucks a month. But I guess I'm a child of the Internet.
How about we go back to the era of humanity where modern marketing didn't exist?
How much faster would consumer software be if adware was made illegal? How much faster would our devices be if we didn't have half the code base supporting malware?
Acting like an ad enabled internet was the only option is extremely foolish, especially when the ad enabled internet was fully chosen and pushed onto the public by very specific people (thanks Newt Gingrich!).
The petition should be to ban the AI theft. If it is on wayback, the bots could as well scrape the NYT directly.
The NYT is of course guilty itself. It did not investigate the possible murder of its star witness Suchir Balaji and is too reserved in examining the consequences of AI in general.
If they don't fulfill their journalistic and societal obligations, soon its own journalists will be replaced by AI bullet point slop like Axios.
After many years of these media outlets circling the drain, this is likely the clearest signal of their irrelevance. It's not like anyone is committing these rags to microfiche anymore.
> the clearest signal of their irrelevance
NYT had $2.82B in revenue in 2025.
And by what standards have you determined that these outlets are circling the drain?
The work of independent journalists is more important than ever before.
More important than ever before and less market value than ever before. :(
It’s kind of shocking to read what you wrote, and realize those big media brands used to be independent journalism.
Are we considering the NYT and USA Today "independent journalism" still? Seems dubious...
> It's not like anyone is committing these rags to microfiche anymore.
I recommend you actually go and read those fiches. The press was not historically high quality. Mass media has had the same problems for decades.
What it used to have was genuine independent competition.
Wouldn’t it be better to let these legacy news orgs (which aren’t really anything beyond advertising and data harvesting firms) block archive.org and thus no one will read their articles and they can go under? I’m struggling to think of a reason I need NY Times. I’ve never had a subscription and never seen writing that I thought benefited me as a citizen (they’re Very pro-war of any kind).
> block archive.org and thus no one will read their articles and they can go under?
…why would they go under if the people who don’t pay for news stop reading them?
Media influence and authority has historically depended on getting cited by writing that is more directly relevant to the reader's concern (i.e. the topic of research).
The paywalls were one thing, but disallowing archival is practically suicide.
Plenty to criticize about the NYT, but I see many people complaining about "legacy media" are often the same people getting "news" from random Twitter/X accounts.
if people are reading the articles through wayback, then they aren't making any money because no data is harvested and no click-thrus or impressions or whatever the metric is are registered.