Ok, it sucks, losing data always sucks, but what ever happened to the 3-2-1 backup rule? even if your off site backup provider goes tits up, your on site backup is still there.
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
> but what ever happened to the 3-2-1 backup rule?
I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
> the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
They signed up with them in 2019. Not long ago.
Even if it is a "new" concept, common sense isn't. Pretty much anyone who cares about their data will do the due diligence, and this is not something that requires technical chops. This has been a concern for well over 20 years!
It really sounds like some senior management trying to be cheap.
This has been a rule for like 30+ years (I have only been in the industry that long). Whether it was called "3-2-1" before that or not, the practice is the same. Have an onsite backup, have an offsite backup. This isn't a new concept at all.
The 3-2-1 rule is ~20years old at this point. But just because an archive is "old" doesn't mean we can't apply modern standards to that archive as time marches on, otherwise the BM_ARC would have never been build and opened a couple of years ago.
I mean, that 70+ year old archive must have been digitised at some point, unless Iron Mountain was storing the reel to reel tapes and/or film from the original recordings. So some "modernation" has been applied to it since its creation.
It's worth noting that PBS affiliates are non-profits that have fairly constrained budgets, especially now that the Corporation for Public Broadcasting has been dissolved due to DOGE.
Non-profit IT has always seemed, to me at least, to be a land of compromises. There's rarely enough money, knowledge, or time to do things "the right way", so you have to do the best you can. If I could only choose one backup to have, it'd be remote on a provider like Iron Mountain. It would seem as if they are storing things on Iron Mountain through an intermediary company, who may or may not have actually paid the bill.
I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
And they sent OSS a legal demand letter to keep paying the Iron Mountain bill for their data. If OSS actually was using IM and not just running a scam, then the data should be pretty safe, if just not accessible for a few weeks/months.
Don't trust Internet Archive as your only backup method though. There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
> Don't trust Internet Archive as your only backup method though.
Agreed, no doubt about it.
> There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
What sort of content? All the content I've judged to be of public interest and uploaded to Internet Archive, is still online there today. But I'd understand that depending on the type of content, that might not be true for everythibng.
Inability to access items does not mean deleted. In the case of PBS content, it is unlikely access would be lost. I cannot speak to your use case, but the Patron Help Desk can confirm if an item is relevant to their preservation efforts if you'd like to check before uploading. Email info at archive dot org to inquire.
Someone dumping their Twitch stream or random Youtube content into the storage system will certainly have that content deleted and be banned if prior arrangements have not been made, for example. Don’t do that.
It's incredibly frustrating. The Internet Archive offers a Vault product, where you pay $2-3/GB one time for perpetual storage into the future of your collections, items, media, etc. This pricing was previously predicated on the cost decline curve of storage. It's gotten a bit more expensive due to the AI bubble inflating the cost of storage, but to your point, the Internet Archive is a superior custodian than some rando tier 2 cloud provider who won a contract from a non profit and will lose all of the data when they go bust.
> This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
I read it as a statement of just how little data this is by modern standards more than anything else. This company is going out of business and PBS is losing this valuable data, and there's legal drama and multiple corporate entities involved, and if you read all that you might think this is some amazing amount of data that is actually difficult to deal with. Instead it's something I could stroll down to Best Buy right now, plonk down my personal credit card, and walk out with enough stuff to do a credible backup job myself, and we're not even challenging my credit limit or anything.
10 years ago, this was a lot of data. It isn't anymore and it makes it that much more bizarre that they're having that much trouble getting it back. It would be less to just give them access and let them download it themselves and perhaps stick it somewhere else temporarily than to pay lawyers to deal with legal filings.
To put it in vaguely physical terms, what once was a university library is now a good solid bookshelf full of stuff. Legally wrangling over a huge library is one thing, but if it's just a bookshelf, metaphorically, let them come in for an hour and clean it out themselves rather than bringing in the law. Obviously not a perfect metaphor on data size, this is more about how lawsuits are more of a hassle then just giving it to them.
The reason nobody's just giving it to them isn't because the amount of data's too much of a hassle, it's because nobody can figure out who has the keys to the now-defunct storage company, and Iron Mountain quite reasonably won't hand out their customer's data to a third party without a court order.
More like 20 years ago (sorry). 10 years ago this would have been slightly stretching, but not by much - 11x8TB raidz3. Shuckable drives would have cost $3300 (plus the chassis), and already out of the husk probably 1.5x more. So let's say $5500 total even back then. (I built my current array around the same time)
Funny, these days you could do it with 4 spindles (4x24TB raidz2) but with the rampant price inflation in storage, you're still looking at $4500.
I recently pulled up my resume from 25 years ago and came across this amusing (by today's standards) tidbit: "Re-located a web-server farm between two data centers with zero downtime. This included migrating over 150GB of data between three Network Appliance Filers and moving more than 12 racks of equipment."
Having that be there only backup solution would have been silly and dangerous. Having it be part of their overall backup solution including with something like the provider would have been smarter and better and prevented this entire problem
> It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on [Amazon] and populate it with [the best HDDs].
I think they just put that stuff in so they could shill their affiliate marketing articles rather than out of any actual concern.
Yeah I bought a five bay Synology NAS two or three years ago I think it and three 18TB WD Red drives total cost $3,000CAD? I was going to save up for two more drives. Now each drive is $1,000 CAD a 150% jump. edit: I checked $1,350 each so about 250% jump ugh!
Technically you're still correct even when you would advocate for both. Every additional copy increases the risk that the data is accessed by someone else.
A NAS is at risk from malware that can erase all of it. 100GB Bluray discs exist and work more easily for archival storage, as does multi-terabyte tape drive storage, both of which can stay disconnected At least two offline copies, combined with error correction codes, are advised. A cloud backup is also advised.
Calling out Nine PBS for not accounting for the fact that their vendor could effectively just decide to commit crimes to get out of their contract and their vendor’s vendor could just decide to compound the problem in the face of a court order is wild. Yes, we’ve all learned to anticipate everyone else on the planet being an absolute shitheel; no, Nine PBS did not do anything wrong by assuming the professionals they were paying to render a professional service to them would do so. We’re all paranoid engineers, but this is absolutely no goddamn way to run a society.
Not PBS but an affiliate broadcaster and the data is not lost as in destroyed, but legally encumbered.
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
This local PBS wasn't playing it's own footage 24/7. The data in question could be just their local segments, an hour or two of local coverage and maybe their fundraisers.
>This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Look, I know that a certain percentage of Tom's Hardware articles are just sales pitches for a NAS, but this is asinine. Iron Mountain deals largely in tape storage. 50TB of tapes fits in one of their small archive boxes. If they're using the latest tech (LTO-9), it fits on 3 tapes.
This is exactly how you are supposed to store archival media for decades, not on a NAS. A NAS is for storing the files you need to actively access.
Ok, it sucks, losing data always sucks, but what ever happened to the 3-2-1 backup rule? even if your off site backup provider goes tits up, your on site backup is still there.
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
> but what ever happened to the 3-2-1 backup rule?
I wasn't 100% sure, but the data backs this up[0]; the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
- [0] Interestingly, Ngram (https://books.google.com/ngrams/graph?content=%5B3-2-1+backu...) says ~2010 while Trends (https://trends.google.com/explore?q=3-2-1%2520backup%2520rul...) says slow build up from ~2020, guess "everyday" language would sit somewhere around those two different truths.
> the "3-2-1 backup rule" is a relatively modern concept, at least by that name. I also remember things like "have one backup in one place, another in another place" since before this mnemonic, but I feel like it's unfair to judge a 70+ years archive on something that just recently actually became a widespread mantra.
They signed up with them in 2019. Not long ago.
Even if it is a "new" concept, common sense isn't. Pretty much anyone who cares about their data will do the due diligence, and this is not something that requires technical chops. This has been a concern for well over 20 years!
It really sounds like some senior management trying to be cheap.
This has been a rule for like 30+ years (I have only been in the industry that long). Whether it was called "3-2-1" before that or not, the practice is the same. Have an onsite backup, have an offsite backup. This isn't a new concept at all.
The 3-2-1 rule is ~20years old at this point. But just because an archive is "old" doesn't mean we can't apply modern standards to that archive as time marches on, otherwise the BM_ARC would have never been build and opened a couple of years ago.
I mean, that 70+ year old archive must have been digitised at some point, unless Iron Mountain was storing the reel to reel tapes and/or film from the original recordings. So some "modernation" has been applied to it since its creation.
It's worth noting that PBS affiliates are non-profits that have fairly constrained budgets, especially now that the Corporation for Public Broadcasting has been dissolved due to DOGE.
Non-profit IT has always seemed, to me at least, to be a land of compromises. There's rarely enough money, knowledge, or time to do things "the right way", so you have to do the best you can. If I could only choose one backup to have, it'd be remote on a provider like Iron Mountain. It would seem as if they are storing things on Iron Mountain through an intermediary company, who may or may not have actually paid the bill.
So, y'know, fraud's a possibility here.
Free account, indefinitely, to whoever needs it to store this data … including the storage vendor.
Just email…
Epic.
This might be the website: https://www.osstorage.com/
I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
https://www.linkedin.com/company/osstorage/people/
It's not PBS, PBS - but an affiliate, Nine PBS (https://www.ninepbs.org/)
From the article it seems like Iron Mountain was doing the actual storage, and OSS was just acting as a middleman and taking some money off the top.
And they sent OSS a legal demand letter to keep paying the Iron Mountain bill for their data. If OSS actually was using IM and not just running a scam, then the data should be pretty safe, if just not accessible for a few weeks/months.
I guess that depends on how long Iron Mountain has to keep their data once OSS stops paying - and we don't know how long ago OSS's money ran out.
This underscores the importance of the Internet Archive and people like Marion Stokes (https://en.wikipedia.org/wiki/Marion_Stokes).
Don't trust Internet Archive as your only backup method though. There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
> Don't trust Internet Archive as your only backup method though.
Agreed, no doubt about it.
> There has been instances where they delete files without warning with the only explanation being that they don't meet TOS
What sort of content? All the content I've judged to be of public interest and uploaded to Internet Archive, is still online there today. But I'd understand that depending on the type of content, that might not be true for everythibng.
Inability to access items does not mean deleted. In the case of PBS content, it is unlikely access would be lost. I cannot speak to your use case, but the Patron Help Desk can confirm if an item is relevant to their preservation efforts if you'd like to check before uploading. Email info at archive dot org to inquire.
Someone dumping their Twitch stream or random Youtube content into the storage system will certainly have that content deleted and be banned if prior arrangements have not been made, for example. Don’t do that.
https://old.reddit.com/r/DataHoarder/comments/sq6wbq/please_...
It's incredibly frustrating. The Internet Archive offers a Vault product, where you pay $2-3/GB one time for perpetual storage into the future of your collections, items, media, etc. This pricing was previously predicated on the cost decline curve of storage. It's gotten a bit more expensive due to the AI bubble inflating the cost of storage, but to your point, the Internet Archive is a superior custodian than some rando tier 2 cloud provider who won a contract from a non profit and will lose all of the data when they go bust.
https://archive-it.org/vault/
(huge fan of Stokes as a fellow archivist, recommend the documentary on her: https://en.wikipedia.org/wiki/Recorder:_The_Marion_Stokes_Pr...)
> This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
I read it as a statement of just how little data this is by modern standards more than anything else. This company is going out of business and PBS is losing this valuable data, and there's legal drama and multiple corporate entities involved, and if you read all that you might think this is some amazing amount of data that is actually difficult to deal with. Instead it's something I could stroll down to Best Buy right now, plonk down my personal credit card, and walk out with enough stuff to do a credible backup job myself, and we're not even challenging my credit limit or anything.
10 years ago, this was a lot of data. It isn't anymore and it makes it that much more bizarre that they're having that much trouble getting it back. It would be less to just give them access and let them download it themselves and perhaps stick it somewhere else temporarily than to pay lawyers to deal with legal filings.
To put it in vaguely physical terms, what once was a university library is now a good solid bookshelf full of stuff. Legally wrangling over a huge library is one thing, but if it's just a bookshelf, metaphorically, let them come in for an hour and clean it out themselves rather than bringing in the law. Obviously not a perfect metaphor on data size, this is more about how lawsuits are more of a hassle then just giving it to them.
The reason nobody's just giving it to them isn't because the amount of data's too much of a hassle, it's because nobody can figure out who has the keys to the now-defunct storage company, and Iron Mountain quite reasonably won't hand out their customer's data to a third party without a court order.
Yes, I agree completely. It just heightens the pathos of the story that it is over what is now such a small amount of data.
> 10 years ago, this was a lot of data.
More like 20 years ago (sorry). 10 years ago this would have been slightly stretching, but not by much - 11x8TB raidz3. Shuckable drives would have cost $3300 (plus the chassis), and already out of the husk probably 1.5x more. So let's say $5500 total even back then. (I built my current array around the same time)
Funny, these days you could do it with 4 spindles (4x24TB raidz2) but with the rampant price inflation in storage, you're still looking at $4500.
I recently pulled up my resume from 25 years ago and came across this amusing (by today's standards) tidbit: "Re-located a web-server farm between two data centers with zero downtime. This included migrating over 150GB of data between three Network Appliance Filers and moving more than 12 racks of equipment."
A whopping 150GB!
Having that be there only backup solution would have been silly and dangerous. Having it be part of their overall backup solution including with something like the provider would have been smarter and better and prevented this entire problem
> It's not clear if the channel has backups of its data, but based on its determination, we suspect not. This is strange, though, especially for a major organization, as one can easily purchase a six-bay NAS on [Amazon] and populate it with [the best HDDs].
I think they just put that stuff in so they could shill their affiliate marketing articles rather than out of any actual concern.
3 copies, 2 different media, 1 offsite. One backup is none
Importantly, "one cloud vendor" is also none
1 copy, one other media, is better than 0 copies. Clearly 1 != 0.
The point still stands, but in what world can you build a 50TB NAS for $1,000 these days? I want to go there
Yeah I bought a five bay Synology NAS two or three years ago I think it and three 18TB WD Red drives total cost $3,000CAD? I was going to save up for two more drives. Now each drive is $1,000 CAD a 150% jump. edit: I checked $1,350 each so about 250% jump ugh!
$1000 is for the NAS, they still need to populate it with a couple of drives.
Technically you're still correct even when you would advocate for both. Every additional copy increases the risk that the data is accessed by someone else.
What risk? The point of PBS is “other people access the data.”
A NAS is at risk from malware that can erase all of it. 100GB Bluray discs exist and work more easily for archival storage, as does multi-terabyte tape drive storage, both of which can stay disconnected At least two offline copies, combined with error correction codes, are advised. A cloud backup is also advised.
Calling out Nine PBS for not accounting for the fact that their vendor could effectively just decide to commit crimes to get out of their contract and their vendor’s vendor could just decide to compound the problem in the face of a court order is wild. Yes, we’ve all learned to anticipate everyone else on the planet being an absolute shitheel; no, Nine PBS did not do anything wrong by assuming the professionals they were paying to render a professional service to them would do so. We’re all paranoid engineers, but this is absolutely no goddamn way to run a society.
> since 2019
At that point, why wouldn't you pick a major cloud vendor? This feels like more of a governance failure.
Not PBS but an affiliate broadcaster and the data is not lost as in destroyed, but legally encumbered.
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
Isn't 50TB just 6 years of videos, in even the best case (1GB/h)?
This local PBS wasn't playing it's own footage 24/7. The data in question could be just their local segments, an hour or two of local coverage and maybe their fundraisers.
From the article:
>This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Look, I know that a certain percentage of Tom's Hardware articles are just sales pitches for a NAS, but this is asinine. Iron Mountain deals largely in tape storage. 50TB of tapes fits in one of their small archive boxes. If they're using the latest tech (LTO-9), it fits on 3 tapes.
This is exactly how you are supposed to store archival media for decades, not on a NAS. A NAS is for storing the files you need to actively access.
Title should be updated to reflect the fact that this is a single PBS station.
That made me feel relief until I realized it's my own local PBS station.
Any interest in connecting me with someone there so I can attempt to obtain a copy for the Internet Archive?
Ah, "Channel Nine PBS", not a cluster of nine pbses