file storage and preservation for the future

[sevenELIXIR]

twoists- I know many of you can give me some good insight into this. I'm probably among some of the "older" members of the board, so maybe some younger, more tech savvy (or older too) of you can enlighten me.

I have been archiving my personal computer documents since around 2002. recently I have read an article discussing preserving data and how much of it will be obsolete in the future. I have thousands of photos and thousands of hours of music that I am trying to preserve. my greatest fear would be that somehow the hard drives become physically damaged, or just wear out and become damaged files that are no longer operable.

now a days I hear about "the cloud." to this, I say fuck that. I don't like the idea of having my information, art and other media personal to me on a nebulous cyber server. I have 2 western digital external hard drives that look like iphones. I have a Toshiba satellite laptop from 2011 or so. I'm sure these are already basically obsolete.

but are there really other ways to preserve my saved digital photos/ music and other media? for instance, how will I be able to open a jpeg or a text format in 30 years (provided my hardware still holds this information)? how long are external hard drives good for? are CDs a better way to store? what if there are no disc readers 20 years from now? please share your ideas and thoughts. I have so much precious material accumulated that it would be a shame to lose. so far, not many of my documents have become randomly corrupt. but, like a digital librarian, I am looking for alternatives to this. thanks if you've read this far

[Cupz]

Its always a gamble, I think digital stuff isnt really safe anywhere unless you're constantly migrating to the latest technology. Its all about minimizing risk. I've been thinking of burning all my shit to dvd's, and have multiple copies for each so if one stops working there are 10 others to take its place. Still not fool-proof. Of course it can happen that dvd drives become non-existent though probably not very soon (and you would notice well in advance and copy it to newer media), or if all the disks would magically explode simultaniously.... Maybe keep a little disconnected 20$ laptop with dvd-drive and optional internet-connection next to the dvd-storagecase in a wallsafe like a time-capsule. That might actually work for the next couple of decades and is a pretty cheap solution. Maybe throw in an extra pair of dvd-drives just to be sure.

I have had really bad experiences with anything running through usb and I cannot recommend keeping all of your stuff single-copy on western digital hdd's as I know someone who lost 1tb of home-video because of a relatively tiny error and no other back-ups and he was very devistated. It makes it look like usb hdd's are great for portability and not for archiving.

When I asked twoism if they had old music I was pleasently suprised and relieved that so much music appeared from everywhere around the world. I can sleep well knowing multiple friends have most of my stuff, that might be the true power of the cloud. I hope in the future I can download my own music through soulseek or whatever p2p software the cool kids will be using in the future but untill then I'll burn the so-many-gigs of music I have created over the years to disks very soon. Maybe you can give copies of your data to friends you really(!!!) trust for safe-keeping on their computers.

I agree that the cloud is a scary place, and you might not want to put personal files on a dark server in silicon valley, but there are allot of non-corporate servers and you might even know someone that runs their own server in their attic and could spare you some space on it exchange for, idk, beer, dick or money I guess. I'm not sure how much data you are talking about, but a couple of gigs shouldn't be to hard to fix, but he might get bored with his hobby server. You could also just encrypt everything well enough and they'll be safe on any server, even the dark scary ones, and you could spread it over the internet like a virus and nobody would be able to see what it means. It would just be random data untill someone knows the magic character string (which you have made impossible to guess of course).

What I can recommend is making a difference between essentials and non-essentials. Things you NEED even in times of great struggle and things you don't. I know those 20gb of raw-format photos of that vacation in france seem important...



My absolute essentials fit on a 4gb thumbdrive. Mostly plain text-files, essential/hard to find/homemade software and really really special pictures (and not in absurd-res). Its highly portable and backed-up in no time. Things I gathered from the internet I can probably gather again.

As for file-formats becoming obsolete, I wouldn't worry. The open-source community got you covered.

[Techboy]

I was thinking about this last night actually - I saved all the hard drives from my old computers before selling them and have several terabytes of data... however getting round to cataloguing the stuff worth keeping will take a long, long time. What I am going to do, though, is send them all up here:

https://www.tarsnap.com/index.html

It does use "the cloud" to store your files, which you seem averse to, but it's worth remembering that the people who run these services are the experts on preservation of data... they will have super redundant storage and backups and all sorts of things. Plus the tarsnap program encrypts all the data so nobody could go in and look at the contents of your backups.

It's a paid service but if you're serious about backups, it works out far, far cheaper than buying hard drives myself and investing time into regularly replacing the media they're stored on.

[Waterbagel]

EDIT: just read Techboy's post, and I agree with everything he said about cloud storage vs. physical hard disk. A company like DropBox, for example, while maybe not tamper-proof or intrusion-proof as a hard drive in your closet, might turn out to be more reliable.. It'll come with a subscription fee (if you're serious), but it will be there.

Cloud storage - Yeah, your stuff is "someone else's" too, but if you go with a reputable service, you can bet that they're going to do everything they can to make sure your stuff is always there.

CDs - Small space, subject to disc rot, but easy to do (though writing takes foreeeever).

Hard drive - comfortable blend of privacy (looking at you, cloud), speed and convenience (looking at you, CDs), and will also be compact. Western Digital and Hitachi are two brands I stand behind with disks failing. With these two, if the drive doesn't fail within the first couple of months, it likely won't ever, at least for as long as you need them. I'd say go with this option. Fairly cheap, dollar-per-gigabyte as well, all things considered.

Lastly, if you're really into the idea of permanent storage..

Tape Drive.

Most of your companies that have been around for a while will still back stuff up to tape. It's cheap as hell and will always be that way, just about. However, this requires some background knowledge before you can really get into it. If you use Windows stuff, you might be able to find a reputable go-between for the tapes and the computer. Gonna have to front some money for that, though. If you're on some *nix OS, might have a problem as you need to know how the tape was written before you can read from it.

If you know anything about tape drives and writing/reading to them, by all means go ahead. But, as far as my knowledge goes - which I will admit is not entirely comprehensive or professional - I would recommend hard drives. Especially ones with USB interface (externals). Internals are a little harder to store.


DOUBLE EDIT: don't bother with flash drives. Too expensive (per gigabyte), too faulty.. Just not worth it.

[Lens Larque]

I had a technology professor who made a point that with smaller and smaller storage and more and more info stored on it, and she pinched her fingers together to indicate something really tiny, that it would take a smaller damage incident to wipe out that much more info in a swoop.

And that a good way to preserve information would to replicate it as many times as you can!

[fujee]

Just a quick note for anyone considering burning their files to CD-R or DVD-R, choose wisely, and buy the most expensive possible. These are more likely to have their reflective base covered in an inert material such as silver, or even better gold. Gold based discs are said to be resistant to disc rot for at least 100 years. Some of the cheaper CD-R's have observable disc rot after 5-10 years of (most likely poor) storage. CD's are still pretty good as a fail safe for backing up medium sized data, and most music CD's from the 80s and 90s are still going strong, when stored correctly.

[zeitgeist]

I have a 4TB drive at home as my main working storage drive, and another 4TB drive at work. I have used BitTorrent Sync as well as SyncThing to automatically sync files between the two drives, and I love them both. Either way I have an offsite backup that I can work off as well--it syncs both ways currently. Every other month, I also manually backup one of those to a series of 500gb drives with a really basic USB3 drive enclosure, and keep all those in my car. This way I have redundancy in 3 places.

IMO, CDs/DVDs are garbage, but as others have pointed out, I may need to buy better ones. It just isn't worth it for me though. I can fill up 5 DVDs of data in one trip outside shooting photos. I bought a blue-ray burner for my new rig at home, but never use it.

I would think that anything backed up to a SATA interface is pretty safe, especially if you pay attention over the decades to any changes in backwards compatibility, etc. So far it has been decades without any real change there in ability to access old data.

I think the mode is not as important as location--if all your copies are in your flat, and it burns down...

[Waterbagel]

I think the mode is not as important as location--if all your copies are in your flat, and it burns down...

Super good point. I mean, hell, if you have a safety deposit box that you could deposit files to twice a year.. Maybe not quite as convenient as DropBox, but you know..

[zeitgeist]

For sure. Cloud setups are great! I put most of my day-to-day crap in my 30gig Google Drive bin, and work from there.

I'll admit, I have a relatively atypical quantity of data as an amateur photo/videographer, and I suppose my personal situation is a little different. I work in IT these days, and that position allows me the uncensored bandwidth to create my own "cloud storage" at work. It saves me quite a bit of money, and also keeps the data in my hands alone. I like that all my backups are within a 20 min drive of my house, versus some rack in SanFran or whatever, and the additional stress of maintaining those drives myself is worth the savings.

For others though, who just want to keep it simple, I would certainly recommend cloud storage. Drive and dropbox are great. Tarsnap is great, as techboy mentioned, if Linux is your flavor. It really just depends on how much you value the data you're archiving--I would be completely gutted if I lost my photo collection. If you're just cruising the internet, typing school papers, and making some music, you could certainly get away cheaply with burning some backup DVDs--just send some copies to your friends's house a couple times a year, and make sure you check on them to make sure they're not peeling!

[untitled]

Tape Drive.

Be aware that disk rot is also present in the case of tapes (tape rot?) as the backing/binding material is made of organic compounds which degrade over time. Hard disks are made of sand and silver, totally inert materials.

That being said, I'm waiting for diamond storage (the highest density material with a regular structure). A little cube of that could hold millions of terabytes - although I can't find the link anymore, being drowned out by all the news about "diamonds for quantum storage" bs...

I also use cloud (mega.co.nz) but only for non-critical stuff I don't mind losing.