UKH

Hard Disk Failure: Remember it is WHEN, not IF.

We've had a hard disk go in our 4bay QNAP NAS.   Only 2.5 years old.  Decent WD Red NAS drive as well!

Thankfully hot-swapable, so new drive now in and RAID rebuilding.

Had many HDs go over the years (even an SSD last year)

So make sure you've got a backup regime sorted!

(I'm not just relying on the NAS RAID)

Thanks for the reminder, most of my photos are on a single external portable!

Press that BUY button quickly!

I occasionally use CrystalDiskInfo to check my drives. https://crystalmark.info/en/software/crystaldiskinfo/

Also, my Synology NAS box gives me a weekly/monthly health report of my two disks. You can't be too careful .

I work in media archiving, I'd say we get a fail rate of 5%-10% of hard drives over a 12 month period.  We don't keep anything on just one hard drive and I spend my life telling clients (mainly sports federations and rights holders) that hard drive is simply not a long term storage solution...  Yet still they opt for it, and I have to hit them with the "I told you so" when something becomes irretrievable.

Ironically everything was so much safer when we all used video tapes!

The NAS was reporting good health on all drives, then Bay-3 just went.  And with weekly scans/checks & regular RAID scrubbing.

Do you have lots of dents head height in the wall next to your desk?

Must be a constant battle!

We have 4-Bay NAS RAID main storage, plus 2x other internal/external hard drive/NAS back-up, plus BackBlaze cloud backup.

And I still get nervous over losing stuff! (inc fire/theft)

I had a single-disk NAS that just died (after several years service mind), luckily I had a backup on my personal laptop - have since replaced with a dual disk NAS and a RAID setup but always a little nervous and still keep a backup of important things on a second device.

Unfortunately, there is little correlation between SMART values and catastrophic failure of a drive. You can usually detect read/write errors but the head crashing on the platter or electronics blowing up are usually spontaneous events.

In terms of back up regime, could you give an example of what might be considered a reasonably safe back up regime for photography?

See my post above.

Some might see that as over the top. 

What do you think Iamgregp as a pro in the business ?:

I'd say that depends entirely on how much stuff you edit or create, and how much you're willing to loose.

If you only take a few every month, and they ain't that critical, then perhaps once a month is enough.

If loads and all are critical, then as frequent as every few hours...

And pretty much everything in between.

And remember, that as was pointed out RAID =/ backup... And one physical backup in the same location /= backup.

So if you really want to be sure. You backup the RAID every so often, and keep one backup off site (cloud or physical drive at work or something).

Photography really isn't really my are of expertise, I work in video, but I'd echo what HeMa says... Your backup regime really depends on how much you have, how often you're adding to it, and how often and how quickly you need to recall things so it's very difficult to be specific as the perfect solution will be different for each individual.

To speak in very broad terms, ask yourself what happens to your archive if the building burns down?  If the answer is you lose your data then you ought to look at something to address this.  

I'd say the relatively small file sizes of photographs compared to the video files I work with (roughly 1GB per minute for HD video, 4 x that for UHD) would make cloud solutions a very attractive solution. 

The service providers will backup your data in a number of separate locations, using a mixture of media.  If you have a large amount of data that you just need to store, but not get your hands on very quickly the cost can become much lower as the provider is able to use less costly storage mediums.

Sorry I can't be more specific and apologies if this me teaching grandmother to suck eggs, but I've not a clue about photography, unfortunately.

This is (roughly) my main set up for backing up important files etc (it is actually more complicated than this due to data volume):

MAIN STORAGE:

4-Bay RAID-5 NAS. 

If one the drive fails , the systems keeps going fine. You just take out the failed drive and replace it, while NAS running, no downtime.  System sorts itself out over the next few hours.

Local  Backup

Using AllWaySync-Pro (alternatives available), I do routine weekly back-up to:

Local Drive-1 (a two bay RAID NAS)

Local External Drive-2. 

Cloud Back-Up

I Use BackBlaze. Fixed fee $60/year, unlimited online storage.  This runs automatically in background.

It can do multiple hard drives (internal & external) and multiple specific directory locations.  It is a great service.

The initial back up to cloud can take weeks!

As this version of Back-blaze wont back up from NAS, it backs-up the NAS via the backup on Local Drive 2 (plus it back up other drives I have & the main system C drive)

Fire/Theft Protection

Addressed mainly by Backblaze, plus taking the Local-Drive-2 away on holiday etc. 

Plus storing Local Drive-1 remotely when on holiday.

Shortcomings.

I live with the risk of losing 'some files' between backing-up weekly, should NAS fail catastrophically or stolen/fire.

Out of interest did you consider/have you ever used CrashPlan in place of BackBlaze? I currently use CrashPlan but slightly begrudgingly.. I’m on the lookout for a cost comparable alternative. 

When straight to Backblaze after friend recommended it. Didn't look at Crashplan.

The rule of thumb is to keep your data in 3 places. I.e. NAS, off line backup and off site backup.

If you have a home NAS, buy your disks from two different manufacturers. If there is a bad batch of disks, sods law says that when you bought your n disks they all came from the same batch.

I've used multiple RAID 50 and RAID 60 arrays with 2 or more hot spares, at work, for data we really want to keep safe. The arrays are replicated in real time to multiple sites, in case of a site outage. When you get high end arrays, they send an email to the support provider when a disk goes. Even though you have multiple hot spares, there is still a risk of more disks going. They courier out a disk and send an engineer to install it. All at 4h notice. You get all this and more, when you use a cloud storage provider.

Ticked !

It's a 4-Bay 12TB QNAP SME NAS, streets ahead of the Buffalo NAS we used before.

The drive that went was a Western Digital NAS drive, replaced with a Seagate NAS 'IronWolf' NAS Drive (who thinks these names up!), so ticked the different brand. (drive checked as 'compatible' on QNAP website).

I'll get another in as a spare, as now one WestD has gone, more bound to follow ..

Only a couple of hours RAID re-building left now.

Good to know.

Cheers

For the amateur photographer Blu-Ray or DVD's have limited storage but will outlast any drive and far less likely to be stolen if your house is broken in to. Old school but I still use them as well as cloud & drives for photos I consider important. 

I'm a happy amature however I am sh1t scared of losing my photos and movies so I adopt the 3-2-1 principle. That's three copies in two different mediums. And one copy off site. At the moment I keep one set of data storred in a fire-proof safe which I bought from Argos for £25.

I am also loving the Microsoft Window's 10 File History app. This allows me to back up my desktop computer to my NAS drive once an hour. At the moment I only keep this going back three months. I'm guessing that it works very similar to Apple's Time Machine. It's got it's limitations so I also use FyleSync to ensure that the NAS box is copied onto external hard drives.

Every so often I make backups of my Operating System so that if something goes tits-up such as Ransomware or a rogue graphics card driver, then I can wipe my "C" Drive clean and install a clean and working version of Windows without all the faff of putting my software and updates on again.

Genuine question, what is it that makes you carry out the offline backup?  If you have an off-site is the offline on site copy necessary? Is there a cost of retrieval or lead time to the offsite one that makes it DR only?

Not questioning your set up, just interested!

If you have a Synology, you can use the Cloud Station app on you documents file / folder / disk, You can then set up the NAS to back this file up to a USB stuck in the back of the NAS. Obviously if the NAS exploded while writing to the USB then you fooked, but depending on you data size, this write might be very short.

Won't help for fir of theft, but it's belt and braces especially if you do an external back too, to some other device.

thanks for the reminder though, it's been a while since I did a full back up to ext, device. Now done.

Surely, surely backup hard drives are a thing of the past? They're not reliable (a lot less reliable than my computer in my experience), and I've got loads (5?) gathering the dust on the top of some cupboard. Surely now it's all about the cloud? On a Mac there's the iCloud; in addition, I've been using the superb Dropbox for about 5 years. Paying a bit more for huge amounts of storage. Never lets me down; works anywhere. If you're working on some particularly important project, e.g. as a writer, it's a piece of piss just to use a memory stick as yet another safety backup.

PS. I've also got two computers, a desktop and a laptop - synchronised. So my backup system consists of a spare computer hard drive (should one computer fail), iCloud and Dropbox. So that at any one time all my files are in four different places - and in the case of Dropbox, in two different storage places, I believe: one in Iceland and the other in the Philippines. So, five different places. Surely more than adequate?

The trouble with hard drives - unless you have a small portable one that you take away with you (a right pain in the arse in my experience) - they're in the same house as your computer, so worth sweet FA if you have a fire, or a very thorough burglary. I'm frankly astonished that separate hard drives are still selling.  Plus: they should be about 99.9 per cent reliable, and they are typically not, even if you get a good expensive make. Many fail after about 2 years.

This was kind of what I was wondering with my earlier question in this thread. If you have a good cloud solution which is updated frequently is an on premise backup necessary?

However sometimes factors such as cost, ease of retrieval or time may be a factor in making an cloud only solution necessary.

At my workplace, for instance, as we’re getting towards a petabyte of data cloud solutions aren’t ideal as the bandwidth to get everything up and down from the cloud would be prohibitive.

Out of curiosity and to work out if it's cost effective, how much does Cloud storage cost for about 3TB? 

Yes, obviously a problem if you have to back up a huge amount (btw I don't even know what at 'petabyte' is ) But for most mortals something like Dropbox is surely the best way forward. Yes, it takes many hours to upload it all initially, but after that, it's a breeze. I'm always astonished by how fast it is - mostly a matter of seconds. The reason I chose Dropbox was because most publishers favour it for good reason: it doesn't matter a damn if whether you have a PC or a Mac, they can access your latest file/s instantly. It's virtually standard.

Gosh, I can't remember, but I think - off the top of my head - that it would be something like £40 a year. Unfortunately, I don't want to look all this up now as I'm in the midst of decorating ... and I have to go downstairs now and put on some more coats of paint and varnish tonight. Carpeters coming tomorrow am.

Absolutely, for single users Dropbox and iCloud are fantastic. They cost very little and take care of all the headaches for us! 
 

A petabyte is 1000 terabytes...

Blimey - putting on my false Cockney accent

Last time I looked (and cancelled my Amazon cloud), it was much more than 40 quids a year. More like 50€ a month for a few TB, after the Amazon stopped their previous "unlimited for photos" offer.

My opinion (as an IT architect, not that I claim to be an expert on this particular topic) is that there is value in defence in depth. Any solution you put in has failure modes and shortcomings and the easiest way to mitigate them is to have multiple different solutions in play.

I keep most documents in a Dropbox folder*, time machine backups of the laptop (including the Dropbox folder) and also a cloud-based real time, versioned backup of the laptop, including the Dropbox folder.

The time machine backup is on a 2 disk RAID 1 NAS, which is itself replicated to the cloud backup solution. I also synchronise a copy of the NAS roughly monthly to an external hard drive which I keep offsite.

Any cloud services that I use (google calendar etc) and any local devices (my router, the NAS itself) have their configs dumped monthly and stored in the Dropbox folder.

What you have to consider is the scenarios where you might need your backups. Concerned about a disk failing? RAID can protect you from that. Concerned about someone stealing your NAS or your house burning down? RAID is useless. Concerned about recovering from an immediately obvious corruption/failure? A  single point-in-time backup is all you need. Concerned about having to recover to a particular point in time? Then you need a backup with a change history.

The cloud solutions cover the most threats on paper, but the risk with them (at the consumer level) is that they are cheap and come with no guarantees. If they lose your data, best of luck finding someone who cares. Also, if you have any significant data loss then replicating it all back from your cloud service could take some time. Finally, what happens when you decide you want to leave the service?

* with the exception of Dropbox, all of the above are full disk encrypted. The cloud backup service is one I specifically chose because the encryption key stays client-side. The lack of security on Dropbox is a risk, but bearing in mind the massive amount of diverse data they have I’m not concerned that anyone who was to compromise the service would have much luck finding anything specific - there’s just too much data to sift through. 

No not at all a thing of the past. Very much alive and kicking.

But part of the back-up mix I use (see post Weds: 15.26), that includes primary 4-Bay RAID NAS drive, 2x Back up drives AND cloud storage (unlimited storage for $60/year, currently running at 3.5 TB).

Eggs across many baskets. 

Dropbox no good for us as a primary backup option due to the way we have things organised files/folders/PCs NAS-server. Blackblaze gives much more flexibility in the number of 'sources' you can back up.

And if you crash and loose everything (or fire/theft), BackBlaze also give you the option of sending you hard drives with all your data on, rather than downloading it

Use Dropbox locally for some things though.

Thanks.
That sounds quite reasonable and would be a suitable replacement for my current system of keeping a back up external HDD in work.
Happy decorating! 
P.S. are carpeters people that put a floor/stairs in and then lay a carpet on top of it?

Ooooohh

Smack talk.

Runs off to get popcorn.

RAID rebuilt, NAS all hunky-dory again.

Glad this thread has got some people backing up stuff

Thanks for this, really good, detailed reply that answered my question. Plenty of food for thought there.

Is 'Smack Talk' a new term that you've learnt recently but don't really know what it means?
 

I'm so On Point with this reply.

Swweeeetttt

On Point - really?
By the the way, it's 'Sweet', the way you've added extra letters for added emphasis/perceived coolness is akin to Dad Dancing at a wedding.

What is the largest file that you have reliably uploaded to, and shared, via any “cloud” service?

I often have to share files of greater than 5Gb (for a single file). I have not yet achieved this successfully on any Cloud service. 

I politely suggest that the hyperbole of your opening paragraph is a little misplaced and akin to me saying “surely, surely files that can always be ported to the Cloud are a thing of the past”  

"There are only two kinds of hard disc - those that are going to fail, and those that have failed". 

I think the problem is that many services are still locked on the max 4.7 Gb per file from the old FAT16 file system (even while they are not using it anymore).

That being said, there are numerous cloud-based worksharing sites where much greater files are delivered (e.g. in the video editing industry) without a hitch. There are also pure file-delivery system (e.g. FileCatalyst), where no such limitations exist.

Alas, this is not what you're lookin' for. But I'm sure there are options available.

One aspect is the ease of access. Back in the day, we used to use tape (DLT) and one copy was stored in a firesafe. A courier would take the second copy to an external storage facility.

There is also the question of can you guarantee that your offsite (disk, tape or cloud) is available and within the right time span (recovery time objective).

What tends to happen is that people do backups. They run like clock work. Right up until a disaster strikes. It is only then that you find out that your backups, taken diligently, don't contain the correct data or the data is corrupted in some way.

These things can strike even the largest of companies. This is quote about company, who handle about ~20% of the worlds shipping. They may have had hundreds of backups, but if they are not accessible (due to an outage) or corrupt (due to a virus) then they are useless. (The full story is a good read, btw)

"After a frantic search that entailed calling hundreds of IT admins in data centers around the world, Maersk’s desperate administrators finally found one lone surviving domain controller in a remote office—in Ghana. At some point before NotPetya struck, a blackout had knocked the Ghanaian machine offline, and the computer remained disconnected from the network. It thus contained the singular known copy of the company’s domain controller data left untouched by the malware—all thanks to a power outage. “There were a lot of joyous whoops in the office when we found it,” a Maersk administrator says."

https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashe...

A further point which may be obvious: in line with most mechanical things the failures have a spike when the devices are new, then a low failure rate that slowly grows as they age.

You should never get a brand new drive, replicate your data to it and immediately dispose of the old device, because that’s actually your highest risk period for the drive failing on you. 

Edit: one more related point. Since spinning disc drives have moving parts they have a higher chance of failing at startup than they do while running in a steady state. If you’ve had a NAS that has been running happily and continuously for 2 years and then you power it down and go on holiday, for example, you have a higher chance that the drive (and other parts for that matter) are going to fail when you turn it back on and it has to spin up from cold. We used to see a lot of this in data centres where servers would go years between reboots, then you power cycle them and the fans have seized/clogged to the point that they were able to keep spinning but couldn’t manage the start from a stopped position. 

I am probably tempting fate here but since I started using a PC about 25 years ago I have never had a hard drive fail. Am I just VERY lucky?

I do back up to an external drive, but must admit it is when I remember.

Ignoring the concern that cloud storage could fail/be hacked (which I think is a pretty low risk) and I agree it it the way forward, the main problem for some of us is a very slow broadband connection. Fine if you can upload reasonably quickly but if you are getting an upload speed of 1(or even less) Mbps then its not really an option.

Just looked online (https://downloadtimecalculator.com/Upload-Time-Calculator.html#file=500x2^3...); to upload 500Gig at 1Mbps would take over 50 days.

Dave

Agreed, we often share files of several hundred GB. Aspera and Signiant are the systems we mostly use...

I think it is possible to send a HDD with your data to some cloud providers for the initial "upload". Might be an option if you have a few TBs.

The Amazon Glacier Storage service, which is what you would want for data backups, is $1 per terabyte per month, so $36 for 3 TB per year.

It is the initial upload that takes the time. You can leave it running in the background for days, if required. Once the bulk of the data is uploaded, it is just the changes that are sync'd each time. So the day to day bandwidth requirements are lower.

My initial upload (>3TB) to cloud backup via BackBlaze took over six weeks!

And yes, you are very lucky not to have had any disk failures in 25 years.

Thanks! Back when I looked, there was no Deep Glacier, just the Glacier. But their pricing and terms (since all the tiers are more for data centers than normal users) are very complicated for the layman:

Glacier ($4/TB/month), S3 Deep Glacier ($1/TB/month), paid Upload requests (one request means one file or what?), etc. I can't even make out what are the retrieval costs if I need the data back. I understand that Deep Glacier has slowest availability for retrieval (tape drives or something), which is fine for archival purposes, but I am at a loss with the other pricing (requests, retrieval over time etc).

Do you know of any examples or pricing calculator? Say I have 3TB of photos (on average 20MB each, usually with a few kilobyte XMP sidecar file), that would be $36 per year, fine. My NAS fails, I need to retrieve the whole 3TB to the new NAS. There are so many prices in their price list! Data retrieval for free is just 1GB per month or so? More data retrieval from Deep Glacier is $20 per TB for Standard, $3 per TB for Bulk (?). Then there is a charge for Data Retrieval Requests (again, Expedited/Standard/Bulk), is that per file? Object? What?

Might there be some hidden charges I missed or bad retrieval scenario, leading to a $150 bill like that poor guy who just wanted his 60GB music collection back?

https://medium.com/@karppinen/how-i-ended-up-paying-150-for-a-single-60gb-d...

It's just a bit too complicated for a normal customer

Agreed - the AWS prices are hugely confusing!  I often get people at work asking me how our prices compare to theirs and there's really no straight way to answer the questions other than "it depends".  Some of the actual $ per GB prices can be very low, but the cost of retrieving the data is very high. 

Really these kind of services are for archiving, rather than daily backup services - using an example from my industry - say we shoot a programme, master and deliver it and it's gone to air.  Chances are we'll never need all the rushes again as the programme will never go out again.  We don't want to simply delete the rushes, as it will never be possible to re-shoot those scenes, so we can stick it in the deep glacier so we can pay a very small cost to know that it's reachable in the unlikely event it's ever required (hence the name).

The lower prices reflect the storage medium - the primary storage method for the deep archiving like this will be LTO tape, which is then offline meaning if you need something the tape has to be brought online, the data restored then transferred to you.  LTO tape is great as the cost per GB is low, it has a low power requirement and is pretty reliable... Buuut it's slow to restore data, and most systems will have a significant disparity between the number of tape drives operating, compared to thew amount of tapes being managed resulting in a long queue of files waiting to be restored, hence the lead time in retrieval.

Yes! Tape storage probably (I think the older Glacier used very slow RPM off-line drives). But for my use, the scenario is somewhat similar - the cloud would be used only for last resort archive, with data added at first (several TB) and later perhaps only every month. Retrieval is only ever if all else fails (the local RAID, the NAS, the off-site off-line backup). For that, the AWS Deep Archive sounds ok. It's just that it's so hard to figure out the real pricing and retrieval pricing (and it used to be so much worse!), since in such a last resort retrieval, it's several TB at once.

I tried to dig a little deeper into the new Amazon Glacier Deep Archive pricing, and it seems the 2016 $150 retrieval example for just 60GB might have been with old Glacier pricing (which was even more esoteric than now, and led to huuuge bills if used slightly incorrectly - like trying to get the data too fast!). Still, the pricing is quite hard to figure out, and there are some potential caveats you need to iron out in your setup.

I think it might be around $1 per TB per month stored, with around $95 per TB retrieved once (you have to add in the $0.09/GB AWS Data Transfer Out price, which is quite higher than the quoted Data Retrieval price, since you are retrieving it over the Internet back to your NAS), for the typical "backup my whole photo archive" users. And you have to factor in an additional charge of $0.05 per 1000 PUT requests (does that mean every file I upload?), and for retrieval $0.03 per same. Not bad, if I counted it right, as you need to retrieve only if your on-site backup fails. But I might still be wrong

It's primary purpose still seems to be more suited for data centres and companies to use instead of tape archives, not so well suited for the average photographer or content creator. But I will give it a closer look.

Likewise.

Yeah sounds like you've got your head round it pretty good...

Like you say it's more for businesses to archive stuff that they don't anticipate needing much of very often, or very quickly!

That said they could be a good option as their sheer size means they have economies of scale, redundancy and security that others may not.

Microsoft Azure seems to be equally confusing to me!

AWS pricing does seem complex, but that’s mainly because the services are not really intended for consumers and they’re expected to be slotted together in complex infrastructures where one size fits all pricing isn’t really feasible. They provide a calculator that can work out costs for you on the website, though it is itself fairly complex (you have to understand what you want to be able to estimate it).

A common factor with AWS pricing is that you only really get charged when you’re using something that it costs amazon money to provide. Accordingly you can move as much data around as you like inside an AWS region - you only pay when you need them to send it over the internet, which costs them money.

Of course, as a home user you’re always going to need to send your glacier data over the internet as you’ve got no other use for it. 

I just hope if I do try it out I don't end up with a huge bill because in my stupidity I issued "too many requests too fast at once" or something like that

Still looking back with nostalgia to their Unlimited Photo Storage plan for $12 a year. It was just too good to last (and it didn't). Just a year after they "upgraded" that to Unlimited Storage for $60 a year, which was discontinued just 9 months later for $60 per 1TB per year  Needles to say, I passed... (and had a look at the original Glacier storage, but was put off by the totally convoluted and unclear pricing for retrieval back then).

Yeah that's fair enough. As a programmer I'm happy to knock up a Python script that uses AWS to do a periodic backup, but I realise most users will want something a bit more user friendly. Ask me in a couple months and I'll let you know what it costs...

AWS is simplicity itself when compared to Azure...

Yes. It's just that there do not seem to be any easy to price (and understand) options for the smallish SOHO user or a bigger hobbyist with just several TB of data to archive and a need to retrieve in bulk only ever in case of local disaster.

BackBlaze was (last time I looked) more like a glorified copy (at least they have versioning now) of your HDD (not that useful for archival purposes), BackBlaze B2 is more like to classic AWS, AWS Glacier Deep Archive is still new and the pricing is (as you rightly noted) modelled for slightly different sector, etc. Any ideas? Thanks

Microsoft making something unnecessarily convoluted and difficult to decipher?  Surely not!

Well, the service I use is CrashPlan, but I hesitate to recommend as I’m not totally happy with them. The backup client is a pig of a Java program that thinks that the sole purpose of any device it’s on is for running it and claims RAM accordingly. They also don’t have a consumer product anymore (I use small business) having exited the market and referred customers over to Carbonite, which I’ve not tried. 

It does offer continuous backup with full history, unlimited storage, client-side encryption and its fairly cheap - though not as cheap as the consumer tier was (which is why they ditched that business, I imagine)
 

Nice idea that...  Cloud storage without all the bullshit.  

I have come to the conclusion that Window's 10 File History is too complicated and it somehow decides what to and not to back up, even if the files are in the correct folders. As mentioned in the documentation.

What other software can I use to automate backing up my documents/photos/movies to a NAS drive?

I'm hoping for some sort of way to go back several versions just in case I over write or delete a file that I regret happened.

How long have you been running Windows 10?

I've been using Allway-Sync-Pro for years.

I don't worry about saving 'versions' though and I don't use the real time option (my rate of file change isn't that fast to warrant it).

They'll be other programs out there for sure.

I'll give it to a go.

Cheers

My contribution to the thread-

1) You need to test backups every so often. physical media fails, corruption can occur.

2) tapes and CDs/DVDs deteriorate over time. Backup media needs to be replaced regularly. Those wedding pics on CD from 2000 might not be readable now, copy them to a new CD.

3) Be wary of synch settings - if you accidentally delete an important file on your machine and the synchronisation process deletes it from your backup drive or cloud service you're stuffed. The synch software which came with WD passport drives had this ridiculous feature.

I’m a professional filmmaker and photographer and have captured around 5tb of media a year for the last 20 and I’ve never had a hard dive fail. Seems to be more of a if than an when for me. Must be luck 

17:34 Wed: "I am also loving the Microsoft Window's 10 File History app"

20:07 Thu: "I have come to the conclusion that Window's 10 File History is too complicated"

You are quite ridiculous. Fair enough to pose questions - but I hope no innocent ever takes your advice on any computing matter. Perhaps think carefully before offering any.

Sounds like you are an outlier!  Especially with 100 TB of data stored!

Q: How were you capturing 5TB of data in 1999, when you were 20 years old? 

Another pro (Iamgregp Weds 13:33) had a different take:

I work in media archiving, I'd say we get a fail rate of 5%-10% of hard drives over a 12 month period.

What’s an outlier? 
I do have far too many hard drives collecting dust and I’m sure there are better ways to manage my data than what I currently do, but I’ve never had a drive fail. Ok so 22 years ago when I first started out it was probably a lot less than 5tb pa, maybe only 1. Not sure it matters. The rough numbers were to illustrate what I thought was significant to this thread - that I’ve not had a drive fail in all that time and with all that data. 
Law of sod dictates every hard drive I own will now spontaneously combust   

Well, most writable CD/DVD/BDs are now long gone now anyway (perhaps except the M-Disc or the original DataTresorDisc - if they still exist - for low capacity but very important data with their non-organic writeable layer). The capacity is just not there.

Since almost nobody in SOHO or hobbyist market can afford enterprise LTO drives, that leaves just HDDs for off-line and off-site storage... At least with those you can rotate the old, filled-up drives to off-site, having a second (and after a while a third) copy of part of the whole archive, as you increase the primary backup storage in the NAS or whatever. Unless the whole interface changes, like IDE to SATA

And you rightly note that sync settings can be quite dangerous if done incorrectly - you don't want to overwrite the good file with a malware infested or ransomware encrypted file! Or just an accidental delete propagating back to your NAS and from it to the cloud (especially simple mirroring cloud like some, Dropbox, BackBlaze and the like).

Some of that could be mitigated with careful versioning and access rights - for example, using different credentials for the NAS, where the backup rsync (or whatever you use) job only has create file and create directory rights, but no overwrite or delete rights. For reading the backup, you would use a different credential with read only rights. And use checksums! But it's quite harder to set up and most simple backup software doesn't bother with that...

To protect against ransomware, the backup software could also compare the file extension with known patterns of the file (like the unix "file" command which determines file type from the magic patterns even without knowing the extension, ie a jpeg picture from the jpeg headers etc) to protect against overwriting valid files with ransomware encrypted files (although really smart ransomware could circumvent even that). That's where good versioning with the different credentials and finely divided access rights would help, especially for photos - you would only ever store the originals (RAW) with checksums and if anything changed, it would be only the XMP sidecar files and perhaps processed jpegs or tiffs.

That's why I generally dislike software like Aperture or Photos or Lightroom that stores your whole photo archive in a single library, makes it very hard to archive effectively and safely! 

Back when I worked with photos for a living (no video though), I had around 1TB per year. Most of it archived by the company, though (I did my own archive as well, for my own sake, portfolios and competitions, you don't really want to access your archive through most media companies intranet, it's usually clunky). And I had a few HDDs fail. Fortunately, I always had a second backup. 

Outlier means that you have some distribution curve of failures (gaussian or whatever), usually a "hill" with most bunched up in the middle (where most people experience only a rare drive failure), but the outlying regions are these on the right and left where the the minority experiences either no drive failures or many drive failures. In other words - you have been lucky

Even 1 Tb in 1999 was huge!

The flagship digital camera in 1999 was the Nikon D1 at 2.9 mb.  No video.

1,000,000 mb = 1 TB. So you'd be shooting nearly 350,000 images in 1999 to get 1 TB.

And 0.5 TB drives didn't arrive till 2005, with the first 1 TB arriving in 2007 at $400 each....

And in 1999, a 'Big' hard drive was 20 GB....

So I'm not quite sure how you were storing all that data back in the day!.

I think you've got the wrong idea about how LR works.

No, I do think I have pretty good idea about it, I have used it for years - you have two options, and I was talking only about the second one. Either it indexes the photos at original location (if you ingest them from the card via another software or just copy) and the Library contains only the previews and metadata (writeable back to the originals), or it copies them from whatever location (card, HDD, etc.) to its Library (the originals).

The second option is the problematic one - you need to backup the whole library, which can be pretty big (it includes the originals!). The first option is usually quite ok (especially if you write the metadata back to the originals in their original location, but not all do!). But I always preferred Photo Mechanic (like most pros) - ingest a card to two locations at once, do the edits and selection on one, enter the metadata, cull the keepers, export the keepers and keep the second copy for a little while just in case. And for thousands of photos like from a sport event or a wedding, it was so much faster than LR anyway...

Perhaps I should have phrased it differently - that I dislike any software that forces you to use a proprietary "Library" to store your originals, which is admittedly only one of two options in LR, but (AFAIK) the only option in Aperture or Photos or several others. Storing your originals in a clear filesystem structure (year/month/day, perhaps concatenated into something like "20191101_Halloween") is simply better for archiving and backups.

So have I , since V1.0 beta

Nope, you've got it wrong.

LR does not make a 'library' copy of the images.  That's the WHOLE point of LR, it's non-destructive editing. Originals or image copies are not stored in the .lrlcat catlog file.

I've worked out what the anomaly was that was causing me problems with using/trusting File History, after I re-built a new computer and found out that File History isn't all that straight forward in restoring/re-building data. But you don't want to know or care.

Kudos to you for going through every single comment I post.

Really? So there is no second option to "copy the originals to the library"? Are you really sure ? 

Yep. I'm still right

That option says 'Copy' to new 'Location' and add to 'catalog'.

'Location' is a Hard Disk location, not 'in' the 'catalog'.

Look at the panel on the right where you specify the 'Destination' of where your want to copy to.  These then become your 'originals'.  The 'Destination' is a hard drive location.  The copied files never become part of the catalog.

The main use of this option is to copy off a memory card to your PC.  It's the one I use all the time to get from Memory Card to PC Hard Drive.

The other Import options are Move (to a Hard Disk 'Destination') or Add (from an existing Hard Drive 'Destination').  Often use 'Add'.  

You can use all three methods to get images added to the catalog.  They are not mutually exclusive

BUT: at no stage ever, under any of these options, do the image files become part of the LR .lrcat catlog file. Never, ever, never.

But you know all this, right?, as you've used LR for years.

Do you want to keep digging yourself deeper? 

Edited to change 'you 'to 'your'.

I'm guessing you've never expanded the 'Destination' Panel on the right if you've used this 'Copy' option and the files have just copied to a 'default' file location (probably somewhere on C: drive) without you actually understanding what LR is doing. 

Then you mistakenly thought the images were being 'added' to the catalog.

If you've been backing up the LR catalog, and been thinking the images are in the LR catalog, then you've not been backing up the images.

... oops

I find it is always better to learn how to use the software correctly (and understand what it is doing), before you spout about it on a forum.

By 2000 I was using mainly using Beta SP directing 16 camera opps at premiership football matches and in charge of multiple £1m+ OB trucks. Showing off now but you seem to be trying to pick apart my numbers (yes probably exaggerated) and missing the point - the only interesting part of what I posted....

I've collected a lot of data over a long period, I've saved it all on hard drives and never had one fail.

It maybe only one persons experience. But I thought it might be of interest to this thread

Lucky me

Gotcha. Thank you

You chose to quote big TB numbers ! Don't moan if they get questioned. 

Yes it is interesting, but using one persons lucky experience is not a good strategy to protect data on hard drives.

You've got to assume they 'will' fail at some point. Else you could be in for trouble.

You've got to plan for the 'fail rate of 5%-10% of hard drives over a 12 month period' that lamgregp has to deal with across his client base. 

Cut me some slack please, it's almost like I have trodden on your favourite program (although I do happily admit you were right and I was wrong!)

I never used LR to do my ingesting... Just added to the catalogue (with full knowledge that option just indexes the files from their location on the HDD). I have always used Photo Mechanic for the initial ingest, into proper folder structure by year date and event, edited and captioned with PM (much more powerful). But you are right about LR of course - stupid me, memory is a funny thing, I must have confused that with Aperture or something 

My mistake - satisfied now?  But the rest of my points (well, except LR) still hold

If you are giving advice about important (in terms of computers) technical stuff like backup - best to make sure what you are saying is right!

I did try to cut you some slack, but you just kept digging yourself deeper ....

It is, use it near enough every day.

I also use a Year based folder structure on the main HD (though there are other ways to organise your photos on the HD: long as it is logical and structured, and suits your needs).

Importing straight into LR from memory card is pretty easy and has advantages in speeding the RAW processing up.

- Add new folder & name at desired HD location from within LR - 'Folder Panel' on left, right click "create new folder inside'

- Then right click the new folder and use 'Import to this folder', LR then open up the import dialog all set up to copy to that location from the memory card (previously plugged in).

- I then have a Rename Template that adds a YYMM prefix to each file name on import (LR remembers I did this last time). This makes sure every file is unique as cameras do cycle round eventually and cameras brands can replicate file names across different cameras.

- Then apply a camera-specific 'Develop Setting' Preset that I've set up previously (LR remembers I did this last time) to get the RAW file most of the way there processing wise, better than any in-camera jpg.

- Build 1:1 previews on Import (saves time later). I've set LR to store these for 1 month in 'Catalog Settings' with Preview cache is set as High Quality and 2560 to match my screen.

I don't keyword.  I know where everything is .... (perhaps).  You can add some keywords on LR import, but they are applied to all imported images.

You are wrong about Backblaze (which I use) being a 'simple mirroring cloud'.

As default it has a 30 day version history, which you can upgrade to 1 year or for 'forever'.

You can also set it up to back up 'continuously', once-a-day, or on-demand.

Even the 'continuously' is not in real-time though, it adds some lag and backs-up at regular intervals over the day (else it would always be at it).

Plus you can:

- Select whole drives or specific HD folders, multiple drives, internal & external

- Exclude file types

- Set bandwidth preferences/priorities.

- Set up a Private Encryption Key, should you so wish.

- Get sent hard drives (up to 8TB each) should you want that option

Not sure if this has been mentioned, but....whilst cloud back up is great, I would suggest you need a pretty fast broadband connection.

I used Backblaze in a house with fibre, but moved to a house with standard broadband. It made Backblaze unusable...just too slow to upload my photos.

So I am now relying on a main external hard drive, home back up on another external drive and then another external drive at work. Works pretty well with though I did invest in some back up software.

we are on 30 Mbps.

The initial upload (> 3TB) took 6 weeks, but now its fine and it just chunks away when I add many GB of photos. 

I had the same in my old house, which was fine; but I'm reduced to about 2Mbps now which couldn't keep up, so I cancelled my subscription. It was shame as I much preferred to use the cloud back up and one over hard disc. Hey ho I'll pay more attention to broadband speeds for my next move.....

ohh, 2 Mbps!

Not aimed at you as you're undoubtedly aware of it and more aimed at people potentially looking at using Cloud storage.
The broadband speed that needs to be used to calculate how long it will take to transfer your data is the Upload speed and not the Download speed which is the normal one stated in all the ads.
The Upload speed is normally much slower that the Download because most users generally need to get a lot more data than send.
For example, my Download is 200 Mbps, but my Upload is 20 Mbps.
That's with Virgin. If I changed to BT, I would get 50 and 10 Mbps.
 

And What should be remembered is that the sold speed =/ actual. 
 

benchmarking and measurements is the only way to go (like speedtest.net). But Backblaze seemed to also do a speed verification/test on their page.

I'm in perhaps the unusual position of my actual speeds being higher than those stated.
My advertised service is for up to 200 and 15 Mbps. I'm getting 220 and 22 Mbps and that doesn't seem to change at even peak times.
It would take me two weeks to upload my 3TB of data if that speed does indeed stay constant. 

Thanks for replied re upgrading my current back up system. I have x3 hard drives, main machine and x2 external which use a mirror function to duplicate.

I sell world wide through agent so defo need to have immediate access to RAW TIFF jpegs all edits.

All edits saves/disgard etc I make (photography) are updates to x2 external hard drives....but both are onsite thanks for info re off site solution....I will certainly look at that, really helpful

thanks