Data Storage Advice

New Topic
This topic has been archived, and won't accept reply postings.

Hi all, 

Looking for some advice on data storage. I currently have an internal hard drive reader to which I connect two 4TB drives. Both of these are a mirror image, so I have everything backed up twice. A 4TB drive will currently last me about a year, but I'm starting to burn through them faster now that 4k video is taking hold...

Is there a more future proof system I could look at? Is there a cost-effective way of backing up huge amounts of data? What are people's experiences with RAID systems?

Thanks in advance.

Nick

In reply to Nick Brown - UKC:

How critical is the data?

What risks are you trying to mitigate against? That's your starting point.

HDD failure?

Fire/Theft/Flood, etc?

Then you need to consider how much loss you can cope with. Is the data critical from the point of creation? Or could you cope with the loss of 'most recent' data, and reconstruct it from other sources?

Does availability matter to you? Or can you cope with a fairly lengthy downtime whilst you recover from backup?

Generally, a scaled approach to backup is required, to deal with the different types of loss.

In reply to captain paranoia:

Thanks for the reply. The main risk I'm mitigating against is HDD failure. 

Losing 'most recent' data would be bad, as this would generally be the video files I've just shot.

Once video files go onto the backup drives, there's usually no real urgency to get them off quickly. How long is fairly lengthy downtime?

Cheers, 

Nick

 mullermn 22 Nov 2019
In reply to Nick Brown - UKC:

Depending on how technical you are/how hands on you're prepared to be it might be worth looking at something based around Amazons S3 Glacier Deep Archive service.

This is designed for stuff you don't want to lose but in all likelihood will never actually need to retrieve, so it's crazy cheap for long term data archival. 4TB would cost you about US$7.38/month. You can have a petabyte for US$1100/month! You do have to pay for the data in/out though and it's not intended to be cheap for 'live' data.

It comes with a 99.999999999% durability rating. That's durability as in 'we won't lose it' rather than availability, which is 'you'll be able to get to it any time you choose', and the service has a 12 hour access time - basically it's probably on tape somewhere (or in reality, multiple somewheres - hence the durability) and a job has to be scheduled to go and fetch those tapes and copy your data back to where you can get it.

The advantage that this has over the NAS route is that at any sensible price point a NAS only has so many disk bays, so sooner or later you're going to max it out and then you're looking at replacing otherwise fine disk drives just so you can replace them with a larger device in the same bay. Plus a NAS is another piece of technology that has a small but ongoing requirement for your time in terms of maintenance.

Edit: Just realised I priced that up in the EU through force of habit. If you have no data protection requirement to keep your data out of the US then you can cut nearly 50% off your storage cost by putting it over there.

Post edited at 13:37
 Toerag 22 Nov 2019
In reply to Nick Brown - UKC:

There was a massive thread about data storage recently, look it up.

 mattrm 22 Nov 2019
In reply to Nick Brown - UKC:

Unless you're really technical, then I'd go for a NAS, Synology NASes are well regarded.  Go for as many bays as you can afford.  But that does come at a cost, a 8 bay box will cost close to £900 without any disks.  So you'll find you're paying £2k if you fully populate that.  Even more if you go up to 10tb (or larger) disks.  Then just go for RAID 1 (Mirrored) as it's normally pretty easy to recover from.  You'll be better off if you try and buy the same make and model hard drive each time.

Backing up large amounts of data can get expensive I'm afraid.  If you're putting multiple TB into the cloud isn't cheap.  However companies like BackBlaze say they'll do 'unlimited' data, so if you use your main PC with a large internal hard drive as a backup, then you can abuse that.  It'll probably ignore things like RAWs, but once you've got finished products, eg videos, then it should back them up fine.

It would be useful to give us an idea of budget as well?

 sbc23 22 Nov 2019
In reply to Nick Brown - UKC:

The ever-increasing hard drive size is the problem with a NAS type solution. It's ok if you have NAS with loads of bays (like 8), you can just keep dropping another drive in. If you have 2 or 4 bays, you are constantly upgrading drive sizes to make it work. The synology ones are really good and you can get a cheaper expansion housings to take another batch of drives. This is a £1000+ solution including the drives. If you have plenty of bays, you can optimise the size of the drives to get best value per TB. Have a look at Synology DS1817 + DX517 (expansion). The synology hybrid raid (SHR) allows for a single drive failure, whilst allowing considerably more than 50% of the total available drive space. 

 Sam W 22 Nov 2019
In reply to Nick Brown - UKC:

Combining 2 of the suggestions above I have a Synology NAS backing up to Amazon Glacier. This is straightforward to set up and gives you the advantages of both solutions simultaneously.

 mullermn 22 Nov 2019
In reply to mattrm:

> You'll be better off if you try and buy the same make and model hard drive each time.

Yes and no.. there are situations where this can be simplest but if you go for different drives from different manufacturers you reduce your exposure to the risk of a flaw in any one drive. This is a rare occurrence but it has happened and is not theoretical. 

At the very least you do not want to buy both drives of the same model from the same place at the same time to try and mitigate a risk of manufacturing defects. 

 marsbar 22 Nov 2019
In reply to Nick Brown - UKC:

Do you actually need every video and picture you save?  Do you go through and delete any that aren't as good, or you no longer need?  

1
 mattrm 22 Nov 2019
In reply to mullermn:

> > You'll be better off if you try and buy the same make and model hard drive each time.

> At the very least you do not want to buy both drives of the same model from the same place at the same time to try and mitigate a risk of manufacturing defects. 

It's a pretty minor risk and honestly not some thing that tends to happen as much these days.  But yes, you may want to avoid getting hard drives from the same batch.  This can normally easily done by simply buying them from different companies.  I have had it happen to me, but then I suspect during the course of my career I've been exposed to more kit than many.

If you have a system which has some form of mirroring/syncing of the data and then a good (regularly tested) offsite backup, then you can probably just not worry about that.

 Toerag 26 Nov 2019
In reply to marsbar:

> Do you actually need every video and picture you save?  Do you go through and delete any that aren't as good, or you no longer need? 


The problem is that you never know what might be important in the future. Yes, by all means delete redundant pics with the same info in them as the best one, but that crappy pic of you with your cousin in the background pulling a face? That may end up being the last pic of them when they get run over by a bus.  That embarassing pic of you at a party when you were in the first year at uni? That's the only pic of you when you were slim back in the day.  That team photo of you after a days climbing - the pub in the background burnt to the ground but local historians need to know what it looked like.


New Topic
This topic has been archived, and won't accept reply postings.
Loading Notifications...