For the short ill be filming this year, we're working out how much space we'll need to store it. I suggested building a "Hard Drive Tower," I named it on a whim and it turned out to be a real thing apparently.

http://www.willudesign.com/BlackDwarfTop.html

http://www.willudesign.com/Black_Dwarf/BlackDwarf12.jpg

The one pictured uses a 1.6Ghz Intel Atom and 2GB of RAM. It holds about 16TB of storage. Which, if we shoot at max resolution (5k), we'll be eating up a gigabyte per second of footage. Which, we won't be shooting at such resolution but we'll be close at 4k.

I like this idea because it's compact and simple enough. I also like the USB ports so we can transfer footage to it. We might switch the USB to eSATA for speed, but we'll have dual 320GB SDD's for the camera to record to, each holding about 3 hours of footage. So it's not a huge deal if we get stuck with USB.

Does any one here have any experience with one of these? What components am I looking at? I'll likely build two: one for use and another for backup. Maybe using 8 or 10 drives in a RAID array for each. So half the drives are mirrored on the other half, RAID 0?

He lists a lot of the items he used in its construction. Would I need to install a form of Linux on it? Especially if we plan to record data to them. We might have a laptop connected to both so we can simotantaniously write to both arrays. Would Linux still be required if the laptop handles both drive arrays in a software RAID?

The technical name for this sort of thing is a NAS (Network-Attached Storage) device. As far as RAIDS go, here's what you need to know:

RAID 0: Two drives. In colloquial terms, the "0" stands for how much data you get to keep if one of the two drives fails. Half the bits are written to one drive, half are written to the other. If you use two N-byte drives, you get 2N bytes of storage. Not recommended unless you hate your data.

RAID 1: Two drives. Here, the two drives are kept in identical states. If one drive fails, the other is an exact duplicate. If you use two N-byte drives, you get N bytes of storage. It's good, but a bit wasteful of disk space.

RAID 5: At least three drives. One drive is the parity of the other two drives, so you get 2N bytes of space from three N-byte drives, an efficiency of 2/3. RAID 5 is the best balance of safety and utilization; if one drive fails, you can hot-swap it with a working drive, and as long as a second drive doesn't fail before the RAID is rebuilt, you lose no data.

Yes, you would want to install a form of Linux on it. You should get a motherboard (or controller) that is a hardware RAID controller. The NAS will expose a single logical drive over the network; your laptops will have no idea they're accessing a RAID.

There's no special term for it? Haha, wow. Okay. Would I still need a Motherboard and a RAID controller if the laptop handled the RAID array? It's not going to be attached to any network, sense we'll be filming where there isn't any WiFi signal and at times without any cellular service. Granted, I could setup an AdHoc network but that's a different topic.

I know I can setup software RAID arrays for internal drives but I've never actually tested - or looked into - if I could do software RAID for external arrays.

It sounds like RAID 1 is the way to go. I was looking online and buying a bulk of 20 drives (way more than we'd need) is about 1.8k, which is more than what we're spending on the rental of the camera! The cost of the drives is a bit high for any of us to swallow. But we'd need about six drives for each array. Three for data, three for back up (RAID 1) then, a secondary array for a backup of the first array. It appears highly redundant but ideally you want at least three copies of the film stored in three different geographically (again, ideally) different locations: One to store at the editing house, one at the directors and one at the editors, or wherever you want it stored (Saftey Deposit Box?). That's so if one place burns down, gets damaged or whatever act of nature you have at least two more good copies of the original footage. But, I'm hoping we can make do with only two copies. Maybe our third will be on a non-RAID array at the editing house while the two RAID arrays are at our individual place of residence.

And since our source files are in a RAW video codec, which holds about 45MB's of data per frame at 24 frames per second. That's roughly a gigabyte a second for the (we believe) 5k resolution video size - which we aren't shooting at thankfully. If that were the case, we'd fill our 320GB drives with five minutes of footage! So, my friend & I are a bit skeptical at that figure.

We've read many reports than shooting at 4K - our chosen video size - nets 180 minutes on RAID 0 320GB drives. We don't plan to record that way, since we want to shoot video while the other drive is off loading.

For the drives, large 2TB drives are great for storing but pitiful for reading since they'll be slow. Our plan there, is once we have the ProRes footage (the RAW files converted to a lower quality for initial editing) we can store it on a single 500GB or smaller drive.

TL;DR:
Looping back to the topic at hand. We'll need at least two arrays, preferably RAID 1, to store the footage. I'm hoping to achieve at least 6TBs in each array. If I can software RAID these from a laptop that's even better but if it must be a hardware RAID then so be it.

It would be cheaper to build something like this correct, or would we be better off purchasing two arrays?

NAS *is* the special term for it.



RAID 5 is what you want: redundancy + usable storage space.



No they aren't. Bigger drives are almost always faster than smaller ones, sometimes so much so they overcome an RPM difference. The denser the data, the more passes under the head at identical RPMs.

Regardless, you'll be limited by the speed of whatever you are connecting via anyway. It's pretty easy to saturate gigE with a consumer NAS, and gigE is a hell of a lot faster than USB2.



Why 2 arrays?

Just get one of these: http://www.newegg.com/Product/Product.aspx?Item=N82E16822122064

And fill it with 6 of these: http://www.newegg.com/Product/Product.aspx?Item=N82E16822148844

That'll give you (6-1)*3 = 15TB of storage for $1,900



5k vs. 4k only tells you half the resolution. 4k is the number of pixels horizontally, how many vertically? A resolution of 4000x1 is technically "4k", after all. Assuming you meant 4k shot at digital cinema AR (2.39:1) that would be about 7 million pixels a frame, or 28MB uncompressed raw per frame if it's using 32-bit, although ideally it would be 24-bit and thus would be 21MB per frame - 500MB/second.

RAID 5 is ideal, but I don't think we can push that much money into storage. As it stands for us right now, RAID 1 is ideal and RAID 5 would be a luxury. We're making this film off fundraising rather than pocket money.



Redundancy. It's a rule of thumb to keep multiple copies. As mentioned in my prior post, it's ideal to keep 3 copies: one at the editing house, and two at different off site locations. Though, depending on our funding we may have to abandon the RAID for three or more non-redundant drives. Copied to another set. Though, with the copy of the footage I obtain/safekeep I plan to protect the data as best I can at my expense. Which would likely be in a RAID 1 across six drives or so. I've had bad experiences with loosing data at my own fault and I'd cut myself if I lost any of this. Ha.




I like that idea, it'd be easier than building the case from scratch. I'll check into that and explore ways to make it cheaper.



That calculation likely only accounts for the video without the audio. The camera is capable of 4 uncompressed audio channels at 24-bits. I can't find any reference for the size per second of audio at that specification, but going off this (camera specs) it looks like it'll be about 21MB's a second for the audio. Which, doesn't add much in the end of things.

As for vertically, we'll be shooting and editing in 16:9 (4096x2034) (though our ProRes files used in editing will likely be 720p) while keeping a 2.35:1 ratio in mind. We'll also use center extraction for extra wiggle room. Once we finish the editing, we'll render and export the video in 2.35:1, reducing the final video size to around 2k at final output.

What is going to fail on a harddrive that will make the data completely unrecoverable? All the harddrive failures I've had were either of the filesystem(a problem any Linux live CD will laugh at) or with the controller card, which can be replaced if you have the part number.

Do you realize that you get to use 2/3 of the HDDs with RAID5, but only 1/2 of the HDDs with RAID1?

Which is why you have a RAID. The drives themselves in the RAID are redundant of each other. I won't argue that it's not also a good idea to have a moderately stale backup off-site somewhere (cf. the 3-2-1 rule of backups) but having two parallel RAID arrays in the same place seems a bit silly to me. Not to mention that you're then getting even less HDD utilization, 1/4 for RAID1 or 1/3 for RAID5, which doesn't make sense with your claim about cost.

With RAID 1, three drives are mirrored to another three. With RAID 5, I must not understand it completely.

With RAID 5, 4 drives are xor'd against another 2. So you get an extra storage drive.

RAID 5 is "cheaper" - it gives you more usable space for your money. RAID 1 storage is (n/2), RAID 5 storage is (n-1)

So for example if you buy 4 1TB hard drives RAID 1 would give you redundancy and 2TB of storage, whereas RAID 5 would give you redundancy and 3TB of storage. Bump that up to 6 1TB drives and RAID 1 would give you 3TB of storage whereas RAID 5 would give you 5TB of storage.

The only advantage RAID 1 has over RAID 5 is in minimum drives needed. RAID 1 only needs 2 drives, whereas RAID 5 needs at least 3. But you're talking well over 3 drives anyway, so RAID 5 is cheaper.

Thanks for the detail, I'll go with RAID 5 then. It just seemed like RAID 1 was more secure since if any drive failed I'd have a complete backup of it on another. The whole idea with RAID is to recover data from failed drives, anyways. So whether I choose RAID 1 or 5 seems a bit moot with more than three drives in the array.

Er, double check that. 48KHz at 24-bit and 4 channel on that site gives me 1977mb/hour, or 32mb/*minute*, not second.

Oh, hell. I assumed the 60 in the field box was for seconds, not minutes, without reading carefully.

It's odd that audio doesn't add much in terms of file size then. 20, even 32, megabytes seemed reasonable to me but I'm no audio expert.

Modern speakers can't reproduce much over 128kbps anyway (there are studies that prove this). Therefore, the only way you can realistically increase audio file size per unit time from that is to add more channels, as far as I can think. I think the rule of thumb for 128kbps MP3 is about 1MB/minute, so uncompressed multichannel audio at 20 or 32 mbpm seems reasonable if not large to me.

I suppose coming from a background where uncompressed means I have free reign to edit colors, exposure and such in a photo I just assumed I'd have similar ability to manipulate uncompressed audio. But I suppose there's just less information to store in audio than in visual media.

Er, no there aren't. 128kbps is bitrate anyway and is 100% meaningless by itself. Without specifying what compression you are talking about, what sample rate, how many channels, etc... bitrate just doesn't mean anything.

I assume you mean 128kbps 44khz stereo MP3, though, in which case you are still wrong. Besides, you sure as hell don't want to record in a lossy format even if your claim was true. Could totally record into a lossless compression, though, like FLAC - the issue would be the extra overhead while recording, and audio isn't that big so it's not really a big deal.