I have backup problems. They are big and numerous and they relate mainly my server, TECT. Firstly, let's go over TECT briefly. When I got TECT, it was a playground. It had 4 gigs of ram and a 250 gig hard disk. The idea was that I was going to put in a lot of ram and do various experiments with it. It's a really beefy computer, so it wouldn't be unreasonable to run a dozen or so VMs on it, either in a really small production environment, or as a test system. It's aging, but it's still my fastest computer by far, and it has more RAM than anything I own, and can hold more than anything I'm likely to own for a few years yet.
Over the years, I added a better storage controller, the idea being I'd have some sort of fault tolerance with some level of capacity, but I just kept adding disks, because let's be perfectly honest about this: the bare desktop-grade disks that TECT uses are actually very, very cheap.
Along the way, TECT became a single-OS server and became a fairly critical part of my home network, in that it runs my personal e-mail at stenoweb.net and my SharePoint web site (personal blog, etc) as well as being where I do my document management and it's also the location of my home directory, where I centrally store data for use on all of my local computers.
As it stands today, TECT has 12TB of total usable data space, and a big problem: I am not backing up most of the data. It breaks down as follows:
C:\ (2TB)
- OS
- MS Exchange Data
- SharePoint Data
D:\ (10TB)
- Home Directories
- Video
- Music
- Software
- Static HTML sites on personal.stenoweb.net
The biggest challenge is two-fold. Firstly, there's no really good software that will back up this amount of data that's inexpensive. The cheapest I can get is Acronis Backup & Recovery for Small Business Server, which is about $500.
The second part of this challenge actually has some really interesting interplay with the first part. I'll step through it. The first aspect of this challenge is that I would really like to treat the whole thing as a single unified dataset. This is maybe a bit odd, but I'd rather back up the whole system at once, or at worst, the C:\ and D:\ disks separately. (Daily for C:\ and weekly for D:\ would be ideal.)
The second part of that is that because each dataset is huge, and because I'd like the weekly to actually encompass both C:\ and D:\ -- I need really huge media to hold the whole backup. In an ideal world, this would involve three or four 10-15TB external hard disks that were just hooked up, did a full and then daily incremental backups, I would move the media off-site after hooking up a new disk, and the cycle would rotate. Unfortunately, 15TB external disks don't exist without strapping a bunch of smaller disks together, and that's a bad idea for a "backup." In addition, Acronis Backup & Recovery will not just span a backup across multiple fixed disks, which is how the program will recognize regular USB hard disks. (It will span a backup across "removable media" -- such as SDXC or Compact Flash media in an outboard reader, but I can't express enough how terrible an idea this is.)
And so, I come upon backup hardware. The problem here is that all backup hardware is expensive. I like my data and I want to have it for a very long time, but backup hardware is even more expensive than that. To give you an idea, I've been trying to solve this problem for just under two years now. The three basic choices are LTO-6 tape media, regular external hard disks, and RDX removable disk cartridges. As I mentioned above, most backup software won't span backups across regular external disks, so even though they are the least expensive option, they are also not a practical one. Writeable blu-ray media is also technically an option, but it is insanely expensive, exceedingly slow, and although suitable automation hardware for it does exist, that automation hardware does not play well with any known backup software capable of protecting a live Windows environment, especially one running Exchange and SharePoint.
When I first started comparing the cost of backup media in 2012, LTO-5 was the dominant tape format, DAT320 had a slightly stronger market than it did today, and it became clear to me that RDX was the perfect example of the razor blade marketing model, and blu-ray was going to be hilariously deficient for the task, in addition to insanely expensive. Bare hard disks (and even built up externals) are the cheapest way to store a lot of data, but also one of the least convenient to use.
Let's look at a graph!
I don't remember if I included the cost of buying a blu-ray mechanism in this calculation, but although they are among the cheapest ways to accomplish this task, or were in mid-2012, let's recall that they're still literally the worst for the task.
It's interesting how this has changed though. Since that time, a new version of LTO tape has been announced, DAT has long been dead, despite the fact that the manufacturers all leave up proclamations from 2008 saying that a 640 gigabyte version will be out soon, and I haven't even bothered including the costs of external or internal hard disks. It would be neat for comparison, but let's get real: LTO-6 media is $57 a pop. They hold 2.5TB (native) of data, and that's way less expensive than a decent bare 2TB disk, at almost $100.
I don't know if it's a change n the market, the fact that the cost of LTO mechanisms literally never goes down, or if it's simply some better research on my own part, but let's take a look at a version of this graph today.
The cheapest starting point for LTO-6 is a "bundle" for just pennies shy of $2,500. It is technically possible to get started for less, but let's be honest: you have to buy a SAS adapter, and then you have to buy cleaning media, and then you have to buy some starter media, and the bundle includes some neat software.
The cheapest starting point for RDX is a single cartridge. I have started RDX at $0 because you can buy docks that are universally compatible for something like $30 on eBay. It's almost always possible to find a deal where you can get a dock for free when you purchase the media, etc.
There are two interesting points on this graph. The first is 23TB, which is the amount of (native) capacity at which the cost of 1.5TB RDX cartridges crosses over with the cost of LTO-6 media and the mechanism. It's way further out this time, and 23TB is probably enough for two full backups of TECT. In fact, if I trim down some data (laptop backups, for example) then it's probably actually enough for three sets of weekly full and incremental dailies of the whole system. That's pretty neat.
The other interesting point is $2,500. For $2,500, as I mentioned above you can buy an LTO-6 drive and a cleaning tape. I think the bundle might include a single piece of media, but that's really just good for one or two backups of the C:\ disk. For $2,500, you can buy just about 17 terabytes worth of RDX media, and with current promotions going on, you'd get two free docking stations.
The question then, really becomes one of ecosystem, incremental costs beyond the cross-over point, and other neat things the backup hardware might be able to do.
Let's presume I'm insanely pessimistic and TECT is 100% full, and there is no possibility that anything could be compressed. In order to back up the system, I'd need 12TB of raw, native capacity. LTO-6 tapes are 2.5TB each, and the RDX cartridges that are the best deal right now are 1.5TB. (Don't ask me how this happened, because RDX carts are just ruggedized laptop hard disks, and there aren't any 1.5TB laptop disks of which I'm aware – they jumped right from 1TB to 2TB.) You'd need 36TB total in order to have one full backup each week for three weeks, which lets me keep a copy of the backup off site, and another will be either in transit or sitting next to the server, waiting to be changed out. From the cross-over point at 23TB, there's about a $1,000 difference in the cost of LTO-6 and RDX, purely for the purpose of backing up TECT. LTO-6 will cost about $3300 and RDX is a bit over $4,300.
The trick is that in the real world, I may never even reach the cross-over point.
The other thing to consider is archiving. LTO is good at it, and RDX just isn't a good choice, because a tape can sit on a shelf for years without deteriorating or going bad. A laptop hard disk can randomly kick the bucket for no good reason whatsoever.
A neat possible application for LTO-6 hardware, if I were to choose it, save for it, and then buy it, would be to archive old data. Every once in a while, I like to make a copy of a large dataset in an effort to preserve it, but I don't necessarily want it to live on TECT, taking up valuable spinning disk space. I could put it on an external hard disks, but I would use LTO-6 tapes if I had them available, since those are cheap and will last a very long time.
On the other hand, RDX is very flexible, and because of the lower starting cost, I can get started backing up data on TECT immediately. In addition, because the docks and cartridges are fully compatible, I could at any given moment start buying 2TB or 2.5TB cartridges, when those become a more cost-effective option. Another neat option that I don't necessarily have with LTO-6 is to connect RDX docks directly to other computers, such as my Mac mini, any given laptop, or even to build one into my next desktop computer. This way, the home backup software on any given computer (which will also span volumes using removable media) can back up directly to my chosen backup hardware. This is a neat idea especially given that at some point in the future, those larger cartridges will become less expensive, but the old media will still be good.
It's a question, I suppose, of what kind of operational flexibility is more valuable. In my case, I very much value the speed and inexpensive availability of LTO 6 media today, but I also like the idea of the more incremental nature of purchasing RDX cartridges, the docks being very inexpensive, and the cartridges working with a variety of backup applications on a variety of different computers. On the other hand, I like very much that with LTFS, and LTO6's physical durability, that media can be used (instead of a more traditional situation involving optical media or Amazon Glacier) as an archive option. RDX media is simply too expensive and ultimately too fragile to be thought of as a permanent storage solution for an infrequently used piece of data. At $57 a pop, with an LTO mechanism, I wouldn't bother buying more external hard disks to hold data I only use infrequently.
If I didn't have quite as much data and capacity as I do, and more time to spend sitting in front of a server swapping tapes, I think that I would consider RDX (and even DAT320) more strongly. If I had the budget, I suspect I'd go right for LTO-6, but in actuality, I think it's going to be dictated by the immediacy of my backup needs on TECT. Any given laptop or desktop I have will be happy backing up to a single external disk, so the need to specifically seek out RDX for that purpose is minimized. RDX could be nice if I end up building a giant desktop system with a whole bunch of disk space I want to have its own backup system, but I could also do that independently of whether TECT is using RDX or LTO.
I suppose the core of this post is that over time, the values for both RDX and LTO have changed. I'm sure LTO-7 will actually get announced before I actually bother to (and can afford) to buy anything, changing the Backup Calculus™ once more, but hard disks and the costs specifically of RDX media is also going down at its own rate, so it's possible that the odds could continue to tilt in RDX' favor.