As you may know, TECT is desperately in need of a good, scalable backup solution. This is especially true as TECT starts to play a more active role in the network storage of my various data, store shared data within the crazyhouse, and run more virtual machines that I use to complete actual work. Additionally, backups of many of my other machines may be stored on the machine.
There are many types of backup solutions available, all of which have various negative and positive aspects. For example, Blu-Ray media is the cheapest per gigabyte, and I already have a blu-ray burner available. However, over 200 disks would be needed to make a single full backup of TECT as it stands today, and performing the initial full backup would take nigh-on forever. A networked storage location (or a single big external volume presented by an enclosure like a drobo) provides almost the exact opposite -- it's a completely automated process which can go very quickly, but it is one of the more expensive options.
The main options are as follows:
- Blu-Ray disks
- Pre-built network Accessible Storage
- Drobo
- Custom Network Accessible Storage
- Tape
- RDX Cartridges
The main considerations for a backup infrastructure are the following...
- Cost: This isn't necessarily the first concern, I may not end up with the cheapest solution, but it's important that I at least try to find something that doesn't take years to save up for.
- Convenience: The backup rig may have to live in the garage with TECT, enduring relatively cold temperatures in the winter and high temperatures in the sumer.
- Scalability: TECT has been upgraded in 2012 from four-and-change terabytes of online storage, to twelve terabytes of online storage. Not all of it is actively being used, but the backup system needs to be designed with the ability to handle one or two full 12-terabyte full backups in mind.
- New: I have bought all of the parts for TECT new -- I odn't see why I should change the pattern for something like its backup system.
- Environment: TECT lives in a relatively harsh environment with high humidity and changing temperatures.
Possible Backup Formats
Blu-Ray Disks
4TB data (uncompressed) would take approximately 200 25-gigabyte blu-ray disks. Writing 25GB of data to a blu-ray disk is also excrutiatingly slow, especially as it often does CPU-based compression inline. None of this would be automatable, at all.
BluRay is, of course, nigh-on unusable as a backup format for a system so huge.
DAT Tapes
DAT holds far more than Blu-Ray per piece of media, and is faster (DAT320 is often cited as being able to transfer 86 gigabytes in an hour.) However, there are few if any modern DAT automation products. With larger DAT formats, TECT's C:\ disk may be backed up (in full) onto a single cartridge, either in full daily or with a weekly full and daily incrementals onto one cartridge.
The D:\ disk may be backed up in full once per week onto a new set of cartridges, and is generally not so critical that daily backups are necessary.
DAT drives and cartridges are likely to be around nigh-on forever.
Although, once the data-set on the D:\ disk grows (it is already over 2TB) it is likely to take nigh on forever to transfer this information to tapes. Under perfect conditions, without verification, a 3TB backup will take 60 hours using DAT320 and 19 days using DAT72.
LTO Tapes
LTO is far higher capacity than DAT, and automation is available. LTO is an open standard ("Linear Tape Open") and continues receiving updates. In the time between the original writing on the Classic Stenoweb Wiki Page and today, the sixth generation of LTO has been released, with a native capacity of 2.5TB.
At present, LTO 4 tapes tend to go for about $25 a pop, LTO5 for $35 a pop, and LTO 6 tapes are still hovering above $100/tape. In this format, the mechanism, the automation hardware and even the necessary computer interface card are the most expensive components.
LTO is fast and big enough that a single cartridge should suffice in backing up TECT's C:\ disk for a whole week, and the cartridges that will be needed for the D:\ disk can be swapped in and out quickly enough on a Saturday, even with a single, non-automated mechanism.
RDX Cartridges
RDX cartridges are unique in that you do not need to match the "dock" and the cartridge media. At the moment, backupworks has a good deal from Imation for a USB 3.0 "dock" and two 750GB cartridges for $229. Not-matching the dock and the media is clever because it means I can buy the largest and fastest possible cartridges to back up the D:\ drive (only two 1.5TB cartridges would be required today, with no compression) but smaller cartridges (such as the starter 750GB ones) for the C:\ disk. RDX may even be left connected for daily incremental backups to the cartridge.
There is RDX automation hardware available, but it costs nearly as much as low end LTO automation hardware.
However, it still merits mentioning that TECT does live outside in fairly unfavorable conditions, and RDX or even RDX automation hardware may make a lot of sense if the size of the available cartridges doesn't keep pace with the rate at which data on TECT is growing. Another nice factor of the RDX system is that something like an Acronis or Backup Exec boot CD won't need to have special drivers for an HBA and tape automation hardware installed in order to recover from a system disaster.
There is more information available on the Classic Stenoweb Wiki Page for this topic.