In our June 2022 Spotlight article, titled “Comparison of LTO and Cloud storage costs for media archive“, we provided a comprehensive comparison of media archive storage costs, and uncovered some interesting and perhaps alarming differences in cost and convenience. In this Spotlight article, we will look at the process of migrating archive data between different types of storage. This is something becoming increasingly common among our P5 customers. This coincides with Archiware releasing P5 Data Mover, a tool that enables such workflows for those who already have archive data stored using P5 Archive.
The concept of a ‘media archive’ is a little special within the broader IT storage landscape. Such archives have to exist for many decades, and they’re responsible for preserving what may become historical artefacts, years from now. They can be the digital equivalent of a storage room full of old reels of film. As soon as an archive starts to receive data, plans must be made for the retention of this data for the long term. This usually involves purchase of some LTO tape hardware, or provision of cloud storage from a commercial storage vendor. Over time however, one realises that the data in the archive is more important than the storage media being used, and perhaps some migration is necessary to best preserve the data into the years ahead.
There might be many reasons that migration becomes necessary:
- Ageing LTO tapes and LTO drives/hardware, reaching limits of lifespan
- Cloud storage vendors costs becoming high as more data is added to the archive, more attractively priced vendors competing for business
- Number of LTO tapes required by archive unmanageable
- Desires to move from on-premise archive to cloud-based only
Let’s look at some of these in more detail:
Migration between LTO generations
Every 2-3 years, the capacity of a single LTO tape increases significantly. The table below summarises the last 5 LTO generations, over 11 years. LTO-10 is slated for release near the end of 2023 with a 27-36TB capacity. We stick to the ‘uncompressed’ capacities, since media files are often already compressed and therefore do not benefit from further compression provided by the hardware of the LTO drive.
| LTO Generation | Year of release | Uncompressed Capacity (TB) | Ratio to LTO9 |
|---|---|---|---|
| LTO-5 | 2010 | 1.5 | 12 |
| LTO-6 | 2012 | 2.5 | 7.2 |
| LTO-7 | 2015 | 6 | 3 |
| LTO-8 | 2017 | 12 | 2 |
| LTO-9 | 2021 | 18 | 1 |
The ‘Ratio’ column in the table indicates the number of tapes from each generation that would fit on a single LTO-9 tape. LTO cartridge manufacturers typically state that tapes are designed for 15 to 30 years of archival storage.
We are presented with a classic trade-off: if we leave data on older LTO tapes, we have older, perhaps unreliable hardware, and large numbers of tapes which might be approaching the end of their lifespan. By upgrading the tapes and hardware, there are several benefits, including a lower cost per TB stored, but it’s a time-consuming and costly task and therefore easily put off for another year!
Migrate between cloud vendors
Compared to LTO tape, the marketplace for cloud object storage is somewhat crowded and complicated. The complication comes from pricing structures where the following factors may influence the monthly bill:
- TB of data stored (all vendors begin with this cost metric)
- Minimum retention period per object / minimum monthly charge
- Wait time before objects can be retrieved
- Replication of data between different locations
- Metered upload / download costs for getting data in/out
- API call costs (add/delete/list objects)
The vendors with simpler pricing may charge only for the first two of the above, the more sophisticated charge based on all of them. On this basis, it can be complicated to anticipate what the costs will be.
In addition, every month the bill for cloud storage will rise if more data is added to the archive. All data in the archive has to be paid for, over and over, each month. This can be surprisingly expensive over time.
Migrating from one cloud vendor to another is less complicated than with LTO tape, since there’s no hardware to purchase, dispose of and configure. However, the software tool used to create the archive will need to permit such migration of data en masse while presenting consistent access to archived assets.
Hybrid migration (tiering)
This is where things get interesting. If we balance out the pros/cons of LTO storage for archive versus cloud storage, we might decide that a combination of both is our preference. Maybe we would like to use on-premise LTO hardware for our initial archive phase, so that we have fast access to recently stored media, but then migrate this archived data to cloud storage after 1 year, when it becomes less likely media files will be required to be recovered.
Such a hybrid approach would require a configured workflow whereby files are archived once, and the archiving software platform takes care of the movement of the data to cloud storage after 1 year has passed.
This approach has the advantage that hundreds of obsolete tapes stored on-premise will not build up, only 1 year of data will ever be stored on LTO. So complexities are avoided and migration to newer LTO generation is simpler.
Conclusions
With such a complex storage landscape, it’s essential to have flexibility within an archive workflow, so the wide choice of storage options can be used to best advantage. Technology and pricing will undoubtedly change further into the future. Perhaps economic pressures and global changes mean that all types of storage become more expensive in future. The ability to be nimble is worth a great deal.
Those working within the P5 Archive ecosystem, should investigate the P5 Data Mover which is designed to facilitate everything discussed in this article.
