The current generation of LTO tape (LTO-9 released in 2021) stores 300% more data than the generation from 2015 (LTO-7). That’s 18TB today on LTO-9 versus just 6TB in 2017 on LTO-7. The same 18TB of data on an LTO-9 cartridge can be stored in the cheapest cloud services for approximately $220 per year, with additional costs for accessing it again.
In 2023 Sony discontinued its ‘Optical Disk Archive’ product and in 2024 (the time of writing), Chinese scientists have 125TB fitting on a single optical disk in the lab, using new technologies that have yet to come to the market, but inevitably will.
With this varied and interesting landscape for archive storage, commercial archivists have not just to think about which storage to use, but how to move strategically between formats, from one decade to the next.
Media Archive
Let’s recap a little by what is meant here by the term ‘media archive’. Customers in many vertical markets (including media and entertainment, healthcare, and government), create media assets which are relatively large in size and cost money to create. These are audio and video assets, medical imaging data, 3D survey data and many others, with video being the most common. Because these files are typically larger than office spreadsheets and word processing documents, they need to be moved from traditional storage where they’re immediately accessible, to cheaper storage as soon as practically possible. At the same time, because they are costly to reproduce, they need to be retained for the foreseeable future, so they’re available for re-use, repurposing etc.
A common use case is the production of a TV episode or movie. The production creates hundreds of separate files containing video, which are edited together into a final product. Upon delivery of the final product, it makes no sense to delete the hundreds of video files, since they might be useful in the future. At the same time, they cannot be left to reside on the original storage that allowed them to be edited. They need to move to ‘cold storage’ where they can remain into the future, at minimal cost, but with ease of location should they be required again.
The media archive is typically stored on LTO tape or cloud storage, the most cost effective and reliable options at the current time. The cost of keeping these assets for the next fifty years might be considerable. It is therefore sensible to review where they’re being stored, if they really need to be retained going forwards, and whether to move them to a better storage medium in order to save money and keep them secure. For example:
- How long do LTO tapes last?
- How long will the LTO drive used to read the tapes last?
- How is the drive repaired if it fails?
- How much is cloud storage costing, per year, per decade?
- Can cloud storage be trusted with a single vendor?
- What is the cloud vendor doing to mitigate against storage failures themselves?
- Should assets archived 20 years ago but never accessed still be retained?
P5 Data Mover, a Migration Tool
This function of providing storage flexibility for those assets that have already been archived is provided within Achiware’s P5 Archive ecosystem, by the ‘Data Mover’ product, which is able to perform the following functions:
- Filter data in the archive, using stored meta-data, to determine if it should be migrated/duplicated from its existing storage to alternative storage (e.g. from LTO-5 to LTO-9). Based on the results of the filtering:
- Perform data migration, by moving assets from one type of storage to another. The previous storage can then be erased/reused or no longer paid for in the case of cloud.
- Perform data duplication, by placing a second-copy of an asset on another type of storage while keeping the original, providing redundancy of storage.
A common use case is a ‘storage refresh’ whereby assets are moved from older tape technology to the latest. E.g. from LTO-5 to LTO-9 where the contents of 12 old tapes fit onto one new one. This has several benefits:
- Older tapes may become unreliable due to age (although LTO tapes are deemed to be stable for 30 years of storage).
- Having many fewer tapes is easier to manage.
- Maintenance of a working LTO-5 drive (for example) is no longer required.
- Some archived data may be discarded, based on meta-data query.
P5 is able to do this because of its Media Asset Management (MAM) function, which tracks every archived asset by storing metadata, a thumbnail and a preview/proxy from the original assets along with one or more pointers to where the asset is stored, across multiple types of storage if required.
The Migration Process
Note: While we refer to ‘migration’, note that P5 Data Mover can also ‘duplicate’ the archived asset to a second storage location, just as easily as migrating. For ease of terminology, we’ll use the term ‘migration’ to cover both activities below.
Using P5 Data Mover to perform a migration is simple to set up, but potentially lengthy in its execution. An existing archive might be stored across hundreds of LTO tapes, requiring those tapes being available in an LTO drive for the migration to complete.
Therefore the migration needs to be planned for, tapes made available and time allocated. Fortunately, the migration can run during periods when tape hardware isn’t being used, or cloud bandwidth isn’t competing with other uses, e.g. weekends and evenings.
The set up of the migration tasks requires the following steps:
- Select the ‘source’ data pool, e.g. a set of LTO tapes or a cloud bucket
- Select a destination storage pool, e.g. a set of LTO-9 tapes
- Create a filter, if not all data from the source should be migrated. This filter will use meta-data from the filesystem, or technical file-based meta-data to make the selection. E.g. Migrate only files that were originally archived in 2020.
- Create a schedule when the migration task should run. E.g. evenings and weekends.
- Determine whether the data should be migrated or duplicated. The latter results in a second reference to storage being created alongside the original, within the asset in P5’s MAM. The asset can then be restored from either location should it be needed in future. Duplication creates redundancy of storage and potentially removes a single-point-of-failure.
Once submitted, the migration tasks will then begin reading from the source media and migrating to the destination, during the allotted time slots.
At all stages of the migration, P5 presents a list of storage volumes (e.g. Tapes) required for the migration to complete, so that the progress can be understood, and the remaining tapes required for the migration may be organised ready for inserting into the tape library or drive.
Future-Proof
With new and improved storage technologies appearing each year, and entirely new technologies on the horizon, a successful media archive must remain flexible. Over the period of a decade, any media archive will need to investigate migrating archived data to new storage. Storage cost, ageing media, on-prem versus cloud preferences.. there are many factors that feed into decisions around archive storage.
By using a product like P5 Archive with Data Mover, nobody needs to remain locked into their current archive storage.
P5 Maintenance Reactivation Special Offer – Ending on the 31st of May
P5 Maintenance Reactivation, allows P5 installations whose maintenance contract has expired to bring their licences back under maintenance and support. This option is extremely valuable for those with older P5 installations that would benefit from an upgrade to latest versions with latest operating system support.
Until the end of May 2024, Archiware is offering a 50% discount on the cost of bringing an installation back under maintenance. This applies to installations where the maintenance expired up to 3 years ago and represents considerable savings.
Send an email to team@archiware.com to request a quote.
