by Marc M. Batschkus
– Read part 1 –
Implementing the Archive with P5 Archive
Once you know the who, what and how you can start looking at the details of your Archive implementation.
Archive modes:
There are three archive modes. Most people use manual archiving. A person (usually the admin) decides when to archive what and triggers the job.
Alternatively, there is automatic or watch folder archiving. A watch folder is combined with a schedule and data is picked up accordingly. Depending on the workflow and load, that schedule can be anything from several times a day to once a month.
Both modes can be combined with a filter. The filter is effectively a set of rules. Criteria are specified to filter files either for inclusion into the archive job or for exclusion (like temp, word, render files, mp3s etc.).
P5 Archive offers many options to configure an Archive that meets specific needs. There are options for access, display, storage and many more.
A number of features help to configure archive storage. Disk and tape storage can be used independently of each other. The storage pool is the organizational entity for the actual storage volumes which in turn are members of a pool. The illustrations below show where to find which feature.
Considerations for the configuration
Multiple mechanisms allow separation of access for groups of users:
- Separate login areas so that each group of users can only see their respective files.
- Separate archive indexes so that each group of users can only search their own catalog or index (and never see the catalog content of the other groups). This comes with the trade-off of having to search multiple indexes if the origin of a file is not known.
- Separate media pools: one media pool for each of the categories (e.g. departments) to keep files physically separated.
An archive plan is required for each different setup since it defines the index and pool being used.
For each new archive database, at least one corresponding login area has to be configured (Access to Indexes -> New…)
The storage manager helps to perform storage tasks and build efficient views of the different libraries.
The Archive Plan
The Archive Plan is the core instrument to configure your Archive(s). It connects to the storage pool, archive index and filters (if used). This is also where the user groups who have access are specified. There are options for volume use, scanning, verify and deleting after archiving. The important features for preview generation and metadata import are also part of the archive plan.
The Archive in Color
P5 Archive offers the benefit of customizable thumbnails and proxies. Depending on your point of view and the use of the Archive, this can appear as unnecessary or absolutely vital. Archives of financial, business or scientific data might not need any form of a preview. In media production, video and broadcast, on the other hand, this can be the most important key to the Archive because it is the way that material is identified.
Either way, the decision needs to be taken at the very beginning of the implementation and influences the size of the archive index considerably.
Disk and Tape Scalability and Price
LTO Tape as Archive storage has the advantage of scaling very easily since new empty tapes can simply be added to a library. There are also expandable libraries on offer, making it easy to grow. Speed and simultaneous operations can be achieved by adding drives. P5 supports drive parallelization to almost multiply the throughput. Expanding disk RAID storage is more expensive, as new RAID systems need to be put in place. When reaching certain limits, infrastructure needs to be expanded and that can add tremendous effort and cost.
Here’s a simplified cost/capacity diagram:
The investment in a tape library and archiving software saves more money in the long run than it costs. Archiving (=migrating) from expensive production storage to inexpensive tape actually pays off – as is clearly visible. The more the storage capacity is expanded, the more money can be saved by using tape.
As an added bonus, tape can easily be stored off-site for maximum security. P5 offers tape cloning when at least two identical drives are available.
Specific Scenarios:
Ingest Archive
This specific type of Archive is relevant especially in media production (and in science, when capturing large amounts of data in a very short time span). When capturing in 4K or bigger resolution, the amount of data captured might outgrow the size of the available (high-performance) disk storage. Constantly expanding this disk storage doesn´t make sense – economically and from a workflow perspective. The original files are not needed for editing since this mostly happens with lower resolution proxy files. Only at the very end, at the conform stage, the original files are needed. Therefore, archiving at the very beginning of the workflow saves investment in storage and makes excellent use of LTO tape.
Integrations
There are good reasons to build integrations to P5 Archive. Other software and hardware vendors like to avoid the effort of building archive software themselves. Too complex are the mechanisms and features and too complex is the landscape of storage products to be compatible with. This results in an ever-growing number of 3rd parties offering integration with P5 Archive. In the case of MAM systems, Archive and Restore are triggered from within the MAM interface.
P5 Archive works in the background as the storage backend.
Some examples for MAMs with P5 integration are:
CatDV, CANTEMO Portal, FocalPoint Server, eMAM, MediaLoopster and axle video.
P5 even comes pre-installed on some products of these storage brands: Dynamic Drive Pool – DDP, ScaleLogic, Bright Technologies.
Temporary Archive
There are scenarios where data has to be archived and held for a specified time only. This might be due to compliance regulations, the duration of a project, or the requirements of a client. Often in media or video production, a service provider is only responsible during the time he processes the files. For all those cases, P5 Archive offers a feature to specify retention time on the volumes in a storage pool. After this time limit is reached, volumes will be re-used and overwritten if need be. Alternatively, all volumes can be labeled (=formatted) manually to erase all files on them and re-use them.
Departmental Archive
Even in enterprise environments, there often is a need for a local Archive. It can be a department or a specific category of project or requirements that the enterprise archive cannot or should not fulfill. P5 Archive is the deal Archive software for such cases, as it requires no formal training and can start on a very small scale.
Archive Online and in the Cloud
If collaboration from multiple locations is needed, P5 Archive can be run and used completely in the cloud. Amazon´s EC2 service offers great flexibility and scalability. P5 is available in the AWS Marketplace. Additionally, P5 Archive can use 3rd party cloud storage together with this EC2 instance for even greater flexibility.
Special considerations and scenarios
For legal compliance, it might be necessary to store business and financial data in a form that cannot be modified. If LTO tape hardware is already in place for archiving, this can be easily achieved by using WORM LTO tapes. These are certified and can only be used to write once. While data may be read multiple times, it can not be changed or modified in any way and is certified for this protection. Since the volume of financial and business data is much lower than video and media, using some WORM cartridges every month or quarter is the most economic and easy-to-use solution for this specific requirement.
LTFS export for universal compatibility
In some workflows, large amounts of data have to be transported to partners or clients. LTO tape is better suited for this and more robust than hard drives. LTFS is a good solution for this data transport scenario. Various vendors offer free LTFS drivers that allow to mount LTFS tapes. P5 Archive can mount and discover the tape, read from it, write to it or restore to it. Starting with P5 version 6.1, LTFS is also an option for the archive format as an alternative to the native P5 format.
For smaller amounts of data, RDX by Overland-Tandberg might be an even better and more flexible solution since the readers are inexpensive and can be connected to any Mac or PC.
Migration
Every technology has a timespan where it is particularly useful and makes sense. Disk storage generally has to be replaced or migrated more often than tape storage. Any tape archive that starts out with the current version of LTO (currently LTO-8) can be in use unchanged for 5 to 10 years. The following generation, LTO-9, will be able to read the LTO-8 tapes written today. Additionally, some vendors offer a re-purchase guarantee of 10 years, so identical drives can be acquired many years later. It is wise to build a migration plan and put the necessary steps in the calendar even if they are still several years away.
Additional information on Video Archive is available on our website.
If you have questions on how to implement your Archive, please contact your local reseller or our support team.
