More video is produced every year and so more files need to be held on storage and finally archived.
Let’s first take a closer look at Archive and what makes it different from Backup.
Definition of Backup vs Archive
Backup is a duplication or replication of files used for ongoing production.
Archive is a migration of finalised files and projects to (a long-term) archive storage. Data preservation also means archiving.
To learn more about the best way to Backup video files, see this article:
Before we take a closer look at Archiving, it is extremely important to plan ahead and thoroughly think about available options so that they become as useful as possible.
File naming convention becomes of critical importance because the file names stay unchanged in the archive.
A file name can hold some metadata.
Use Cases to Plan the Archive
A use case descries in detail the role of the person using te archive, what the person knows and is looking for for wat purpose. This is extremely important because it looks into the future and includes the perspective of someone who does not know anything about the current and past productions and still needs to find something specific in the archive. What des the person need to see in the archive, what search queries might the person use? Fo what task does the person need files from the archive. Pondering all those questions and implementing the results in the archive make it more useful and future proof.
Check on all aspects of the workflow that might influence the archive and ask the following questions:
- Who configures, who archives, who uses the archive?
- What files, projects, assets and additional informations, notes, settings, production documentation are archived?
- What naming convention is used for the files and directories?
- What metadata (technical and descriptive) are added?
- Who is in charge of metadata and consistency?
- Setup: what OS, hardware, software, interfaces, network and storage medium?
- Who is in charge of the associated hardware and software?
- Who is in charge of the budget for the archive
- Security: physical access, duplication, offsite storage, storage duration, test cycles, migration?
The File Name and Metadata
The file name is extremely important and can carry already part of the metadata. The file name also helps when a file gets misplaced or handed over to a 3rd party. A robust and smart naming scheme or naming convention avoids mistakes and saves time when working with files.
To learn more about naming conventions and their potential to already keep a part of metadata, see this article:
Best Practice Video Archiving: What to Archive?
In general, it is better to archive more files than less. Years after a project was completed any additional data, production notes, settings, comments, preliminary versions etc. can be helpful to understand what was done and how.
To be able to quickly edit older productions it is useful to “archive” a complete workstation when it is replaced. This way, all previous tools, settings, plugins and formats are available.
Best Video Codec for Archive
There are many option when choosing a video codec. There is not one answer that fits all requirements, setups and companies. The best codec for long-term archive should be long lasting.
It makes sense to reduce the number of codecs used in the workflow.
To make the best use of the assets later there are some requirements for the archive codec and container format:
- The format should be lossless
- The format should be in wide use in the industry.
- A variety of tools should be available for file conversion
Over the years some codecs become obsolete. How to deal with outdated codecs to keep your media playable:
Metadata
When it comes to finding files metadata play an important part. To find a specific file or project years after its completion any description or technical parameter can be helpful. Metadata describe data. There are two categories of metadata, technical and descriptive.
Descriptive metadata can be the people and products that are visible in a scene, people involved in the production, locations, weather conditions etc. Technical metadata is oftentimes created when capturing the data like resolution, type of camera, lens, focal length, codec etc. Both make finding files easier.
Learn more about metadata in this article:
The Best Storage for Media Archives
The available storage media options are disk, LTO tape and cloud. Optical storage has disappeared from the professional landscape as it did not keep up with capacity, performance and price per TB.
Disk storage is easily available but not suited for long-term storage. Also, electric power and cooling costs add considerably to the TCO over the years. Smaller archives or temporary archives that only need to be kept for one or two years might still make use of disk storage.
Archives with longer time frame and larger data sets need a storage medium that better fits these criteria while keeping cost low. This is where LTO tape come into play. LTO tape drives and tape libraries are used in industries like finance, insurance and manufacturing. LTO tape offers a number of unique benefits:
- Lowest price per TB (starting at about 10USD/EUR per TB)
- High performance with up to 400MB/s read/write speed
- Longest shelf life (30 years)
- LTFS as universal data exchange format
- Built-in-in air gap between the media and the network creating maximum security
- Easy offline storage (just take tape sets to another location)
- Worldwide availability and distribution by multiple vendors
- Simple scalability by just adding tape media
To learn more about LTO tape have a look here:
Cloud storage evolved over the years and there are now more vendors and offers than ever.
- No local hardware and maintenance
- No defined storage size (expand and pay as you go)
- Build upload services/devices by several vendors
- No guaranteed upload or download speed
- Possible security concerns
- Monthly cost
The best way to store archive video depends on requirements, budget and existing hardware.
From a financial perspective there might be some consideration to take into account. Purchasing storage hardware, media and software are considered capital expense or CapEx. Monthly fees for cloud storage is considered operating expense or OpEx. In some companies there is a tendency to favour OpEx over CapEx because it can be calculated and budgeted in advance. Investment in physical hardware can pay off and work for many years. In reality it is open how long a setup stays operational without additional repair/replacement cost. Usually, if used reasonably, equipment lasts longer than anticipated.
Cloud storage has the advantage of avoiding any upfront investment in storage hardware and installation. You can start immediately. Also the storage can grow to any size at any time on demand. P5 Archive offers the option of running completely in the cloud. This way you can build a cloud media archive on AWS. The P5 Server runs within Amazons AWS and can access any compatible cloud storage (even from other companies).
Learn more about this option here. Or see the step-by-step instruction:
Archiware P5 on AWS EC2 as central archive server – Quickstart Guide
Video and Media Archiving Software
Media archiving software keeps track of what files are stored where and with what metadata. Archiware P5 Archive additionally offers specific MAM-like features to simplify the process of finding and restoring files:
- Browser interface accessible from anywhere
- Supports disk, tape and cloud storage
- Runs on Mac, Win, Linux, QNAP, Synology, NetGear and FreeNAS/TrueNAS
- Customisable metadata fields and menus
- Combined searching for filenames, and metatadata
- Thumbnails of still images
- Proxy clips of video files
- Scales from single LTO drive to Petabytes and billions of files
- Maximum throughput through tape parallelisation
- Maximum security through tape cloning
Planning and building an Archive is a big task and needs planning. Using P5 Archive as Archive software makes it easier. To learn more there is a 2 part series about building an archive:
See what P5 Archive has to offer as a media archive software and test drive for 30 days for free!
