Understanding LTFS – What it’s for and how we use it

Before we discuss LTFS, let’s begin by looking at the different physical characteristics of disk versus tape, to understand where the need for LTFS arises.

Disks vs Tapes

Spinning and solid-state disks are random-access storage devices, providing storage in units called blocks. A block of data on a disk can be accessed relatively quickly. There’s no concept of the beginning or end where disks are concerned, one block being as quick to read/write as any other block, give or take a few milliseconds.

Because of its particular physical characteristics, tape is different. Rather than a spinning ‘platter’ over which a read/write head moves, or an array of storage locations within solid-state memory chips, tape stores data along its length, starting at the beginning and adding until the end is reached.

A modern LTO tape, like previous generations of data, video and audio tapes, consists of a reel of magnetic tape in a plastic cartridge. Physically much simpler than spinning disk mechanisms with their motors, bearings and coils, tapes are suitable for many years of archive storage. To read or write a tape, it must be physically wound to locate a specific block’s position along its length. A modern LTO-8 tape is almost one kilometer long, blocks at the end of such a tape are minutes away, not milliseconds. In addition, new blocks of data can only be added to a tape at the end of the existing blocks. Therefore, it is not possible to remove data already written and replace it, but only to mark data as having been deleted.

Despite all of this, the sheer number of blocks available along such a long tape and the increasing density with which the blocks can be written, give tape some unique advantages in terms of capacity, cost and longevity. Because of these different characteristics, disk and tape often coexist within organisations, each being used where appropriate.

File systems make life easier

A file system is a way of presenting computer storage to a user in an intuitive and familiar way. It’s where files and folders are created. Disks are generally ‘formatted’ with a file system, making it possible to create files and folders. Each file maps onto the underlying blocks on storage. Our chosen operating system (Windows, macOS etc) looks after everything for us, and allows us to browse our file system without needing any additional software to be installed.

Historically lacking file systems, tape has required additional software packages to make the blocks of storage it provides available to users. Typically, such software looks after backup and archive of data. For example, backup software will read data to be backed up from a file system on a disk (files and folders) and write this to blocks on a tape. Meanwhile, it keeps track of which file resides on which tape-block, usually by means of a database. The same software is required to recover data from the tape back to disk. Different software packages are not generally designed for interoperability, so despite its capacity and portability, tape was not used for transferring data between locations (prior to LTFS).

______

Sidenote
If we go back around 40 years, booting operating systems from tape, in the absence of disks, was normal. And slow. Watch this for evidence.
______

In theory, adding file systems to tape will remove the need for additional software tools. In practice however – this isn’t necessarily the case.

Introducing LTFS

The ‘Linear Tape File System’, as the name suggests, is an implementation of a file system on LTO tape. It was introduced by the LTO Consortium in 2010, beginning with the LTO-5 generation of drives and tapes. LTFS exists to bridge the gap between disk and tape by providing a file system, on LTO tape. This is intended to provide both ease-of-use and portability benefits. An LTFS tape written on one system, is readable on another.

Thanks to these characteristics, LTFS formatted tapes are ideally suited as a transport medium: They offer large capacity at relatively low cost and with a rugged exterior and have become a quasi-standard distribution format for the media and entertainment industry.

Using LTFS-capable tapes and drives, a tape can be ‘mounted’ as a file system, and interacted with using files and folders, similar to  disk. This is achieved by ‘formatting’ an LTO tape with an LTFS file system, involving writing two separate sections on the tape, called ‘partitions’. The first smaller partition, at the start of the tape, stores the list of filenames/folders and a pointer to the blocks on the second, larger, partition where the files themselves are stored. When an LTFS tape is inserted into a drive, software can read the first partition and quickly display the contents of the tape.

Unlike disks however, additional software is required to make the tape visible, like a disk. Basic software is provided by the tape drive vendors, but due to a lack of sophistication, it’s common to use commercial packages to write and manage LTFS tapes.

Mounting problems

Making a (relatively slow) tape available within an operating system – looking like a fast disk – has a number of drawbacks, including:

• The time required to access files varies considerably, depending upon where they’re positioned on the tape.
• Operating systems (Windows, macOS) may display errors or even hang because of slow responses from what it thinks is a disk but really isn’t.
• Operating systems often read parts of files to display previews/thumbnails to the user, causing unexpected delays while files are read from the tape that the user didn’t anticipate.
• Deleting files does not free up any space on the tape since all new data is appended to the end of the tape.
• Multiple operations to the tape (constant switching between reading and writing at same time) result in very slow operation.

In essence, your LTFS tape looks like a disk to the operating system, but is so much slower, that problems ensue. Anyone expecting a mounted LTFS tape to behave like a very large USB thumb-drive will be disappointed.

Therefore it’s no surprise that users generally resort to LTFS specific software tools to write tapes in a more controlled way, in an attempt to avoid the problems listed above.

Archiware and LTFS

Historically, Archiware P5 Backup and P5 Archive products write LTO tapes using a proprietary format. It has this in common with most similar products that do not use LTFS as their tape format. By having complete control over how data is written to tapes, the P5 products are able to offer the following technically advanced features:

• Cloning: writing simultaneously to two distinct LTO tapes, providing redundant pairs of media for off-site storage. Should tapes become damaged, they can be regenerated from the second tape to recreate the pair.
• Parallelisation: higher read/write speeds by using several tapes at once
• Spanning: large files that do not fit on the remaining space on a tape can be split across two or more tapes. The original file is reconstructed when reading from tape.
• Concurrent tasks: multiple backup/archive tasks can be running simultaneously, utilising multiple tapes and tape drives
• Bare-metal recovery: additional ‘housekeeping’ information stored on tape, allowing complete recovery of the software configuration and databases from a starting point where just the P5-written tapes are available.

These advantages are not easily implemented when using LTFS, if at all. LTFS is a very useful format for delivery of data between locations however, and it makes sense to have the capability within the P5 product to write data in the LTFS format.

LTFS Import/Export with Archiware P5

Archiware began work on integrated LTFS support that overcame the traditional LTFS issues when providing the operating system access to the mounted tape.

It became clear that it’s not necessary to mount the LTFS volume such that it appears within the operating system. By mounting the volume in a ‘sandbox’ environment, where reading and writing is performed within the P5 web-admin interface, access to the tape can be controlled and the common issues above can be entirely avoided.

The LTFS Import/Export feature was added to the P5 Archive module in version 5.6, and provides the following features:

• All the drivers to support LTFS are integrated within the P5 product so no additional software install is required.
• Formatting of LTFS tapes supported, either in single tape drive or tape library across multiple slots.
• P5-native LTO tapes and LTFS tapes can be mixed within a tape library.
• Reading and writing of LTFS tapes performed within web browser – operating system does not see the tape. Files from local or client file systems can be read/written.
• Integration between other P5 modules so that any mounted LTFS volumes can be used as a destination for restore from Backup or Archive volumes – tape/disk or cloud.
• supported initially on macOS and Linux operating systems, with Windows support being added in a future release.

Conclusion

With large capacities and low costs, LTFS has much to offer, especially as a quick and easy medium for transferring large anounts of data between locations. P5’s new browser-based LTFS import/export functions make it a powerful single tool for using LTO tapes in many different ways, including LTFS. By retaining traditional strengths in backup and archive while introducing simple import/export plus interoperability between the backup and archive workflows, we now have a solution that will see our customers well catered for, into the future.