The Customer’s Problem
For many years, the media and entertainment customer had been archiving their content to physical tape media for backup and offsite storage. Magnetic tape media has long been thought of as the cheapest backup medium, however the cost of new tape hardware and the need for periodic refreshes meant that there were several associated costs draining funds from the company’s budget. They decided to stop paying all of the maintenance fees on outdated equipment that had been there for decades and instead to embrace the very best of the new by finding a different place for their data. They approached Tape Ark in early 2022 with the intention of moving their historical footage from LTO media to AWS. The collection was approximately 22Pb, spanned over some 3,000 tapes. It moreover harbored almost one billion files.
This was the kind of project that Tape Ark excels at, for there are very few vendors who can work at this scale. Tape Ark’s unique technology stack enables the rapid, secure migration of the world’s largest tape collections from our global operations centers to the platform chosen by the customer. We placed our technology, experience and knowledge in the service of the customer to get this job done.
The Process
The tapes were received at our Mass Ingest Facility in Dallas, Texas. They were then put through a media audit that included photographing each tape, reading RFID chips, and applying a supplier QR code. This created a detailed catalogue of the tapes for quick retrieval if needed. Should the customer want them mid-project, we would know where exactly to find them. This allowed business operations to continue with minimal disruption to their team and their workflow – for them it was essentially business as normal. Once the audit was done the tapes were read to a local cache, then the data was pushed to Amazon Simple Storage Service (S3). This was the processing stage of the migration. The tape catalogue files were extracted using AWS Lambda and interrogated to populate an Amazon Simple Queue Service (SQS) database with extraction instructions based on file size and memory requirements. Four work streams were created to accommodate the large variation in file sizes and to maximize upload efficiency.
At this stage Tape Ark provided ongoing support to resolve anticipated and unforeseen issues related to the original state of the dataset. These included missing and duplicate files, files spanning across different tapes, files with incorrect ID numbers, and files that were just too large to upload (5 TB per file is the platform limit). These problems were more complex than they sound on paper, but the Tape Ark team did an excellent job. They once again showed themselves equal to the task. By developing custom AWS Lambda and Serverless routines, the files were uploaded to the customer’s bucket exactly as the customer desired.
As the content arrived in Amazon S3, checksums were also automatically generated and checked against pre-existing checksums. This allowed Tape Ark to properly verify that the content received was the same as the original. With the customer’s online data library now established, they now had immediate access to all the footage in their collection. This has allowed them to be more dynamic and creative with how they use that content going forward. The collection’s now stored in a cloud-based Media Asset Management platform (MAM), for the editing and distribution of their digital media.
Amazon Web Services provides a highly scalable, durable and secure platform for data storage and archiving, all with lower costs and faster access times than on-premises tape storage. Building on the AWS development platform was straightforward thanks to the well-documented and featureful suite of services and products available. The AWS Professional Services Team engaged in the project provided unlimited support to ensure the migration was completed successfully and in line with AWS best practices.
The Outcome
This tape-to-cloud migration put the customer on the front foot where formerly they had been stuck in the past. It’s a representative example both of the problem we address and the kind of solution we offer. Media and Entertainment and Broadcast customers, perhaps more than customers from any other industry, have so much to gain when they move to the cloud. Starting the conversation today can help your company take the first steps.
Click the ‘contact us’ button to connect with our team.