Tape Ark Transforms Historical News Footage with Artificial Intelligence
Industry: Media & Entertainment
Tape Ark recently completed a project to liberate footage from a collection of broadcast video tapes (Betacam format) from deteriorating media that had been held in offsite storage for over 40 years.
Each ageing video tape contained approximately 100 historical news clips from a major Australian television network broadcaster. Unfortunately they were accompanied by a very limited and minimal metadata catalogue, making it very difficult to locate relevant clips when the TV network needed historical footage.
Once liberated from tape and accessible in public cloud storage, artificial intelligence software was applied to the data to analyse the content and extract a rich catalogue and index of the video footage. This extensive metadata not only made the clips easily searchable and useful for the TV broadcaster internally but also valuable in that they can now monetise the footage and create a revenue stream from distributing it.
Due to the undigitised video footage residing on legacy tape and limited accompanying metadata, the TV broadcaster were unable to easily or quickly view the historical footage or search and edit it in order to access and use the content when required.
Each video tape needed to be physically watched to ascertain more detail on its content – a manual, time consuming and expensive process.
The Tape Ark Solution
Using Tape Ark’s scalable and automated mass tape liberation infrastructure and depth of technical expertise in legacy data migration, footage from the video collection was carefully extracted from the Betacam tapes to a local disk cache and then converted to a broadcast quality HD format.
The digitised video was then uploaded into the public cloud to a secure Amazon Web Services (AWS) S3 storage account. The historical film footage was finally available in the cloud for download and use by the TV broadcaster.
Now accessible and liberated, Tape Ark applied AWS Rekognition software to the footage, automatically splitting the video into individual clips in high resolution broadcast quality HD. As an additional step, smaller low resolution proxy clips were also created for internal use by the television network to provide rapid access to the clips for viewing and editing in the local studio.
These newly created clips were also transferred to reside in AWS S3 cloud storage, where Tape Ark could then apply then next level of AI and analytics to the video footage. Using Amazon’s Rekognition suite of software tools to analyse the video and image content, Tape Ark were then able to perform:
Object, Activity & Scene Detection - identifying thousands of objects such as vehicles, pets, species of plants, fashion, furniture and scenes including known locations.
Facial & Celebrity Recognition - identifying celebrities and creating an index of detected faces.
Facial Analysis – locating faces within images and analysing facial attributes such as whether or not the face is smiling, they are wearing glasses, have a beard or the eyes are closed.The facial analysis output also included the coordinates in each frame of each person’s left and right eye, the corners of their mouth and arrangement of other detailed
Sentiment Analysis – detecting emotions such as happy, sad, surprised or angry.
Text in Image –locating and extracting of text within images, including text in natural scenes such as road signs, car registration plates, t-shirts, and captions or text within news tickers in the news frames.
Speech to Text - conversion of all spoken word to text and time coding the text to the frames. So when searching for a particular person saying a particular word or phrase, the search would position the user immediately at the frame in the video where the words were spoken.
The data generated through the application of AI software enabled the creation of a rich, searchable index and catalogue on the historical video footage which could then be imported into the network’s Media Asset Management (MAM) system. The AI created a huge uplift in the depth and knowledge about the clips in the library.
The television broadcaster can now search the metadata of their video to search for footage of a celebrity or politician appearing in an historical clip, at a particular location or point in time and include other search topics/criteria that may be relevant.
In today’s current digital world where social media and being first to press is so critical, this search capability can make a massive difference in terms of monetising the video collection they possess. A historical clip of a celebrity or sportsperson saying a particular phrase can be incredibly valuable fodder for use on social media platforms like Twitter – but only if it is found quickly and easily.
For the television network, or any other businesses that has the opportunity to monetise historical video content, the process to generate this rich data set using innovative new technology can be well worth the effort and minimal spend.
Interestingly, very soon after Tape Ark completed this work, a major radio network approached the TV broadcaster to buy the sound-bytes of the news clips alone for use on their radio shows. This immediately turned the TV networks accumulated historical collection of video and audio into a revenue generator and income stream, rather than purely a cost centre.
Perhaps one of the best things about this project for the client, was that for each full Betacam of video tape (which equates to about 60Gb of data in the cloud), the cost to store each of the average 100 clips on the tape was .002 of a cent per month per clip. Cheaper than offsite storage, cheaper than office real estate prices to have them in-house, and no need for old video reading devices to access the content. The time to access clips changed from hours to just a few seconds, which should result in a significant over-all cost reduction of news room production.
A final thought but equally important takeaway from this video project is that it isn’t always about the value being added to the data after it is liberated. Much of what our client learned is just how vulnerable these historical collections are. The data needs to be liberated first and the number of working tape drives to read the tapes is reducing, the tape media is deteriorating, and in many cases, the tapes are the single and only source of irreplaceable content. So if that tape deteriorates beyond recovery, then the historical video footage may be lost forever – BEFORE it has a chance to be enriched with AI and emerging technology and put to greater use.
Click here for more information on:
Example of Metadata Enrichment through AI Application
On one of the Betacam tapes owned by the TV network was a clip entitled “Australian Prime Minister’s Christmas Message”. Looking at the database, the clip had no date, and it was not known which prime minister it was delivering this message. The only way for the broadcaster to find out more would be to watch the clip (in fact all of the clips) to create a more detailed database.
Using AWS AI software, including Rekognition to analyse the video footage, Tape Ark were able to produce frame by frame .json outputs detailing the faces it could recognise, a total face count, whether they were male or female faces, the expression on the face (happy, sad, confused etc.). It also output a list of each object in the frame, and where possible, the subtype of object (like tree, and species of tree).
After we completed the work, the TV network was able to now search on the same video containing the prime minister, but using far more advanced search tools and criteria. They could search for content to target video that was very specific to their needs. Things like:
“I want to find a video of Australian Prime Minster Tony Abbott, standing next to his wife Margie, mentioning the word Christmas, where he is wearing a blue suit, and his wife looks happy, standing next to a “Abies Alba” species of Christmas tree
Other Applications for this kind of technology include:
Security footage direct ingest and analysis
CCTV footage from drilling rigs, police stations, airports, public venues etc.
Underwater ROV footage of pipelines, drilling platform footings, vessel inspections etc.
Historical archives – both private and public collections of significance