A great example of how technological collaboration between Tape Ark and dataVedik can liberate large volumes of previously inaccessible exploration data to the public cloud (in this case well log data) and then through the application of dataVedik’s artificial intelligence and machine learning technologies empower O&G organisations to increase operational efficiencies and reduce overall costs.Read More
Astronaut Alan Bean that is and the fourth person to set foot on the Moon. A great story in The West’s Inside Cover where renowned WA scientist and long term Tape Ark associate and friend Professor Brian O’Brien shares his memories and reflects on working with Bean and other NASA astronauts during the 1960’s Apollo missions and his 50+ year friendship with the recently deceased Bean.
Magnetic tape data decay, data degradation, data rot, data deterioration or just plain ageing legacy tape data.
It doesn’t really matter which term you use – it all ultimately can result in data loss.
Do you know what it looks like? Click here to see what tape data loss actually looks like and the difference between a clean and a damaged tape. You can literally see data loss being recorded as the tape is read.
I was in a meeting yesterday and was talking to someone about how seamlessly some oil and gas companies have embraced innovation - particularly in the public cloud, big data and analytics space, while others have been much slower to embrace and are still thinking old school, 20th century methods.
We discussed how hard it was to get certain organisations to think differently and we chatted about how best to change their minds about the new possibilities that the AWS, Microsoft, and Google public clouds have to offer. After a lengthy discussion, we surmised that it was the leadership/management culture in companies that was one of the key issues. The junior team members are familiar with and willing to embrace technologies such as Hadoop, AWS, Lambda, Serverless Compute, AI, Watson and a range of other tools and look at things from a new perspective. Conversely, we concluded that often in some organisations the senior team is comfortable with all of the old tried and tested ways of doing things and simply have no appetite to “take a chance on what is new”. They take a blinkered approach and may be unwilling to listen to or learn about innovative technology at the forefront of their industry. There also seemed to be no correlation between size of company and willingness to embrace new innovations.
Maybe exposure to more innovation will start to change minds, or maybe… just maybe, seeing a problem solved in real time would break a taboos? Let me give you an example. There is a company in Perth that does hackathons called Unearthed – run by Zane Prickett and Justin Strharsky. They set challenges that companies have been facing - sometimes for years, and give them to a mob of keen inventors to try and solve them. I was privy to one of these hackathons for a mining company that was having problems with large rocks jamming their conveyor crushing system. The shutting down of the conveyor system to correct the issue was very expensive - like hundreds of thousands of dollars per hour of down time type of expensive.... and the mining company could not come up with a solution after trying themselves, and was seeking a way to detect the large rocks when they were being carted to the conveyor, so they could be detected before they ended up ON the conveyor. The problem had plagued the mining company for over a year.
A hackathon was presented with the problem. The challenge was set to find a solution within 48 hours (over a weekend). Hackers, engineers, physicists, geologists etc all gathered to hammer out a solution. They broke into teams, ate pizza and drank coke, and on the second day presented their solutions. One team surmised that the tyres on the truck (and their shape when a large rock was in the truck) held the answer, while others thought that they could use video streaming to detect when a large rock was dumped into a truck. The winning solution was one that used an iPhone taped to the side of the truck to detect the presence of a large rock landing in it through its seismic signature (the amount of shaking the truck did when a large object fell into the truck) by using the iPhones accelerometer. It was a very simple yet elegant solution that was low cost, easy to implement and ultimately saved the mining company a lot of money. From that day on, the mining company embraced innovation, and saw the light in terms of thinking outside the square. Thinking, for example, that it does not take only mining company people to solve a mining company problem.
Oil and gas pipelines, like coronary arteries, carry a lot of important fluid and are things you never want to have a blockage in. So why can’t a cardiologist be the one who solves the problems for pipelines? If augmented reality can help you design your home décor by showing you how that new coffee table will look in your lounge room, then why can’t it help you review your gas plants infrastructure? If blockchain can be used to buy Crypto, then surely it can be used to track data assets from a seismic survey. Thinking about other parallel industries and taking a minute to see how they solve problems is all it takes.
I said in the meeting yesterday that these companies are now innovating by attrition. That is that they only innovate as fast as senior team members can retire or give in to innovation itself, and I do mean “Give In” almost reluctantly All I can say – is don’t be that guy.
If you have used a hackathon or similar – please let me know your story.
Many of you have probably not even heard of the “Air Gap” theory, which in my view is a good thing.
Once the public cloud started to eat into the market share of tape sales, or at least threaten to eat into it, tape salesmen and manufacturers started to talk poorly of the public cloud and the ease with which its security can be breached or that viruses could be planted. The view was that this was another way to convince people that tape is safer than cloud. It is part of the reason that I started Tape Ark.Read More
Tape Ark Founder and CEO Guy Holmes is honoured to have been invited to address the Australian Data Management Association (DAMA) Sydney Chapter next Tuesday 8th May and will be presenting a talk entitled “Saving the World’s Second Largest Collection of Data”.Read More
It is impossible to put a value on some types of data. How do you value it? What makes it “invaluable?”
What about when that data is 50 year old exploration data from the Apollo moon landings? During a recent trip to the Chinese Academy of Space Technology Tape Ark CEO and Founder got asked just those questions.Read More
SUCCESS……SUCCESS…….. SUCCESS – we did it!!!
A world first in live seismic data acquisition and processing was achieved – forever changing the way that O&G data can be recorded and opening up so many new opportunities for future DM strategies. The room was packed, the door was open and attendees were lined up listening and watching from outside as we successfully shot seismic data live in Oklahoma, saved it directly to the AWS cloud, duplicated it, QC’ed it and processed it – all in the space of a 30 minute presentation at PPDM yesterday. A HUGE thank you to all the people and our partners involved in making this happen yesterday.
Stay tuned – more to come soon. #NoMoreTape #WorldFirst #SuccessRead More
Did you know that Saturday 31 March 2018 is World Backup Day 2018??
Don’t be an April Fool. Be prepared. Back up your files.
We’ve created more information in the last two years than in the entirety of recorded history. That’s a lot of photos, memories, and important projects to lose. Protect your legacy and backup your data.
In June 2017, my article “My New Digital Classroom” was published in the Australian Society of Exploration Geophysics Magazine entitled “Preview”.
I have found a ton of inspiration in the podcasts that I have been listening to. In fact, I have taken both guidance and inspiration from several of the podcasts that I listen to to help me create my new start-up Tape Ark. I listen to numerous podcasts to keep me up to speed on developments in the oil and gas and technology industries as well as for just entertainment.
If you don’t know what a podcast is here is the Wikipedia definition: ‘A podcast is an episodic series of digital audio files which a user can subscribe to so that new episodes are automatically downloaded via web syndication to the user’s own local computer, mobile application, or portable media player’. The name came from the combination of the words iPod and broadcast.
Since I started listening to podcasts, my taste in content has evolved, and of course fellow podcast listeners (some of which I turned on to podcasts) started to recommend new ones to me.
I thought I would offer an update on my recommendations for people working in oil and gas, technology, and data management.
My previous recommendation plus a few new ones include:
Entrepreneurial and Start Up Scene
"How I Built This" hosted by Guy Raz @guyraz from NPR radio is focussed on innovators and entrepreneurs. This is an absolute favorite of mine. It covers companies like Airbnb, Instagram, and Spankz, how they started, and the drive and determination of the founders.
"Start Up" from Gimlet media is a series about what it’s really like to start a business.
"The Pitch" – Hosted by Josh Muccio @joshmuccio I am a massive fan of this one. This one is also produced by the team at Gimlet Media. If you have ever tried to raise capital, or are thinking about doing it. The show has a panel of guests including Jilian Manus @ManusJillian of Structure Capital @structureVC and Phil Nadell @NadelPhil of Forefront Venture Partners forefrontvp.com This one is a must listen.
Technology, Scalability, and General Awareness of Where Industries are going
The ‘Ted Radio Hour’, which is a choice TED talk mixed with in depth interviews from the speakers who gave the talk put out by NPR. The is also hosted by Guy Raz @guyraz .
The "AWS podcast" (Amazon Web Services), which details all of the new developments in cloud based computing, storage, etc. on the Amazon platform.
"Masters of Scale" – Hosted by Reid Hoffman @reidhoffman (Founder of LinkedIn and Partner at Greylock). Great commentary on how to scale a business. This show rocks.
The Society of Exploration Geophysics has a podcast called “Seismic Soundoff” hosted by Andrew Geary. A series of in depth conversations in applied geophysics.
The “Geology Flannelcast” made by three grad students who purportedly ‘discuss geology topics that no one else dares to touch’. It just so happens that the podcast is one that I don’t dare to touch either. But if you like soil analysis and Milankovitch cycles, this one may be for you.
If you are not in a working mood, try these great entertaining podcasts:
“S Town” - A reporter investigates a supposed murder in Alabama, and open the doors to an incredible story where murder is far from what you need to worry about. Written and Narrated by @BriHReed.
“Here’s The Thing” hosted by Alec Baldwin @AlecBaldwin bring you into the lives of artists, performers and policy makers. Alec is a great interviewer.
“Heavyweight” – A brief window into the lives of common people, with uncommon humor, Jonathan Goldstein @J_Goldstein.
I highly recommend subscribing to a few podcasts and giving them a try. Anyone who has a commute to the office of more than 15 minutes can use that time to learn about a new topic in an entertaining format that takes no effort at all to consume.
Let me know if you pick any of these podcasts up. I would love feedback on your favorites as well.
Photo by Lori M. Sousa on Unsplash
Oil companies are just starting to wake up to the possibilities that public cloud offers. The arguments that the public cloud is not secure or is too costly are now being washed away. The benefits that are now available in the public cloud are massive. Here are 5 things that oil companies should consider as “Must Haves” when choosing their cloud based data management system.
- You must be able to install your own applications within the private cloud offering.
Today, oil and gas companies want not only access to data, but also access to the data with cloud-based applications that are resident within the same cloud - so that data access and processing can be done within the cloud itself. This cuts down data transfer requirements and allows work to be done in-situ. If you need to download your data to your internal network or create tapes to use, process, or share your data – then your private cloud solution is not keeping pace with technology.
- You must have the ability to share data between users
The volumes of data created by oil and gas companies tend to be large. The typical data sharing methodologies of oil and gas industry by creating copies of data on tape or HDD to share with third party companies is slow, costly and terribly inefficient. Collaboration both within oil and gas companies as well as between them, not to mention suppliers is core to the modern distributed cloud. Data sharing should be as simple and cost effective as hitting a share button and include industry leading security. If your cloud does not allow data sharing then it is far too restrictive.
- When you need more resources (Space, Processing Power etc.), then transparent, automated scalability is essential.
The whole premise of the cloud in my view is scalability. If your private cloud solution does not scale or requires advance notice of resource allocation, then your cloud is simply not useful in today’s high performance oil and gas sector. Oil and gas companies tend to be both impatient, while at the same time used to long project turnaround times. Clouds with scalability for processing and data storage are going to cuts week or months off projects if deployed on a cloud that scales with your needs.
- If your private cloud isn’t measured, managed, and billed on the basis of what you use, then your private cloud might as well be swapped for on premises infrastructure.
Use it a lot, then pay for it. If you don’t use it much, then it should cost less. If the private cloud is not SaaS, then it is simply going to be more expensive than the public cloud alternatives.
- Your private cloud must replicate in an automated fashion to another discrete geography or your data is at risk.
Private clouds that require tape backups and couriers to send data to other locations are inefficient, costly, and really are not clouds at all. They are essentially JBOD’s (Just a bunch of disks (or tapes)) – and that puts data at risk. Modern public and private clouds automate replication. If yours does not – then think twice about your cloud solution.
To read about the cloud solution provided by Tape Ark that makes use of the latest in cloud technology, then request our white paper on the Open Petroleum Data Lake and the Amazon Web Services page.
For as long as I have been in the industry there has been a longstanding approach to data management that typically involves importing and storing data into a proprietary system that has punitive contract termination clauses. These costs make it very difficult to move from one system to the next. In many cases, oil and gas companies felt held to ransom if they wanted their data back, which was often stored in an internal format that only the provider could unravel. But if the oil company wanted the data back or wanted to move to a new system, the costs of doing so and getting the data back in a usable format was often prohibitive.
In my view, if you supply a great product or service, and take care of your customers, then you should never fear customers moving. If customers decide to move, then either you are not doing your job well enough, or someone has out innovated you. By always keeping the clients desires and wants in mind, you should be able to stay on that innovation curve and address your client’s needs.
In that vein, we have created an open standard petroleum data lake (ODPL) where oil and gas data can be archived, and you can attach your interface of choice to it (essentially any ARCGIS based data management system). In this way, if you want to change your interface or access tools, you never need to move your data – just change the application. The data sits in Amazon Web Services cloud storage, using an industry standard PPDM data model, and allows clients to attach their chosen application to it without ever moving the data again.
See our web pages in cloud consulting and OPDL for further details. Keen to hear feedback on the concept of a centralised, standardised, and globally accessible data lake. If you are interested in a white paper on the OPDL, click the link to request some details.
This cartoon recently caught my eye from The Daily Drawing http://thedailydrawing.com/.
Only pre-millennials will understand this.
I am willing to put my hand up and am proud to call myself a member of the Gen X squad. As a child of the 80's I most certainly got a few pencils out to rewind my mix tapes. Back then I never would have dreamt that those early first steps in innocent data recovery would eventually see me travel the globe and forge a career in large scale corporate data recovery and data storage. In fact, I performed a project in Ethiopia in the mid 2000’s to recover the countries collection of oil and gas data stored on old 1980’s tapes and found myself rewinding tapes by hand - in a fashion not unlike the cartoon shows.
I’m sure everyone over the age of 35 has a few of these babies around their house with their favourite tunes on them. If I look hard enough I am 100% confident that there is a Top Gun soundtrack on cassette tape lurking in the back of a cupboard somewhere requiring the pencil treatment. Thank goodness for Spotify I say.
For the old timers in the industry, the good the bad and the ugly brings up memories of the 1966 Spaghetti Western starring Clint Eastwood. The movie became known for its unique use of long shot and close-up cinematography, as well its distinctive use of violence, tension and stylistic gunfights (kind of sounds like most analyst reports on the oil price).
In the movie, Clint Eastwood is a gun slinger (“the good” from the title), competing against “the bad” and “the ugly” to find buried gold in New Mexico.
I caught a 10 minute segment of this movie the other morning at 2am while researching technology trends and completing my design of a 10Tb storage platform that can be etched onto the back of a grain of rice. It instantly dawned on me that the movie and its characters were a pretty good allegory for the trends we are seeing (or have seen) in the data storage industry.
As with all commercial technology ideas, there are good ones, bad ones and downright ugly ones. In fact many can become all three given the right amount of time, or some can be all three at the same time. Take the Ford Pinto for instance. It was good in that it was kind of sporty, it was bad because it was not the most reliable car and it was ugly because it had the tendency to explode when involved in a rear end collision.
For the oil industry, and the technology it uses to store its valuable data, I thought I would take a look at the past and present to see which main storage technologies found the buried Confederate gold in New Mexico and which ones were slain in the gun fights of time.
The first cab off the rank is the first commercial storage medium ever produced in the industry - the humble reel to reel tape. This one has become all three over time.
Good - in that it was a revolution that kick started a new industry and opened new avenues for sharing data. Bad - because with hindsight it did not hold much data in comparison to the technology we see today. And ugly - because in many cases the tapes have disintegrated, putting at risk the valuable data recorded on them.
The second wave of storage medium in the industry came in the late 1980’s when it moved to closed cartridge media. The fundamental design of this media type is still in use today and the evolution from 3480 to 3590 to 3592 was seamless and driven by a single manufacturer (IBM). To be fair it is a stretch to find the bad and the ugly in this technology. Even after 25 years, the medium is stable and almost always readable. When push comes to shove, I can always find something bad to say about anything – but on this occasion I am going to follow my mom’s advice - “if you don’t have anything nice to say about something, then don’t say anything at all”.
Running in parallel to developments in tape storage and vying for oil industry dollars was disk storage. This one was good, bad and ugly all at once and over time. It was good in that its capacity and speed quickly overtook tape allowing rapid access to data without the need for a tape drive or additional hardware. But it was also bad in that it became unreliable when put through the sorts of read/write pressures put to it in the oil and gas industry with its large files and continuous use patterns. It took some time, but it has also eventually become downright ugly due to the sheer number of manufacturers of this storage medium trying to create lower cost higher performing devices that now seem designed to fail.
I think it is safe to say that the reel to reel technologies of the 1980’s were killed in action - with their last words before Eastwood took them out being the horrible high pitched scream of a severely deteriorated tape passing over a tape drive head.
Hard disk technology has seen its share of fights as well, but continues to battle on the reliability vs cost battle fields of New Mexico.
It is only closed cartridge technology that in my book has found its way to the chest of Confederate gold – both metaphorically and commercially. Whilst I believe that tape storage in general is doomed to lose ground in the coming years to cloud, it is only so because of competing technologies, and not because it has major faults. Tape Ark is here to help on this front.
I better wrap this up. The garage just called and said my Ford Pinto is ready for collection.
As published in The Australian Society of Exploration Geophysics - Preview Magazine Issue 177.
Last week we restored some legacy tape based data for a major Australian mining company. The data were 8 year old drawings of a mine shaft that were needed urgently by the company due to a HSE issue on the mine site.
The data was written across 8 tapes in NetBackup format and was written from an NDMP server to the tapes. Knowing the urgency of the project, we restored the data in less than 48 hours from receipt of the tapes and hand delivered the data to ensure its safe receipt.
What we delivered was data ACCESS however there is more to the story…
The data RETRIEVAL part of this story is another matter and that was handled by an offsite storage provider. The tapes took 3 weeks to get picked from a local vault/warehouse in Perth and were eventually delivered to the wrong address. Then a second request was placed to get copies of these tapes, also in a local vault which arrived a further week later.
So just 48hrs for Tape Ark to access the data, but 4 long weeks to retrieve the data. Good enough?....we think not.
What I still don’t understand is why corporations are willing to put up with this level of service for their highly valuable corporate data, yet at home they expect data-on-demand services like Netflix. Would 4 weeks data retrieval for your critical corporate data be acceptable when you know there is a better way?
#NoMoreTape #TapeisDead #TapeToCloud #TapeArk
Here is a picture of a great friend (Angelo Felgueiras) that I met 4 years ago. Angelo is standing at the South Pole – wearing the pulk harness I used to get to the North Pole and subsequently passed on to him to use to get to the South Pole. Angelo just completed climbing the 7 tallest peaks on each continent (also known as the 7 summits), and then went to both the North Pole and South Pole to complete what is called the grand slam. He is also the only Portuguese to complete this feat for his country.
I was fortunate enough to go to the North Pole with Angelo and spent considerable time talking to Angelo while we trekked along under our own power through heavy wind and extreme cold. Angelo is the chief pilot at TAP Air Portugal airlines and is an amazing individual. When working towards the North Pole, Angelo had already climbed Mount Everest and the others in the seven summits collection and I very much enjoyed hearing the tales of his adventures.
Angelo helped me understand that anyone can do anything they want if they put their mind to it. He helped me get to the North Pole on his vibe and energy alone. After making it to the Pole, I moved on to climb a few mountains of my own. I owe Angelo a lot for opening my eyes and giving me a real taste for adventure.
Where do I get my inspiration from – the people I meet who have tried great things – failed or not. Sometimes you just need your eyes opened to what is out there – and Angelo sure did that. Here’s to adventure, and thank you Angelo Felgueiras for inspiring me to find a new life in adventure and exploration as well as in business at Tape Ark.
What the release of the new LTO-8 Tape drive really means.
As time has gone on, the life span of new technology (including tape drives) has shrunken.
Technology that one used to buy and use for 10 years, is now only lasting 2 years before being superseded and made redundant by newer, faster and “better” technology. Tape drives and tape media are no different.
For those of you jumping up and down with excitement about the release of the new LTO-8 tape drive (yes – all three of you), don’t get too carried away. While the drive does offer increased capacity and speed, and whatever else the manufacturers want you to believe, it also marks the end of yet another generation of your backup data. Tape drive manufacturers used to make an attempt to have backward read compatibility for at least 2 generations of tape drive, but not so with the LTO8, which only has 1 generation read compatibility.
So – if you have LTO-6, 5, 4, 3, 2 or 1 tapes, add them to your list of dead and inaccessible media if you upgrade to LTO-8 or be sure to factor in the costs of having additional and redundant legacy drives on standby just in case.
Some companies offer data migration services from older LTO media to newer LTO media, but it isn’t free, it’s expensive and it sure as hell was never factored into the TCO calculations they shared with you when you bought your first tape drive.
As a result of the decreasing lifespan of tape technology the volume of “archive” or “legacy” data in storage is increasing at an alarming rate. The time frame between the data being active and accessible on current tape technology to it then becoming vulnerable and possibly unreadable on a superseded technology is shrinking. Archive data used to be data that was 10 years old, it is now 2 years or less in the tape world.
There is a solution to this tape life (death) cycle - the Cloud. Yes – LTO-8 has its place, but if you really want to solve your legacy backup tape and hardware issues, then I suggest you review the material here.
#NoMoreTape #TapeIsDead #TapeZombies