The big story today is about Microsoft subsidiary Danger losing all T-Mobile Sidekick customer data from their servers. Danger is the company noted for the T-Mobile Sidekick, the revolution in cloud mobile, and most memorably, almost everybody living in 90210 having to get new phone numbers because of Paris Hilton. Valued T-Mobile Sidekick customers received a notice today from the company updating them on the “data disruption” problem. The good news is that data is no longer being disrupted. The bad news is that there is no data left to be disrupted.
This latest large-scale publicized data loss will surely lead to managers everywhere forwarding a link to the story to their IT departments asking “what are we doing so that this doesn’t happen to us.” It will lead to the issue of data loss and backups being written about ad naseum by technology pundits. Research companies will rub their hands together as they prepare new 80 page whitepapers with titles such as “How Companies Who Pay Us Money Can Prevent Your Data Being Lost” (complete with FDA “may cause drowsiness” warning label on the cover). Consultants will flock to their customers, pat them on the head, and reassure them that everything is ok because their project specification powerpoint shows that they included two of everything (and charged for it).
Backups are a hard sell. Most of us don’t want to think about things going wrong (or put more colloquially, shit hitting the fan). Spending your Saturday afternoon staring at a progress meter that seems to be moving backwards is the polar opposite of fun. If there was a brainwave study of people in the process of backing up data, it would probably show no activity at all (but they could use the results to help calibrate the machines). Furthering the point of no interest, Google trends shows that while the volume of news stories about backups and data loss is increasing over time, volume from people searching about it is proportionately decreasing. We are only shaken out of this slumber briefly when there is an incident such as the one at Danger this week.
Like the death of a celebrity from a drug overdose, publicized data loss incidents remind us that we should probably do something about taking better care of our data. But we usually don’t, because we quickly remind ourselves that backups are boring as hell, and that it’s shark week on Discovery. Our previously well thought out backup and recovery plans are expunged as we scan the perimeter of the clinic for the shortest fence to jump over and bolt back to freedom.
Those who are organized and backup their data usually discover the later, larger, part of the problem – restoring from a backup: Where did I put the backup? It’s an old copy. That file I was just working on isn’t there. It was never actually backing up. No software I use can read this stupid fucking format, etc. For most of us, by backing up, we are only setting ourselves up for a bigger failure down the road.
If you read almost any technology website or newspaper, you could be forgiven for thinking that “The Cloud” solves everything. When “The Cloud” is proposed as a solution to any problem most nod in agreement, not wanting to appear out of the loop by asking what the hell it even means. It certainly isn’t a solution to backups – as Sidekick users found out today, and ironically, as 7,500 users of online backup provider Carbonite found out after the company lost their backups (Carbonite can take some comfort in that they now rank very well for ‘data loss’ in search engines because of the incident. What do they say about bad publicity?).
In the Danger case, it appears from initial speculation that the data was lost because they attempted to upgrade a storage array without backing it up first. Here is a case of smart and rational people who do this for a living at one of the best companies in the world, and they didn’t even bother making a backup – so what hope do we have? Relying on the cloud as a backup didn’t work, because somebody forgot to backup the backup. Roman poet Juvenal foreshadowed this very problem when he wrote “Quis custodiet ipsos custodes?” (at least I think he did, hard to tell because there was no word for “backup” back then).
Storage technology does a reasonable job of keeping data intact, considering that it is only a spilt Red Bull away from not functioning at all. The methods used to store data are vulnerable to simple things such as a magnet, and we live on one of those (hint: The Earth). We have become far too reliant on something that is inherently unreliable.
Every systems administrator has at some point in their life experienced the sickening feeling of realizing that they have lost data – and do not have a backup. It is so common that Eminem even wrote a song about it (Lose Yourself, about a sysadmin who when realizing he didn’t have a backup decides it is time for another career (replace ‘music’ with ‘man tar‘ in the lyrics for the full effect)). The sick feeling that all sysadmins have felt after losing data is because of the pressure and responsibility of the situation, sysadmins run the technology, and we expect technology to solve this problem.
The solution may be to do nothing, certainly not to panic. The biggest problem is that we hoard data. We produce more data and information than we ever have, and we are all vain enough to believe that the data we create is so fantastic that it should live on for eternity. Losing the contact list on your phone shouldn’t be a problem – you should know who your friends are anyway. If you are losing sleep because you can’t find an old email you wrote, you likely have deeper issues to address.
Technology has spoiled us to the point where we feel nostalgic when we lose data that didn’t really matter in the first place. If it did matter, a primal instinct would have driven us to do more to preserve, rather than rely on a sleep deprived sysadmin on the other side of the country. If you didn’t care enough to take care of it yourself, then you didn’t really need it. It is our misguided expectation of technology that causes us to panic when we lose data. The only people who have a larger incentive to preserve your data are those who are using it to target an advertisement at you, or sell you something.
Not only is a lot of this data not important, but do we really want to keep it? I certainly would not want a full account of everything I did in my youth sitting on a server somewhere. I am also certain that we do not want the record of our as a society time being documented and discovered by future civilizations based on Twitter messages.
Data experiences its own form of natural selection. What is important will survive, the remainder will thankfully fade away.
This incident has occured at right time so now people all around will be taking a look at how they backup all the data and how to secure it..
This has definitely come at bad time for Microsoft who are going to release Windows 7 soon..
“Data experiences its own form of natural selection. What is important will survive, the remainder will thankfully fade away.”
Sounds nice, but it’s nonsense.
We’ve only begun to create a network that feels reliable enough to start getting to the next level of utility. We need things to stabilize so that losing data becomes a rare thing.
Nik, you’re a professional at information technology. That attitude would be like a doctor admitting that smoking is a good thing because it weeds out the bad people. We set higher goals. What happened at Danger is not a good thing, and to rationalize it as somehow inevitable and even desirable is pretty lame. Imho.
Dave, recently I was reading the post on your blog from Sep 11 2001, day of the WTC attacks , and found it very interesting from a historical perspective. You made the effort to preserve that page, and others, and they are still here.
My argument is that if the data was not valuable enough for somebody, least not the author, to make an effort to preserve then it is because it wasn’t worth preserving in the first place. If I wanted my grandchildren to read my tweets, I would keep a copy somewhere – heck, I would probably just print them out.
I don’t think we can expect commercial companies to keep all of our data for us for eternity – regardless of how good the technology is. If I have a backup offsite and stop paying the bill, the company providing that service will eventually destroy that data. It may have meant a lot to me, but it is nothing more than an invoice to them.
Preserve what you want to keep yourself, which you would probably do if it was important to you anyway (like your posts). The rest will die, but don’t panic, its dead because nobody cared.
Nik, you are mixing arguments. Sidekick users were paying T-Mobile to store their information. That was the promise from Danger long before the Cloud became part of our vernacular. You can’t blame them for relying on this service being provided, especially when Microsoft can’t provide a reliable sync from the Sidekick to Outlook.
I agree with you about losing data. When I switched from a Sidekick to a G1, I went through and purged old contacts. It was cathartic.
We all create our own permanent records.
That’s Danger’ous.. if there’s a thing of what’s in a name… ;-(
Excellent writeup Nik.
I really wish people were more aware about the pain of data being lost, and the relief that comes when you realize that my data is backed up, someplace, someway.
Its just a week back that I reinstalled my Windows OS and thanks to Mozy, I got my documents and desktop files back just as they were. Saved a lot of time and frustration!
With so many options like Mozy and Dropbox, people should make a small effort initially to set it up, once that’s done, these programs do stuff automatically anyway!
Are you sure about Mozy? Could you tell us more about your restore experience with Mozy?
How much data did you restore? Were you a paying customer? and so on …
Really, how did you get Mozy to work? I have tried it three different times over the space of a year, on two different computers, and every time before it gets past 1GB of backup, it slows down so much that it would take a decade to backup all my data. I talk to their support, they tell me to cancel the backup and start it over, I do, and it does the same thing again.
Ultimately, I ended up going with Carbonite, despite their incident a year ago, And was quite happy when I had an HDD failure, and Carbonite restored flawlessly.
“It will lead to the issue of data loss and backups being written about ad naseum by technology pundits.”
And much of it, crap.
A root cause analysis would be appreciated by many readers. Sadly, these root cause analysis stories are rarely seen — sure, notable exceptions — and we see the minutia repeat in future preventable failures through ITIL or similar methodology adherence.
Vintage!
“Research companies will rub their hands together as they prepare new 80 page whitepapers with titles such as “How Companies Who Pay Us Money Can Prevent Your Data Being Lost” (complete with FDA “may cause drowsiness” warning label on the cover). Consultants will flock to their customers, pat them on the head, and reassure them that everything is ok because their project specification powerpoint shows that they included two of everything (and charged for it).”
Oh how sweet consultancy is!
Microsoft has been pretty good over the years with their shit phoning home what the hell happen.
Number one case of what to be worried about with cloud support just happened; what if the server or servers go with all your shit on them.
All sidekick users should get either
1 months free service (everything on there plan)
If they ever want a sidekick customer again then send Danger the bill.
2 The option to cancel with no fee
(let there last memory of TMobile be good atleast)
there should never be any device that is solely based on cloud support it should always be an option to have your info stored locally if desired no matter what.
the Sidekick brand is destroyed plain and simple nothing they can do besides some how pulling a copy of every single piece of into lost out their ass somehow is going to save them.
Things like this make me want to take my company’s entire operations team out for a beer, because of the work they do covering our ass on things like this.
Doing backups and managing data retention policy is a thankless and boring job, but it really does need to be done.
Do it.
To my mind your idea: “Data experiences its own form of natural selection” is a very interesting point. Especially if you combine it with the Darwin style observation “survival of the fittest” was survival of the most adaptable, not survival of the strongest, fastest, best, most important or most efficient or most intelligent. So the data that will survive is that is the most adaptable formats and copied the most.
I just had a deep conversation about this last night! (the part about creating more data than we really need)… very interesting, it’s an issue worth talking about, because it has environmental repercussions as well, as all these additional hard drives end up as additional e-waste that can hardly be broken down…
Mozy and other services are good at backing up those recently edited documents during your computers idle time – https://mozy.com/?ref=XCY3EB
Yes, that’s a referral code too.
Also I dug up paris hilton’s original sidekick NSFW Warning:
http://web.arch...://66.40.38.42/
“Quis custodiet ipsos custodes?”
+10
Classic mistake, someone linked ln -s or referenced the data a href, you killed that data and so created else’s rotten reference. Then best rule is to avoid corruption, never delete, store everything just undisplay what should.
Personally I think the most amazing thing about this fiasco is that they *can’t* get the data back.
Even if an accidental link or rm “deleted” all the pointers to the file, no sane filesystem deletes the data on-disk by default (plus it would take ages), and it should be recoverable via some forensic analysis.
Even if it was a SAN migration, I would expect the old SAN hardware to still contain the old data.
The fact that they don’t have any external, offsite tape or disk backup is ridiculous but believable. The fact that they managed to delete what’s probably a few terabytes of data in a way they say they can’t recover is completely unbelievable and stunning. It really makes very little sense.
This article is quite accurate when it comes to personal data created by an individual, managed by an individual – I often let my old data get purged when I lose a disk or decide to format. On the other hand, I feel it’s very inaccurate for business data and data created for us by others – we horde it not due to its “importance” or any sort of vanity, but due to its irreplaceable nature.
I think we should preserve all data no matter what. I think there is a lot of good stuff lost in the cracks because we are geared toward a specific view. Imagine if we had the knowledge of how the pyramids were built. It would unload a whole boat load of new ideas that may not have otherwise been thought of. Just my opinion anyway.
Honestly I thought this article was way off base. I should be able to remember my friend’s numbers – really? I for one don’t want to go back to the days of the StarTAC and I certainly don’t see why I should be forced to remember an arbitrary system of numbers when I already know my friend’s names.
The problem with this disaster is that the data on your phone – contacts – is probably the most important data you have. Making light of this loss, or talking about natural selection, is just silly.
Nicely put. Data loss is a bad thing at the best of times but when you’re actually paying for a service to store your data… Thumbs down for this article.
In your contact list, some numbers are more important than others. Some are people you call regularly, some you rarely do. It’d make sense for people to make sure they have other copies (written somewhere, electronically on a local computer/in e-mail) of those particular numbers stored for any event (this data loss situation, a lost/damaged phone). So in that respect, I agree with what he’s saying.
well said, pierre. I have 640 cards in my address book app (that syncs with google contacts and exchange) but I doubt my top 20 called people budges at all month in/month out.
I do admit that I don’t remember even those 20 numbers,
I’d suggest that everyone just move over to Soocial.com’s service. Simple, secure — and backed up — and, not dependent on a network operator, or handset provider.
The first thing I said was like “was that a typo?” and then I googled it. Sounds like a good idea. Good luck!
Hey. Thanks for the heads-up. I’d never heard of the service. Just set it up on my N95 and it copied all 350 contacts in no time. Thanks again
Apparently, Microsoft was planning on releasing Windows Azure which stores data on MS server. Wonder what effect this will have on all.. I read a blog on techgiraffe about the same..
http://www.tech...-sidekick-data/
What is important will survive, the remainder will thankfully fade away.
Maybe, but I sure wish NASA hadn’t overwritten the tape with the hi-res video of the first moon landing.
It seems like every month there is another example of this kind of cloud failure. People assume that because its in the cloud its safe. What would happen if you lost all of your email? It would be disasterous for me.
I created BackupMyMail and BackupMyTweets with this in mind and I’ve been very surprised to find how few people think its worth taking the time to backup their email. Yet lots of people are eager to backup their Twitter account! How strange!
http://backupmymail.com
http://backupmytweets.com
I trust the cloud – just not Microsoft clouds… just as I don’t trust using XP or Vista. Not really that amazing of a concept – since they’ve been screwing this stuff up so royally lately.
Actually I use MS’s Live Mesh service and I have my important data backed up across four geographically distinct machines, and in their cloud.
The sad thing is if they used their own technology they’d be fine. I suspect that Danger came in with a poor implementation. At that point all it takes is losing a key person or two — guaranteed docs and design plans are dated or just flat out wrong. Engineering is trying to figure out, how exactly does backup work. Rather than reverse engineering everything, they decide to move to a system they understand. And in the process they hit a tripwire.
MS’s big mistake was buying an obfuscated service. Unlike buying a product (e.g., Virtual PC), they can’t stop selling the product, and go through and rewrite the service and apply their own test bar. They have to make incremental changes to update the service. They should have learned from Hotmail that this sort of thing is a PITA.
I believe that this is more about cost and consumer’s willingness to pay for a service that guarantees no data loss.
Technology exists to deal with efficient and dependable realtime data replication and archival of rarely accessed data. Paying for people and processes that assure that these types of failures don’t happen also comes at an additional cost.
The challenge is that the average consumer does not recognize the value of paying for those higher level services until they suffer a loss.
In today’s economy, people are concerned about cost. There is an endless stream of service offerings that are willing to cut costs to provide a lesser service. Consumers will continue to be reminded of this experience since most are willing to forgo the added costs of reliable service to save money.
You get what you pay for.
Pretty much the dumbest article I’ve read on Techcrunch in quite a while.
The examples are crap (backing up the Danger-cloud wouldn’t have been a backup of a backup, since Sidekick *only* stores in the cloud, which is btw simply idiotic), and the central argument completely ignores the fact that most of the time, we only discover something is valuable (which itself very relative and ever changing) well after it has “gone”.
Expecting even the collective wisdom of the people living today to make the right decisions on what is worth preserve is at the same time ludicrously naive and incredibly arrogant.
It’s hard to imagine, in this day an age, any network operation NOT doing at least a daily backup to a non-volatile medium (e.g. tape). Sure, you might lose a few hours of data, but not EVERYTHING.
Metal Gear Solid 2, anyone?
Rose: We’ve always kept records of our lives. Through words, pictures, symbols… from tablets to books… But not all the information was inherited by later generations.
Colonel: A small percentage of the whole was selected and processed, then passed on. Not unlike genes, really.
Rose: That’s what history is, Jack.
Colonel: But in the current, digitized world,
trivial information is accumulating every second, preserved in all its triteness. Never fading, always accessible.
Rose: Rumors about petty issues, misinterpretations, slander…
Colonel: All this junk data preserved in an unfiltered state, growing at an alarming rate.
Rose: It will only slow down social progress, reduce the rate of evolution… Not even natural selection can take place here. The world is being engulfed in “truth.”
Colonel: And this is the way the world ends. Not with a bang, but a whimper.
Rose: We’re trying to stop that from happening.
Colonel: It’s our responsibility as rulers. Just as in genetics, unnecessary information and memory must be filtered out to stimulate the evolution of the species.
I like the spin but it is mean. At Open-Xchange we thought about the impact of social and cloud services a lot and built our product around it. There is a nice presentation an blog here: http://www.open...com/en/node/916
What hasn’t been addressed is not just why did Danger not have a backup but why did the failure effect so many accounts. Segregation and partitioning of data is just as important as backing up. With clear partitioning rules any kind of failure is limited in its scope and impact.
I’m fairly new to Techcrunch, but are most of the comments on all posts blog spam or product promotions?
This piece reminds me of the film Strange Days when Angela Basset says that memory is meant to be forgotten, so that you can move on.
I don’t think she meant phone numbers of friends that you wanna get in touch with regularly. A certain level of paranoia is needed for people to backup effectively. Even though the whole point of cloud is that you don’t have to do your own backup, I would still recommend backing up your contacts to a PC or even print the whole contact list.
For people reading this & still haven’t backed up your contacts, you should do that now. There are many free options to choose from.