This is just ridiculous. Three year old CouchSurfing, a beloved service used by some 90,000 members, had multiple database crashes, critical parts of the software and data were irretrievably lost, and the backups weren’t performed properly. They are not rebuilding the service. They literally put themselves out of business.
CouchSurfing allowed people to register their home and offer free accomodation to travelers. Creator Casey Fenton, sent the email below to all users of the service (I guess they didn’t lose that data at least).
Dear CouchSurfers,
Two days ago CouchSurfing experienced what could be described as the perfect storm. The database administrators we hired made two critical mistakes. First, we had a major, avoidable hard drive crash. Secondly, the incremental back-ups weren’t executed in the correct manner, and twelve of our most important data files didn’t survive.
I have been working non-stop trying to repair the data, but as difficult as it is for me to say, it has become clear that certain essential pieces are not recoverable. This crash happened at a particularly vulnerable time, in a transition between two back-up methods. If the crash had happened a week ago, or next week, we would have had a different outcome.
It is with a heavy heart that I face the truth of this situation. CouchSurfing as we knew it doesn’t exist anymore. We’ve had an amazing two and a half years.
Members write “CouchSurfing has changed my life” and I know what they mean, it has certainly changed mine and I am eternally grateful.
My vision transformed. CouchSurfing was born out of a dream I had to meet the most interesting people in world and experience their cultures, and it grew into a living, thriving family of almost a hundred thousand. This community has blossomed in beautiful ways I hadn’t even anticipated. It was no longer about what I got to experience, but rather, what genuine, heartfelt good this community can offer the world. We have all opened not only our homes, but also our hearts, our lives. In sharing important moments, deep and meaningful connections have crossed oceans, continents and cultures. I saw in CS, in you, the power to change not only they way we travel, but change the world itself. Thank you, CouchSurfers. You have shown me more than I could have even known. Your generosity and spirit is a gift to humanity.
I have devoted the last three years of my life to CouchSurfing. I have literally poured every cent I have into the site. I’ve sacrificed my health, my time, and my own ability to travel and meet people. In many ways I’ve put my life and wanderlust on hold to build this network. I’m not complaining; it’s been a fantastic ride. As devastating as it is to consider, it looks like the ride is over.
Life is continuously changing, evolving, dying and being reborn. After a fire, the earth is replenished; after a storm, the air is cleared. It feels to me like this loss of CouchSurfing is how it’s meant to be. This crash is like a sign from the universe. Too many random factors aligned to make it as damaging as it is, and though I’ve tried everything I can and engaged the best and brightest database managers, there’s just no way to get it back. In many respects it’s heartbreaking, but at the same time, what we’ve built together is not dead, it lives on in each of us. It lives in the connections we’ve fostered and the culture we’ve created. I want us all to take this CouchSurfing spirit and continue the mission out in the world. We’ve all experienced this common vision and the potential it has to transform the way people relate to each other. Now it is time for all of us to not bury the dream, but rather nurture it’s growth in our own ways, in new explorations and ventures. We all own a piece of the CouchSurfing flame, it’s up to us to keep the fire going and light the world. So let’s do it, let’s light the world! What will you do with your flame?
Goodnight, CouchSurfing. May our flames burn bright.
I love you,
CaseyIf you wish to send your thoughts, encouragement or positive messages, contact us at shunyata@couchsurfing.com
CouchSurfing enters the TechCrunch DeadPool in the most absurd fashion yet. Thanks to David Weekly, creator of another company in the DeadPool (and one doing quite well), for sending this to me.
Wow … this is amazing indeed. At this day and age, I can’t imagine anything like this happening. Would legal action be possible against the ones responsible?
Well-written letter … I can feel Casey’s heartache all the way …
What!? This guy put 3 years of his life into this, but no off-site backups were available? Absolutely no excuse for that. I don’t feel Casey’s heartache at all. He was negligent.
I never used this service, but I would imagine it was a free service that we’ve all come to (rightly or not) expect. Free or not, this sets a bad example for all online services. Trust is vital for the users in this space, and this site just tainted that for its users.
It’s really sad as to what has happened. It was a great community and I had met some great people via it.
That blows, I’d probably resort to physical violence with those database guys :(.
Yeah I’m still shaking my head at this one.
Someone go re-build this.
A great letter that Casey wrote, though. I think he really did… ‘end off’ the life of CouchSurfing with a bang.
I disagree - saying “It feels to me like this loss of CouchSurfing is how it’s meant to be. This crash is like a sign from the universe.” is weak.
Why not rebuild? The tools available today to do this better and faster exist.
I guess he feels that he just figures this is the right time to move on to other things, after spending so much time on CouchSurfing. Not really any other way to explain it, I suppose.
Yeah, you’re right. And it’s only been a few days. He must have gone through a complete emotional rollercoaster by now. We’ll see how it plays out. There’s a need for this type of service. I always have 2-3 entrepreneurs sleeping on my couches while visiting silicon valley, and I enjoy having them around.
I’ve never had that happen to me before, but now that you’ve mentioned this website, I’d be interested in seeing it up and running again
Maybe someone else will come up with a more… ‘web 2.0′-ish solution for this?
Yah, after all they have the users data with them. they can try to built a better system from scratch.
I once saw a service with a similar definition, http://www.hospitalityclub.org/
IMHO it’s to easy to say “He should have off-site backups”.
As I read the message, they had backups, but they were changing their backup processes, and there was a sequence of bad movements of their database guy.
Damage maybe could have been reduced with weekly offsite back-up, yeah, maybe. But s##t happens, and when it strikes there are casualties often.
WTF——
I guess anything can happen when your dealing with data, but I really liked that service. If feels different when you actually use a site, and it implodes. I have more respect for the “Deadpool” now.
This is not the first time a non-existing backup puts a company out of businesss: it happend 2 or 3 years ago to a bookmark synchronisation service called SyncIt. They had quite a number of paying users but lost almost all data, when 4 of their 5 RAID-HDDs failed and no backup was availbale … In the end the made the code open source and another guy offered the same service again. Now they even took over the old website (www.sync2it.com)
Nice bit of eschatological optimism from Casey (comes from Greek “eschatos” meaning “last things” or “ultimate analysis,” and “logos” meaning “science of.”)
And ye, hospitalityclub.org is a thriving alternate service. I read a blog in Romania that is part of the network: Paranoia and other pleasant things.
There is also globalfreeloaders.com and others, too. This sort of web site is an entire genre. And a really good idea, too. But Couchsurfing was undoubtedly the best, most sophisticated. But the virtue of capitalism and the internet is having this sort of redundancy that when one network fails there are other options. hospitalityclub.org is not scaling well with the increased load - they require that all new user signups be approved and there is now a multi-week backlog …
what a shock!
check hospitalityclub.org …
Surely the db admin is responsible legally for this? If I design a bridge with out following proper procedures and it subsequently collapes, then thats criminal negligence. Shouldnt it be the same for a db admin who doesnt follow proper backup procedures?
i had been using the service for over a year, and had made and maintained many friendships using it. What is not being told here is that many links may have been lost, and frindships with them. This really is a tragedy.
Yes hospitalityclub is similar, but couchsurfing was unique. RIP.
p.s. there will be a lot more told about this. There were too many people, too involved to simply accept the letter. Had been a collective working on the site in Canada, as mentioned almost a hundred thousand dedicated users, ideas of a couchsurfing island, (probably stupid) rumors of a hospitality club sabotage, recent changes to the site encouraging donation and discouraging dating, etc… watch this space.
would also like to point out that the above post by ann, reading ‘what a shock, check hospitalityclub.org’ is being swiftly posted throughout the blogosphere. If this is an HC attempt to cash in asap would be very dissapointing.
I had an account there…. didn’t use it much, but I could definitely see it’s potential. Sucks for those active in the community. Ridiculous.
The interesting part to me was the comment, “the incremental back-ups weren’t executed in the correct manner.”
It sounds like perhaps they didn’t bother to test the “restore” part of the “backup and restore” process; if they had they might have found and corrected the incremental backup problem before the “perfect storm” hit.
Yesterday,
All those backups seemed a waste of pay.
Now my database has gone away.
Oh I believe in yesterday.
Suddenly,
There’s not half the files there used to be,
And there’s a milestone hanging over me
The system crashed so suddenly.
I pushed something wrong
What it was I could not say.
Now all my data’s gone and I long for yesterday-ay-ay-ay.
Yesterday,
Need for backup seemed so far away.
Seemed my data were all here to stay,
Now I believe in yesterday.
Um…has anybody considered the fact that a large percentage of Couchsurfing users (including myself) have supported the site financially? There was a “verification” system that was meant to keep the site safe for users and prevent perverts and predators from using the site incorrectly. However, one of the final steps in the verification process required a financial donation in order to get fully verified.
Will this cause a wave of people demanding their money back? What about people who gave a large amount of money? WHat about people who donated money the week/day/hour before the site crashed? I imagine this will be something that Couchsurfing will have to deal with.
That’s such a shame! I think I’d have been pretty interested in something like this; too bad I didn’t hear about it until it entered the Deadpool.
Incrementals are all fine and dandy, but if you’re not doing weekly off-site full backups (and testing them from time to time) you’re just asking for trouble. My company has picked up and relocated for a disaster drill and recovered the entire data center from just a briefcase full of tapes. The incrementals are nice to get you back to the day you lost the center, but even if they didn’t work you’re still only looking at a few days lost data.
So sure, the admins did something bad but losing the whole enchilada is the fault of the owner.
Two types of people in the world — those who have solid backups, and those who will.
All of you scolding the bad backup administrators should probably take a day and test your backups to see if they’re doing what you think they should.
Most companies I know do take backups, but never test doing a restore. This should be a good lesson for other companies.
What’s that sound? It’s a million databases being backed up simultaneously.
Wow, great way to cop out of managing a website.
Yell “critical unrecoverable errors OMG” and run away. If these people really intended to keep their site up and running, they would’ve maintained it better. Maybe it wasn’t making enough money, maybe it was too tedious to manage. Who knows.
It sounds like Casey is saying, “Yeah our systems have failed… but we enjoyed them and stuff. Oh well, bye!” Why not sell the site to someone who cares more about it instead of scuttling?
Yeah, I agree with Casey A. P. These people clearly didn’t want to run the site nor did they want anyone else to either. You can’t blame this mess on the new database guy, unless he intentionally erased the previous backup tapes and then dropped the database.
The only way this scenario could play out is if the old tapes were never validated and turned out to be corrupt, in which case at least 50% of the responsibility goes to the previous person whom was responsible for backups. Perhaps this new DB guy just assumed everything was working properly (as he/she was probably told). I hate to hear these copouts and see people blamed for things beyond their control.
Old DB Admin: “Whoops, I just deleted the couchsurfing database. Be a pal and hand me that backup tape over there, would you?”
New DB Admin: “What, the one under the bulk erase magnet?”
Old DB Admin: “Yeah, that’s the one.”
PR Person: “The new DB Admin apparently screwed up our system so if you want to be mad, be mad at him. blah, blah, blah…”
Am I taking crazy pills here. This is the most ridiculous letter I have ever read. “..its up to us to keep the fire going and light the world. So let’s do it, let’s light the world! What will you do with your flame?” This is the lamest thing that I have ever read. Everyone step back from your little tech worlds and look at how pathetic this is.
Thank you Michael Ballack…if he really believes his own drivel then why doesn’t he start it up again and use the publicity of this event to make it bigger and better…any good entrepreneur would. Yeah it’s a pain in the butt to re-write code, but if the service is worth a damn then Casey (or some angel) would be more than willing to spend the money to do it…we all know how cheap it is to do these days (he doesn’t even have to do it himself.)
At least if he said “I was bored of the project anyway and wanted to move on” I could understand. But this junk about everything aligning against him? Defeatist hippie.
He must have been tired of the site to give it up just like that.
It’s his life. He wants to move on. Maybe it is time for him to move on to other things in his life. He’s just being optimistic about what has happened, its not the end of his life when this site dies, its just the end of a stage of his life and he has come to terms with that.
If he wants to move on, let him. Why don’t YOU rebuild the service then? Instead of complaining how he won’t. Thats his choice, and if it benefits him but screws you over, then so be it.
He should put the code up so someone would have an easier go of it, then transfer the domain to the one with initiative.
Darwinism at its finest. Good riddance, idiots. The IT gene pool needs a good flushing from time to time. The sound you hear shouldn’t be a million databases being backed up, it should be the hum of a million tape drives doing a restoration test.
Many founders of technology startups I have worked with had zero knowledge of technology (when they self-funded the venture with Daddy’s money), or just a thin veneer of knowledge (enough to talk their way into the money). I’ve worked at several startups where money is spent on salaries but not on tools…usually backup equipment and the time to test the backups themselves are not provided by the decision makers, against the recommendations of the technology people.
Casey sounds like a punk by starting off the letter blaming others. The leader is always responsible. He got what he deserved, and now he wants scapegoats. I am interested in knowing the story from the DBAs point of view, because founders of technology companies excel at telling bullshit stories…Casey’s letter reeks. Teams succeed or fail together as teams. Who would want to work with Casey again, or fund him again, after reading his woe-is-me, it’s-their-fault, I-give-up pathetic letter? I hope he takes this opportunity to go back to his previous career field, because any competent professional in the IT field would not have let this happen.
What is an “avoidable hard drive crash”? Oh, I like the transition from “the DBA’s screwed me” to “I, Superman, worked nonstop to repair the mess”. Where was Superman when he was needed to review processes and procedures to protect the data before trouble occurred? Stupid punk.
It’s funny how everyone is so shocked this happened…a small company loses a database and the company must fold. Wake up people, this happens a lot more than you read about: http://www.itbusinessedge.com/item/?ci=10808
RIP, Casey. Take this opportunity to find a new career field. Please.
before everyone goes too far… whatever screw up casey has made, he has also changed the travel industry indellibly, and has certainly changed the way i travel….and i’m a travel writer. Read this to get some idea:
http://travel.guardian.co.uk/a.....40,00.html
something smells, but lets not forget that something amazing came before all this, thanks to casey.
Some 90,000 members, that is sure a huge database. With that enormous size, I’m quite sure the business can thrive on…. but I just wonder why it didn’t?
Sorry to see a good service die, but the reasoning is pure BS.
unfortunate that they didn’t back up properly. CIS 101
http://www.bookscholarship.com
Couchsurfing is amazing!
No one has mentioned that the site WAS a non-profit organization, so Casey & co. were doing this on their own time. It was ran solely on donations of members and out of their own pocket. I actually have two Couchsurfers from France arriving tonight. This wasn’t Casey’s source of income.
As for the admins that helped create the issue, they may or may not have been volunteers. I would assume the latter since the site wasn’t created to turn a profit. As for the naysayers, you could help out with the resurrection since there is a group trying to fix things. I don’t really care how this happened but it is a total shame since I met some really cool people through the service and planned to use it for future travels.
From an email I received ….
what we need for the first steps is:
Super professional PHP and JAVASCRIPT Coders
Mega professional UNIX/Linux Administrators
Hyper professional Apache Webmasters
Incredible professional MySQL Superstars with Database Cluster Experience
please send to: contribution.couchsurfing AT liwest.at
If there truly is a business here, the cost to rebuild will not be high. The IP that the CouchSurfing team has accumulated over the last three years should allow them to be back stronger than ever in about 18 months.
Can’t the site be restored from an older backup? Given, you and the users will lose some data and perhaps changes to database procedure code, but it’s better than tossing it in the can. I think most people would understand and move on with the understanding that you would have no choice…
Theres no doubt that this is idiocy. He not only can’t do a restore from a recent (daily? weekly?) backup but doesn’t even have a month-old off site restore or anything to restore from?
This conversation thread is a marvelous example of “Internetitis”… namely, a group of people with nearly no information has formed strong opinions of exactly what should have been done, and then pointing fingers at the people in charge because they weren’t willing to continue spending all their time and money keeping a non-profit project up and running.
In my book, Casey was hit with something drastic, did the cost/benefit analysis for himself, and realized it wasn’t smart for him to continue. Should he have spent the time and money to get the site back up and running? Should he have rebuilt the site from scratch? Only Casey can answer that, and really he already has.
Michael, you say “Why not rebuild? The tools available today to do this better and faster exist.” Faster and better doesn’t equate to free and easy. Casey has put in three years and clearly would have to start over. Perhaps he simply doesn’t have the interest (as I think is clear from his letter), or the money (again, pretty clear), or perhaps he feels like there are already other sites out there that he simply couldn’t catch up with. At the end of the day, Casey made a call and informed us openly and honestly about that call. Seems pretty dang reasonable to me.
Raise your hand if you’ve never been involved in some sort of internet meltdown project…..
…. those of you with your hands up - welcome to internet development. You’re new career will be a lot of fun.
Fenton should have been writing an apology, not a eulogy.
Couchsurfing belonged to more than just Fenton. He’s slinking out of a responsibility that he himself admits has benefitted him for years. Why isn’t he asking for at least help to come up with a Plan B, a way to rebuild and pass the torch if he’s tired of it? He doesn’t have to stay on if he’s burned out, but at least pass the ridiculous fire he’s talking about on to the next person who is willing to carry it for a while? 90,000 members and he can’t find a single person to help him at least come up with a plan B?
Non-profit doesn’t mean it’s strictly put together by free-time. It is incorporated, and he solicited funds to help build it. How much? Probably not a lot, but then he should have moved to a membership model.
Couchsurfing is a non-profit (not was), and therefore the public owns it… not Fenton. He must account for all of the money that went into it to the
federal tax authority, and at the same time he must account for all activities as being within the guidelines of maintaining a non-profit status. That’s for the government and Fenton to handle together.
My concern is that he incorporated a non-profit, and provided that the non-profit laws in Canada are not much different from here, he has some explaining to do.
What a shame.
I LOVE YOU TOO, CASEY!!!!