Users beware – if you try out a brand new service in private beta, don’t get too upset when everything goes wrong.
On Monday we wrote about iMindi, a new startup that first showed its stuff at TechCrunch50 in 2008. In the post on Monday we gave out 1,000 private beta invites, which were apparently snatched up quickly.
Then, disaster. The email iMindi sent out, which contains the dreaded phrase “we accidentally deleted all the user accounts” sort of says it all. Brave souls can start all over again here. I know there are at least a few of you out there that are quite willing to forgive and forget.
Here’s the email. Credit to iMindi CEO Adam Lindemann for standing up and taking a beating.
Dear Friends of Imindi,
Yesterday, we were featured on Techcrunch and many of you were kind enough to sign up to the service. Unfortunately, we had not prepared sufficiently for the demand on our servers and then with some human error we accidently deleted all the user accounts. Darn.
We would ask that you forgive us and sign up one more time as members of Imindi. We will set you up with a clean account which we hope you will enjoy using to collect your thoughts and share them with like-minded people.
We are extremely embarrassed by this mistake and we have purchased more capacity and instituted safer backup processes to handle the increased demand to prevent a recurrence of this incident. It’s a private beta, and it will be a while before this service is ready to be launched in public but we hope that you will be kind to Imindi as she grows.
Below is the new invitation URL:
http://imindi.com/invitations/03711dda503b02868903efbed6649d59046952d9
Thank you again,
Adam Lindemann
A second post on TechCrunch… might not be such a bad mistake after all.
Agreed. Also like it when a CEO “mans-up” and says it like it is.
TC praising self for its capacity to bring down new web services??
brutal – it can happen, but then, no backup?
I know it’s early private beta but that’s a good time to test if your backup procedures work.
Just a thought.
You just back up your files with Mozy!!
duh
I was a regular PAYING Mozy user.
Then one day, it just stopped working.
Mozy support couldn’t explain why it broke.
Mozy support couldn’t explain why it couldn’t be reinstalled.
So, moved to Crashplan.
Yes, Agreed Peter. No excuses – just lessons learned.
I tried to use the service the other day and the page never worked. That didnt help either.
reminds me of http://www.tech...elf-shuts-down/
Whats amazing about Couchsurfing is that not only did it get resurrected but ended up thriving bigger than ever before…
Maybe erasing the user DB actually helps a product, gets PR, brings the community together. Myspace should try this, hahaha
FULL-DISCLOSURE – I know the CEO of Imindi personally.
Fortunately the kinks are getting worked out in the beta, as opposed to production system which carries the “beta” moniker well-beyond the actual beta stage. It’s as if the word “beta” has lost it’s meaning, much like when someone says “I love such-and-such object/experience/etc.” We’ve been trained to see everything as a beta when it’s not really, and put ridiculous expectations on those things. I also concur that it Adam has displayed a terrific example of “manning-up”.
OTOH, was the proper testing and planning done before this thing was open to the world? I recall Donald Rumsfeld once saying in a press conference “…you go to war with the Army you have, not the Army you wish you had…”; we all know where that got us. Of course, for anyone to question what transpired with the meltdown is armchair quarterbacking. I just can’t get over the fact that with horror stories about reliance on SaaS gone wrong, like Mag.nol.ia & “I Want Sandy”, I would have thought they’d be significantly more careful. Further, I would jettison whomever is responsible.
I *am* very glad to see this app see the light of day again, especially after all of the hard work put into it after last year’s TC50. I’m also glad to see that people are putting this event into perspective. I’m also glad that it’s turned out to be a learning experience and not a catastrophic outage for the company.
-Nate
Thanks Nate.
I guess the ole’ daily backup doesn’t work so well when you sign up a thousand users then delete them in a single day.
They didn’t hear of daily backups? LOL!!
Daily backups definitely help, but when the accounts get wiped about 12 hours after the tech crunch artictle — dead between backups — there is still a lot of data that is lost, unfortunately
Daily backups? are you guys still living in the 90s? its 2009. Hourly backups and snapshots are dirt simple now, not to mention you can outsource all this stuff. Im guessing they skipped all the unit tests also…
That photo of him is legendary! LOL… Reminds me of Kramer http://academic...OTT2/kramer.jpg
I thought it was more like George! http://www.retr...nails/46525.jpg
awww..he’s changed it now
its way better than “oops we published your soc sec #”
Good to see that startups selected for TechCrunch50 clearly know what they’re doing.
hahaha i needed an afternoon LOL
$@#* happens.
Seriously… how do people let this happen. It’s beyond Amateur.
Awww shucks. I feel their pain. Years ago we launched the first generation of pch.com – they ran a Super Bowl ad and we stood there watching the usage spike then ultimately crash the servers, right there during the Super Bowl! While I was happy that I was not in charge of that particular fiasco, it was painful to see happen, to say the least.
You can plan and you can anticipate. Yet if you don’t have proper contingency plans or perform things like a simple data backup at such a crucial time…
Bandwidth explosion is hard to predict, but simple database backups? PUHLEASE. This is a big red X on this service, no thanks.
And this is why you don’t launch or do heavy promotion during Mercury Retrograde. Should have waited until after May 30th.
I told them to wait for the full moon, or Jupiter’s orbit to elongate or something, but noooo. We’re just a bunch of bloggers, no one listens to us.
lol…Must perform voodoo magic on the evil forces before you go live.
They should’ve prayed to Zeus and Hermes prior to launching and doing heavy promotion. Then all would’ve been fine.
This issue of not being able to withstand serverload is no longer acceptable. If TC is going to write about you after much lobbying or…. you should be ready! Get AWS or pay for more bandwidth than you need for the first few days. As for not backing up, I can accept mistakes happen.
Er, actually, it is the other way around. Version control, backup, nightly build, issue tracking, etc., those are all the processes you need to have in place before you start writing code. Besides, people would rather not be able to access their account for a short period of time due to a spike than log in to find out that their data is gone forever.
I can’t believe you didn’t leave out “deleted” in the title. You lose over 9000 internets.
oh jeez, i bet less than 1000 people had signed up. big whoop.
sometimes you can judge a book by its cover. look at their logo it says a thousand words.
MissingLocator.com – locate yourself
Incompetence.
“Darn.”
I thought sounded pretty flippant. Know that Adam did not intend it that way, but that’s how it sounds to me.
Was going to say something much worse sounding actually – but we edited it to Darn. This was a really stupid mistake. But that said, people have been incredibly generous – the site is seeing a lot of activity and Imindi`s brain is starting to learn and grow. Again, we are really sorry for this and hope that this episode serves as a valuable lesson not just for us but for all web start-ups.
I hope Imindi’s new brain has more horsepower than that of your (hopefully former) sysadmin.
Mistakes do happen… Let’s not become desensitized to the fact that technology was after all, created by humans, and humans do in fact, err. Good thing for owning up to the mistake instead of covering up or using the “beta” status tag as an excuse.
That’s what happens when you let scientists play at software/systems engineering. Oh, and the other thing? I can’t even load their site now.
But I do feel for them. I have nightmares like this.
I always get this image of the new guy saying “what’s this button do?….Oops”
bwbahahaha!!!
I’ve had so many nightmares about this sort of thing, that our procedures are uber tight.
It’s too easy to run a single bad line of SQL and wipe, or corrupt, tons of data.
Thing is, the more your site is used, the more often you should back up. So as soon as you get a Techcrunch hit, and your load has diminished – back up! You just got a boatload of new users, don’t risk loosing them…make your “checkpoint” now.
…knock on wood…
Thanks Dean! Heard loud and clear.
I wish imindi comes back even stronger after this episode. I felt that some of the comments were really harsh on them. Mistakes happen, thankfully it was not as critical as someone’s bank details
Good luck imindi!
Thank you!!!
hmmm…my account appears to be fine this morning with no disruption. didn’t have to re-register at all
This stuff is just nuts, magnolia too…
I’m a one man shop and manage to back up the site, the user accounts, everything, a few times a day. It’s not hard.
It takes a lot of guts to handle such a situation.. most guys can crack up under pressure.. it’s good that you have come back strong and are using this as a learning opportunity. Given that the burnt child dreads the fire, this will only make imindi safer and stronger..
yikes, what a mess. Hopefully, they’ve gotten the major screw-up out of their system and they can move on!
Check out the new interface over at online backup site MyOtherDrive.com. We’ve redone the whole site to make it more appealing to help sell our already solid functionality.
Thanks,
Ken Partlow