Toodledo, a basic but nifty free online to-do list management tool, is having a very bad day. According to a message posted on its website, a storm hit the city where their datacenter is located, prompting the engineers to switch to power generators.
This caused Toodledo to wind up with a corrupted database, which it is currently frantically trying to restore.
So, here’s the story. A big storm went through the city where our datacenter is located. The datacenter decided to proactively switch to generators. During the switch, something got screwed up, and the power went off for a few minutes. As (bad) luck would have it, this caused our database to get corrupted. We are currently working to bring it back online and restored from the live backup. The crack team at Rackspace is on the job. Thanks Rackspace! Unfortunately, the database is so large, that it will take some time to transfer and verify all the data. Hopefuly not more than a few hours. We know that this is very bad, and we apologize for any inconvience that this will cause. Please check the forums when we are back online for a full report.
Update: Its obviously taking longer than we expected and we are really sorry for that.
Update #2: I know that our downtime is totally unacceptable. I am literally sick to my stomach about this whole thing. We have successfully restored all of the data up until June 10, 4am CST and we are working very hard to restore the remaining 12 hours of updates that happened before the servers crashed. Obviously, this has been a major event. We had multiple simultaneous errors happen that have made recovery slower than anticipated. Because our database is so large (many many many gigabytes), it just takes a long time to transfer and import the data. I know that many of you are upset, and all I can say is that I am deeply sorry. We are already starting to rethink our backup strategy so that recovery doesn’t take so long in the future.
Update #3: Its coming along. I know it is taking forever. We have restored up to June 10, 11:30am CST. At this rate, it should take about *crossing fingers* 2 more hours. We’ll then run some sanity tests and hopefully go live. Again, I can’t tell you how sorry I am about this. The good news is that we shouldn’t have lost any data.
I feel genuinely sorry for Jake Olefsky, the sole independent developer behind the Toodledo service. If this is any comfort to him: an Amazon datacenter got hit by a lightning strike this morning, causing its EC2 service to suffer a fairly major outage.
Or how cloudy weather can have a devastating effect on cloud-based services.
(Thanks for the tip Gavin)
hmm. thangs that you go hmm.
A positive side to it : More people got to know about the nifty service, including me.
That was also my thought. How can someone call ToodleDo basic? It is BY FAR the best and most in-depth system out there.
Yikes…
Basic o_O ?
It is by far the most customizable and powerful GTD manager which exist on the market.
That was also my thought. How can someone call ToodleDo basic? It is BY FAR the best and most in-depth system out there.
Can’t wait for them to come back up!
Someone’s getting free hosting for a while.
Not really. They only cover a % of the bill based on downtime. Welcome to “disaster recovery”. You think your covered, but….Oh Sh#!
there is no such thing as bad publicity. hold on jake
Just 2 days after I upgraded to a paid subscription. Must be my fault, I can crash anything!
I just upgraded too, with the hopes that I wouldn’t loose anything. Unfortunately I had urgent stuff to deal with last night and today that I only have recorded in toodledo. I feel that once you have paid subscriptions you should have some redundancy in place. Like two DC’s in different parts of the country. I feel sorry for the guy as well since his little product has grown up and gotten really big, then some bully came and punched him in the face. I was thinking about switching, but, really, this service is the best there is. And after this I feel he’ll make it even better.
I feel sorry for the poor guy! hope he is able to pull off this one!
Jake’s Toodledo list
#1 [ ] Get Toodledo back up and running
Date Due: ASAP – Priority: High – Context: @Server
#2 [ ] Make sure #1 is done
Good luck, Jake! Toodledo is hands down the best online todo list manager out there, and despite the pitfalls of keeping your data in the cloud, it’s still worth it.
As a business owner and developer of a web-based service (SaaS), I feel Jake’s pain! These things happen (no matter where you host your servers), and I want to applaud Jake for being honest, transparent, and sincerely apologetic about the situation.
I had actually just signed up to use Toodledo yesterday and this won’t cause me to stop using the service. I’m just glad I hadn’t put any data in yet.
Hope things work out. I don’t live and die by my todo list, but I use the site to keep track of things on the horizon and it is a terrific (albeit a bit cluttered) website.
Sorry to hear that. I think toodledo was hosted by Rackspace?
There are a couple of GTD alternatives, http://www.gtdagenda.com , http://www.nozbe.com …
This is pretty bad. I am a user but will now seek a competitor. I paid for their iPhone app, but someone who can have this kind of outage isn’t an entity I can deal with. I would expect with the kind of cash they rake in from app sales and the other apps that synchronize (I am assuming that pay a royalty) they should be a more serious business with server mirroring offsite.
Maybe you missed the part where Rackspace hosts Toodledo. Rackspace promises 100% uptime guarantee. It’s Rackspace who is trying to get their systems back up and online in this particular datacenter. Corruption can’t necessarily be avoided by mirrors. If you corrupt a file on your main system, guess what? The corrupted file gets replicated on your mirror. It doesn’t know the difference and now both your copies are corrupted. That’s when you have to go restore from backups which is a PITA. I will continue to use Toodledo after it’s back up online because it is an awesome application that NO competition I’ve used can equal for me.
Actually, the kind of corruption as described can be avoided by a mirror, but that mirror needs to be in another data center. If you’re accepting money for a cloud app and your cloud app is located in one data center, you’re asking for trouble. It is more expensive, but services like EC2 (3+ data centers) do allow for creation of instances in different locations. Throw in circular replication and you’re good to go. Of course, it is helpful if you architect this before writing the app.
Jake is a great guy and a great internet entrepreneur. Good luck Jake!
… this was caused by global warming, according to Al Gore….
Seriously, how about backing up your data?
You do it over and over again. An then, you do it again!
PS Also, what about a mirror/redundant server, preferably with another hosting company, like this:
MyCompany.com
MyCompany.net — etc.
posting the alternative URL on the main site… just in case.
We did this from the beginning, about 6 years ago, and *never* had a problem…
Neal Martin
MedixNet.info
You clearly have no concept of the difficulties of running a database in multiple locations simultaneously.
Static content is easy to mirror all over the place.
Also, why would you use two domain names? Are you sure you’ve done this before?
Fingers crossed for toodledo. I just started using it in the past 3 months, and I’ve been noticibly more productive.
I’d love to see some information about this Rackspace outage. From traceroutes, it looks like this is served out of a Sungard data center in Dallas. Didn’t Rackspace have a catastrophic outage in 2007 at their Dallas datacenter?
Why are we seeing an apology and explanation from Toodledo.com, but we are not seeing press/apologies from Rackspace and Sungard? Shows that Toodledo.com is customer-driven and Rackspace and Sungard are likely shareholder-driven (incident management anyone?)
Note: Sungard is allegedly the leader in business-continuity (disaster-recovery) services. It is really shocking to hear that Sungard of all places would ever drop power.
I’m guessing Rackspace runs all their boxen on a single power supply, or dual power supplies (stupidly) off the same PDU. I doubt the datacenter would switch both the A-side and B-side power to generator simultaneously!
Best wishes to Jake, best of luck, I enjoy your service immensely.
This is a good time to think about having a desktop version of the app, just like Evernote has. However, that would require more programmer resources than just the one brilliant one currently handling the whole app.
I noticed this am when my iPhone app ToDo failed to sync with ToodleDo. Thought I had messed something up. It’s a fantastic service!
me too.
It sounds like he’s expecting a lot of people to be really pissed. I don’t mind, now i don’t have anything to do!
don’t worry jake, it sounds like there are lots of others like me that are committed to your product.
Yes, thanks Rackspace, for the interruptable uninterruptable power supply.
Tis’ the season for lightening strikes in many areas. Sorry to see this happen to Jake as he’s a good guy with a great service. I’m sure he’ll get through this incident. It’s good to see that his hosting company is being helpful too. It does remind us the limitations that online services present when they go offline.
Hang in there Jake! I use your service and it is by far the best task management system out there. The day you came out with your iPhone app I couldn’t have been happier. Hang in there! You will come out ahead.
May I suggest http://www.vitalist.com for GTD users out there.
This sucks, I hope they can solve the issue fast. This is the first time I heard about them
Jake, it happens. In the SaaS and hosted space, we’ve all been there. After you get service restored, I’ll wager that you will seriously reconsider how you’re using MYSQL (assuming you’re using open source) — Master/Slave replication could have made this restore process a heck of a lot faster than full volume restore… But you have a great product and you’ve done a terrific job providing a good GTD service. Good luck!
Sorry to hear about this Jake. I love the app, hands down better than any other GTD tool I use except OneNote (I can’t get the hang of Evernote).
On the plus side, massive free publicity, and I’m sure a buyer will be in the works to give you the money to provide fault-tolerant contingency.
commenter John (at 8:08am) is going to have a difficult life if he continues to react to things like he just reacted to the ToodleDo downtime.
Commenter Neal must not have read the last sentence of ToodleDo’s second update (included in the article above), though his ‘advice’ is apt.
Like ALL the other commenters so far, I am very happy with ToodleDo and will gladly continue doing business with them: its a great service and their iPhone app is very good.
One hundred percent uptime is a fantasy. The service will be back online eventually and in the meantime all my data is on my iPhone’s ToodleDo app.
Don’t let the experience and naysayers get you down, Jake! You’ll be back and better than ever!
It will be interesting to hear about additions/changes to your backup solution. Other than a ‘duplicate’ service (as Neal pointed out) its hard to imagine how you can improve your existing solution – especially the length of time it takes to restore such a large dataset.
Rock on!
As Director of Software Sales for an Enterprise/SaaS Software Company in Manhattan- unassociated & noncompetitve with Toodledo, I can say that this software is BY FAR THE BEST IN IT’S CLASS.
Maybe Toodledo’s collective users won’t rally behind it through this misfortunate event, but I know I will. It’s a great product, both robust & flexible and I quickly upgraded from basic to a pro subscription and it organizes my life – professionally & personally. I did a great deal of research and “kicked a lot of tires” with competitive vendors and Toodledo wins hands down.
Good luck to them getting back on their feet and I for one, wish them the best.
Shame – makes me realise how much I rely on this brilliant site. But Toodledo iPhone app means that I have an offline backup. Great app!
Toodledo is a great app. I have it on Iphone and have been uninteruppeted. Hope all goes well.
So did this affect Jake’s other websites? Braingle? Is this why that site is down too?
It is back! It is back! It is back!
This is a lesson to everyone who stores data in the cloud.
I’m a long-time Toodledo user, and I’m completely lost without it, so here’s hoping it’s all resolved shortly. Toodledo is my favorite all-time web app.
A tip for users of Toodledo: Once it comes back up, you can save a copy of your data by going here:
http://www.tood...edo.com/xml.php
Actually, I just accessed this URL, and it allowed me to save my data to my desktop. (Not sure if it’s the most recent data.)
Good luck Jake!
Yup, Toodledo is back up. Thanks, Jake!
Off site mirroring? Apparently not. Oops.
try note cards.
Toodledo is by far the best task manager out there. Jake Olefsky is a stud. I’ll be hanging with them for sure.
I just wanted to say that I am completely overwhelmed and teary-eyed over the positive support that I am getting from everyone. I was expecting to be ripped to shreds. For those that want a full update, I have posted it here: http://www.tood...ged-outage.html
That’s because you earn that kind of loyalty Jake – when you treat your customers so well in good times, they cut you a lot of slack in the bad times – something a lot of outfits could stand to learn.
All press is good press.
I tried to sign up just now but the site is not working properly yet. Pages aren’t rendering properly (hovering reveals pop-up functions but they won’t work and the text is mixed up with text on the page itself) and creating to-do items sometimes doesn’t work. I’ll stay tuned as this seems like a useful service.
Jake, I think the sympathy really comes from the fact that, whatever you do, sometimes you just can’t cater for every eventuality.
It’s kinda weird, half my work is helping companies mitigate against risk, and the simplest thing in the book are the servers located in the office and the simplest solution is to drop them into a data-centre because of their resilience/failover etc.
But you can only mitigate so much, and when you have done as much as you can and things still go horribly wrong then the right minded amongst us can’t help but feel sorry for you.
Mike
Jake’s Toodledo list
#1 [X] Get Toodledo back up and running
Date Due: ASAP – Priority: High – Context: @Server
#2 [X] Make sure #1 is done
#3 [ ] Explore Alternate Backup Plans
Toodledo is not basic, it’s very complex, and it’s a great service. A little bit of outage like this should be expected. And it’s a lot nicer them doing it that wrestling with a Microsoft PC that’s gone down, on your own…
Do they not use Amazon for the data centre, so I think that the outage is the same one.