A little over two hours ago, a Google employee posted a note in this Google Groups thread indicating that Google App Engine was “seeing elevated Datastore latency and error-rates, as well as elevated serving error-rates.” He noted that the problem began around 6:30 AM Pacific time and that the team was looking into it. A few minutes later he updated that Google App Engine was going into “unplanned maintenance mode” — over 4 hours later, it’s still not back up.
That’s a long time for any service to be broken, but especially one that is the backbone for many startups’ web apps. What’s worse is that while Google is updating the Google Groups thread, the actual App Engine Status page has been down the entire time as a result of the problems, so people are going there for updates and seeing nothing.
The last update from Google came about an hour ago:
Read-only mode continues. Elevated latency and error-rates persist for Datastore reads. Memcache writes have been reenabled to better soak read-only load. Our engineering teams are looking into the root cause of the problem. Will post more information as soon as it’s available.
Obviously, the natives are getting restless on Twitter. This outage follows the popular hosting service Rackspace experiencing some rare downtime earlier this week.
Update: And it looks like this downtime is even hampering the development of Chrome for the Mac. Here’s was lead developer Mike Pinkerton just tweeted out: “Popup blocking UI is done, but appspot apps are all horked so I can’t get it reviewed. #chrome”
Update 2: 6 hours later, it looks like things are finally back in working order. The latest update from Google in the Google Group thread:
Datastore writes are reenabled and functioning normally! Overall App Engine health is back to normal! We will update this thread if anything else develops, but at this time we anticipate no additional problems. Thank you for your patience.
[thanks Adam and everyone else who sent this in]
Someone needs to lift the hood and fix that engine!
His Latest Post-
http://teentech...ndows-for-free/
Man, I hate to be one of those people, but there are so many spelling/grammar mistakes in this post. You should really re-read your posts to make sure they sound coherent. At first glance, this seems like you wrote the post in 4 minutes, and then posted it without reading it even once. Spell check would’ve grabbed some of the mistakes too.
I normally laugh at the people who post comments like “it’s they’re, not their”, but this comes off very unprofessional for such a powerful voice.
Its not only that; GOOGLE SYNC DOES NOT WORK on my IPHONE, and for many others as well. all CONTACTS are GONE and CALENDER is EMPTY. search twitter and you will see many have the same issues. arrrr…
Even the front page of my app is down. All it does is pull some stuff from the datastore and display it. I do believe it caches, but I’m a bit surprised that they would throw errors on memcache writes. Since there’s no guarantee that stuff in memcache persists, I would think it would fail silently.
They recommend adding a handler for the CapabilityDisabledError, which would be nice if they didn’t also turn off deploying of new versions.
Wow. Not good. Not at all.
What makes this funny is that didn’t a Google Server Engineer guy just basically make fun of how Microsoft sets up their Servers, saying it’s flawed by design. (Can’t find the link anymore)
Yet, both Gmail and now App Engine have been down in recent weeks.
In my own experience I have yet for a MS site, let alone entire system to be down other than for regular maintenance.
I am with you buddy…
I agree with you… google is has down more often then my number of teeths I have…
no wonder all their services are in BETA…!
they get all their revenue from google search…and now with BING gaining market share… i am counting days for goog….
really, MS server don’t crash? Here I was under the impression they did…
You guys go easy on the big G when they drop their billion dollar ball eh?
Would hate to be the poor startup attempting to demo an AE app for an investor at this moment :0
First rackspace, then this? Amazon next?
Please don’t jinx, that would just damn the entire cloud movement if all 3 went hurt at the same time.
“Be your own cloud,” I always say.
Oh by the way, I love the image on this post
True. I’m just setting up a new application – registered a domain name for that app but cannot attach it to my GAE app. Additionally, uploads of new versions have not been working.
The application is here, and it’s being served:
http://rsstfeeder.appspot.com/
It’s a mashup of RSS feeds from top technology blogs with ranking from Retweet, updated every 15 minutes.
As you will see, it’s still very basic, but hey, I can’t update it to a newer version..
Still, Google App Engine rocks, I have several applications running there and its an awesome cloud!
Your app is down.
All my apps are down, and the funny thing is that I used my own app (it polls Techcrunch’s RSS for new posts) to learn that my app is about to crash… and it did!!
Applications are served again, but still can’t upload new versions to the Engine.
I’ve seen this error on my site intermittently since last week. I didn’t know what to do since the error message was unhelpful.
There’s also nowhere to turn to immediately once your site has errors. I wish they would offer paid support.
I hope it doesn’t stay down for a long time.
All my apps just say:
Error: Server Error
The server encountered an error and could not complete your request.
If the problem persists, please report your problem and mention this error message and the query that caused it.
the cloud might be passing
This is why Google has a long way to go to be competitive with enterprise customers. These failures happen often, and when they do, Google is a black hole of non-communication.
No serious business that I know of uses GAE. GAE is for hobbyists and for people testing stuff. You don’t have to spend a dime but if your service takes off, you’re screwed and stuck on it and at Google’s mercy. You can’t just buy your own cheap server and serve it yourself because all the APIs are proprietary.
So you save few days by using GAE and yet you lose flexibility to move anywhere else.
GAE is a giant trap and only a fool would rely on it. It’s like heroin… you get a free hit and then you’re their slave for the rest of your life and can’t quit.
“GAE is for hobbyists and for people testing stuff”
“The Personal Computer is for hobbyists and for people testing stuff…”
Notice that Idiots existed in every era?
What a retarded comparison! Show me one big site that runs on your regular PC!
servers != PCs
The only problem with apps relying on other web services…
A provider goes down, we all go down!
Never trust a young, small and unknown company for your hosting!
Good thing the cloud is the future of computing…
My apps aren’t exactly down, but they are erroring out a high percentage of the time. It’s annoying, because I’m using a personal web proxy with nagios to test web round trips, so I’m getting a half dozen emails every time it fails.
Anyways, what’s interesting in my logs are that the gzip library is timing out due to excessive CPU usage.
I hate to see this happen to Google because historically I have always liked them. They bring out new and interesting and useful services all the time.
But I always say that “stuff happens” and it wasn’t that long ago there were some outages with Google Apps.
Google is going to have to achieve a higher level of operational excellence to earn the confidence of Enterprises.
Ken Godskind
Chief Strategy Officer
AlertSite.com
kgodskind@alertsite.com
Google App Engine is now back up!
app engine has always had problems with their datastore. Read google groups on appengine and you will see many people complaining about problems with their datastore.
AppEngine is a joke. No serious business would rely on a proprietary service like AppEngine.
If you’re a CTO/CIO of a company that’s using AppEngine for production, you should be fired for incompetence.
100% agree. And that goes for all of them not just AppEngine. There needs to be some consolidation and standardization.
Agreed! Standardization and opening of the platform should occur before it should be considered a serious platform. The way they lock you in is like nothing else!
Even if you use MS stack, you can install your own WIn server and host anywhere you want. With AppEngine, you’re COMPLETELY locked in and your code will run only on AppEngine.
The level of lock in is astonishing and anyone running a serious business there is a total fool and I don’t feel ANY sympathy for them.
app engine is back to normal….with write to datastore..
I keep saying it but no one ever listens to me — here is yet again another example of why “the cloud” will never become what it’s being touted to be.
It’s great for your average consumer, but for businesses it’s not only a security nightmare, but as evidenced between all the outages already this year and some of the heavy-hitters, it’s unreliable. And that, ladies and gentlemen, is the name of the game in technology — reliability/quality/uptime — whatever you want to call it.
Are you kidding me? Were you even around in web 1.0 when companies had to build and design their own infrastructures?
Nobody is listening to you because it’s foolish to build a web startup now and not utilize an infrastructure provided by Google, Amazon, or Rackspace. Well-funded companies are building on the cloud, wake up.
Reading over these comments, sounds to me like a lot of IT/server admins are trying to justify their role as their job becomes commoditized by technology.
Brett (Google App Engine) just posted that its all back up. About 6 hours downtime to the minute.
http://groups.g...7596d1d0bd0f0f9
Fixed, for now anyway.
BTW, what ever happened to Facebook Connect having an ‘always skip’ option for the ‘post this to my wall’ dialog? I hate getting asked everytime I comment.
Well, fire me! We’re using it and we love it. It is a great solution for school districts and I truly believe a valuable experience for our students!
Do you work for Microsoft Steve Marlin? Just curious…
My iPhone contacts just came back online. For anyone panicking that all their iPhone contacts are erased, just sit tight and wait until the server comes back up and they will all show up again.
“…hampering the development of Chrome for the Mac”? I didn’t know they still had a significant number of people on that project. What’d they do, pull off the one project engineer or phone-answering secretary?
Maybe they should fire up a few EC2 instances to help with the load?
Google Maps not downloading completly for me either.
David
6 hours is a long time to be down. Hopefully they’ve now fixed the problem. Shouldn’t they have a backup plan and redundancy?
Whats up with them stating 100% uptime in the past 7 days in the graph on their status page?
http://code.goo...tatus/appengine
the amazon cloud went down (in europe) a couple of days ago too which caused a huge headache for our business (and many others no doubt)
http://develope...08&tstart=0
“cloud” doesn’t mean 100% uptime, it just means that someone else fixes the problems which inevitably happen (eventually…)
I don’t know why, but MG Siegler is pretty annoying. I will figure it out eventually.
This reminds me of the debates about web-based CRM five or so years back. “But what if the Internet goes down!” Now most companies don’t worry about it at all. I suspect the cloud will be the same in a few years.
Simplifying things, the biggest argument I see is that putting all your eggs into one basket, a basket held by a 3rd party is a terrible idea. In other words, cloud computing.
What amazes me is that so few consider the fact that failures happen in EVERY system, regardless if it is in the cloud or not. Is AppEngine mature? Not yet, and its growing pains are showing. But does it have a track record of at least securing your data and providing mass redundancy/disaster-recovery? Yes, it does; in fact, it provides a multitude improvement in security & data redundancy over in-house consolidation.
There are drawbacks to cloud-based computing, one such event being that when it goes down, everyone attached to it goes down, rather than just the one organization that has the failures. But there are a great deal of benefits to going this route, such as cost savings, proven tools, constantly maturing technologies to leverage, security (when a vulnerability is discovered, it is patched for all), mass redundancy & disaster recovery, and statistically even average up time, and the list goes on…
If you are too naive to see the future of cloud computing, that’s really too bad for you. Regardless of your perception of cloud computing though, it is here to stay, and its only begun to make an impact.
Absolutely agree.
It’s hilarious to me that people dismiss cloud computing as the last time I checked, every business and consumer was connected to the internet through a 3rd party connection that is typically down far more often than a cloud-hosted website. Multiple times a year I’ve had no internet for much longer than 4-6 hours.
I made this comment above but most of the negative comments come off as IT/server admins trying to justify their role as their job becomes commoditized. But, I’m expecting far to much rational thought from “anonymous” comments on TechCrunch. The internet is plagued by armchair morons.
Update from google…
http://groups.g...237fc7b0aa7df5#