Many folks in the professional product services community (especially technical product support) who have been investigating the social network model for end-user tech support have just started to see this mote in the eye of Face Book.

I have been advising two clients who are seriously considering a ‘facebook-like’ model to consider a semi ground up approach, or alternatively, to take a look at one of several White Box social network solutions.

There is power here for professional support applications, both interuser and for technical professionals - we just have to get over the notion that Face book is the only platform that can deliver.

The author’s notion that Facebook should provide some services for scaling hosted apps is spot on.

I’ve been casually doing some facebook applications lately, and keeping up to date with the discussion in the facebook developer application discussion board. A lot of these “Error while trying to load page” errors are Facebook’s fault, not the app developers.

Some topics on this

http://cariboobc.facebook.com/.....opic=10794
http://cariboobc.facebook.com/.....opic=11143

Also, although I do love the API, Facebook has a nasty habit of changing things on live without telling the devs. I recently logged in to my apps to find out that “background-color” declarations for a tags generate errors.

You’re right Mark.

I am a facebook developer, and I can say that this isn’t the first time someone blames the apps for problems that Facebook ALSO contributes to.

Many app developers run into mysterious intermittent problems like random redirections or server timeouts, or application key becoming invalid. If you run into an application not loading, just remember, IT MIGHT BE FACEBOOK’S FAULT. Why, just two days ago, I couldn’t run ANY applications on Facebook for quite a while.

I think this article suggests an interesting alternative for Facebook apps, but I DON’T THINK FACEBOOK IS READY FOR THIS YET. There are plenty of issues Facebook should first iron out about its own API support.

Um, many many times this is an issue with Facebook and has nothing to do with the app.

“I’d like to see Facebook itself step in and fill this need”

I am pretty sure they opened the API so they didn’t have to deal with lots of problems like this.

This is really an insult- you need to get your facts straight before making blanket statements like this that are patronizing and insulting. 99 percent of the time, these are issues with Facebook, not developers.

If you want to learn, send me an e-mail. If you don’t care, keep writing articles like these that involve no research, no insight, and no real value.

Completely agree… well put

This is pretty interesting information. I didn’t know that the failure of Facebook apps was so wide spread we are launching our Facebook app soon for http://www.goboz.com “shameless plug” and I am wondering if we may need to paunch it more slowly until we can guarantee it wont fall flat on its face.

#6 - Phil: I’m sorry to have insulted you, but what “blanket” statements are you referring to?

I didn’t mean to blame anyone with this post, neither Facebook nor developers. I realize that scaling websites is not easy but it’s something that must be done consistently if people are going to put up with Facebook apps. I’m just hoping that Facebook and developers can work together to create solutions, maybe by adopting suggestions like the one I proposed.

I try to be careful not to insult anyone; please do the same when you comment in the future.

I think that Beta versions are being exploited, beta should be private. Why? People start using a beta version, there are bugs and errors, another similar product comes out and is completed (99% bug free). People will move to the bug-free program and it will be harder to get them to move back.

Twitter is a good example of a buggy service, however they fortunately get to get away with it because it’s a service that isn’t needed. People are using it for the fun of it, and some are using it for business reasons. But either way, they are not relying on it.

I think the worst beta release I’ve witnessed is Skype Mobile - VOICE DOESN’T WORK. If someone has a better alternative to Skype on the mobile, let me know ASAP. Leave a message on my website (http://www.dallasfreeman.com).

Uh…let’s see- I’ll use some textual evidence here—

“I can’t tell you how many times I have tried to test a Facebook app only to wait impatiently and finally succumb to screens like the one below:”

followed by a screen shot of an app failing, then:

“There seem to be two primary reasons why Facebook apps have a particularly bad time with scalability. ”

Now, I don’t think it necessarily takes a incredible leap of logic from that text sample to assume that you are placing the blame on developers.

The whole article is about developers scaling websites. A cursory glance at the developer board could have told you that the problems are almost always because of instability in the API or on Facebook’s end. You’re using a pretty influential position in the Web 2.0 world to state that Facebook Developers are unable to handle their users- that’s simply not true a lot of the time.

At the same time, I don’t think that users and developers should be unsympathetic to Facebook- the company has the guts to be leaders, not followers, in everything they do in social media. I think they’ve done a great job. So I’m not necessarily transferring your sentiments onto Facebook. I’m just saying that your blame is misplaced and your article is, at best, a rehash of erroneous conventional wisdom.

#11 - Phil: To reiterate, it wasn’t my intention to blame anyone. While I can certainly understand if you yourself are a Facebook developer who takes this post personally, I assure you that I simply meant to point out a problem and seek a solution.

As for the many times I have encountered problems with Facebook apps, I cannot tell you whether the majority of instances were the result of scalability or API instability issues. It has been my sense that scalability issues were to blame for most of the problems. If I am wrong in this regard, then I certainly apologize for not being better informed. I would certainly like to see hard data on this matter, but all the info I have come across so far has been largely anecdotal.

If other Facebook developers are following this thread, please share your experiences to enlighten the conversation.

mark , these are guys - who are so defensive because they need someone - blame the un-success of their app on; Dont worry about them…

- good post and great point …

I’m sorry- but what anecdotes are you hearing? Have you ever checked the anecdotes provided by the developers of the applications that you’re writing about? Believe it or not, there’s actually a source for these stories- the cryptically named “Developers” page- http://www.facebook.com/developers/. Who would have thought?

I can guess what anecdotes you’re referring too. You read some circulated e-mail from iLike’s Partovi about needing servers. Then, you half-remembered a post from Mashable about Where I’ve Been having 250,000 users. Convinced that this qualified as sufficient research, you decided to go ahead and write up an opinionated post on the subject.

“it also seems as though many casual programmers produce Facebook apps that they never seriously intend to scale successfully”

But that’s not blaming anyone, is it?

I really have no idea who is responsible for what. I try to minimize my app usage on Facebook. It does seem reasonable to point the finger of blame on Facebook. Just because I am having other problems with Facebook, I will say this is mostly Facebook’s fault and not the developers.

All apps are pointless if they dont work.

Mark,

I’ve been checking out Facebook a lot lately for potential development. I don’t know for certain whether app availability problems are primarily on the developer side or Facebook side, or some percentage mixture, but I have definitely seen a lot of anxiety directed at Facebook for API issue on the development board. Also, the help and advice developers give each other is great, and I have not seen scalability mentioned as an issue.

http://www.facebook.com/develo......php?id=74

Facebook had 2 HOURS of outage today. Enough said.

I have developed over 10 applications — and NONE of them have had 2 hours of outage because of my lack of “scalability”.

Mark –

Facebook has been having a ton of outages lately, especially in the evenings. Look at the Facebook developers forum posts for details (from developers, not Facebook staff).

Yesterday, the entire site was inaccessible for some time. I was wondering when TC would get around to noticing the stability issues at FB.

Phil - stop being so argumentative. It’s just annoying.

Regardless of why the failures occur, I think Mark has an interesting idea here of Facebook providing some sort of service to developers to host everything related to their app. Of course, I couldn’t imagine that service being free, but that’s expected.

There are some startups working on just this problem.

good post, Facebook is searching for revenue streams, and you hand them one (on a silver platter).

Phil is spot on. Most of the problems are from Facebook itself. It’s a little odd hearing you say you need evidence of that, since you didn’t need one bit of evidence before assuming it was the fault of the developers. No one should be angry about it, but Phil’s correct in his points.

FACEBOOK IS USELESS …..PERIOD. BUNCH OF ADULTS TRYING TO HOOK UP ON LINE AND GET LUCKY AND LAID. I THINK ALL SOCIAL NETWORKS TAKE PEOPLE OUT OF THE REAL LIFE CYCLE. LIVING YOUR LIFE AS AN ANONYMOUS GHOST IS NOT MY IDEA OF LIVING. IS THIS WHAT OUR SOCIETY HAS COME TO. SAD COMMENTARY.

I am the developer of GridView, a Facebook app with ~150,000 users. Facebook provides a tremendous growth opportunity. However it is also a scalability nightmare.

A month ago, I was at 20,000 users and my shared server with 1and1 was grinding to a halt. I switched to (mt)’s (gs). That has grown with me for the last month.

As a newbie programmer (I’m 16) I made some stupid mistakes (like storing a few million files in a single directory), but along the way I am learning from my mistakes. It’s certainly a great experience in scalability.

But Facebook has also been having problems. They show that page pictured above all too often. I strongly believe that apps should be able to set that page to read a more friendly error which doesn’t just point the finger at us. Also Facebook has been dropping photos, not updating profiles, not accessing and rendering FBML on canvas pages.

Now guess who gets blamed for all that. And when you try and contact their support it takes about 36 hours for them to answer.

Facebook needs to beef up it’s applications infrastructure and work out some of the bugs. They are doing a lot to keep users informed, but perhaps a more automated system status and more intuitive end-user error messages are required.

Good to see some conversation between TC writers (Mark) and those that help TC generate revenue (Phil) by visiting TC.

Anyways, I don’t know about charging facebook developers. The relationship is symbiotic. Developers do the work, which brings in Facebook users (and keeps them) while Facebook offers the platform. Facebook developers get a profit if they sell the app, Facebook gets ad revenue. Why make developers pay? They already help Facebook’s bottom line.

interesting idea, mark.

I’m an app developer too, and I would be jumping for joy if my app is so popular that I can no longer scale, and I would be selling it to someone. I don’t see scalability as a “PROBLEM” for developers in that sense.

I have a similar experience to other Facebook developers. Application errors seem to often be caused on the Facebook side of things. Error pages as shown in the post occur as often for me now as they did when I had 100 users, indicating the problem is not with scaling of my server. Facebook also has a tendency of changing the API on VERY short notice, breaking hundreds of established apps.

Having said that, I am very impressed with what they have done, and am happy with my experience as a Facebook developer. The Facebook crew has been improving the API constantly, and things generally work very well. You just have to stay on your toes as a developer.

excellent post I must say. I don’t know whom to blame but the fact remains - large number of facebook apps don’t work. In the fast few months, there has been an explosion of FB related posts (TC had a lot of them). I got interested and signed-up for FB. While the platform is very advanced the much touted apps hardly work. I read about a google app on popular blog sometime back but that has not worked. Likewise there are few others to name.
My point is - Its no body’s fault, specially in this crazy web2.0 world. But there should not be unnecessary hype. In this case FB got that.

Couple of points:
- the error screen you copied is not necessarily related to scalability; it could be many other problems, most likely coming from bugs on the developer side (it happened to us, mea culpa); yes, Facebook was down for a while, but about 2 hours in 4 months, so barely noticeable for users
- “I’d like to see Facebook itself step in and fill this need for developers so they don’t have to scrounge around for extra computational and storage capacity (after all, there is a waiting list for Amazon’s services now)” I TOTALLY DISAGREE. That’s not the business facebook is in. It’s a commodity business, very competitive. Amazon is a great player in it, there are others. It makes no sense for a high added value player like FB to enter that business

I, too, am a developer who has built several Facebook applications for a client. The apllications have been taking off and this last Sunday we thought we had run into a problem of scalability. Our servers were pegged and all the applications were returning the dreaded Facebook error described above. Our usage has continued to grow over the last week but our servers have stabilized. After further analysis I am pretty convinced that Facebook inadvertently launched a Denial of Service attack on our servers.

I can only say the client was enraged, users were frustrated, and the Facebook Platform pointed its finger squarely at our development team.

My point is that based on Facebook’s current demonstration of ability to manage scalability issues, they are the last I would trust to provide a scaling “solution” for our applications. I certainly wouldn’t pay them for it.

Mark — My own experience with Facebook apps mirrors yours. It’s frustrating. Facebook needs to get to it.

People, you can argue all you want about whether it’s Facebook’s fault or developers or whoever, at the end of the day, it impacts users and developers. Whose users? Whose developers? Facebook’s. So, guess whose problem it is.

[Seriously though, can someone get the Chuck Norris app to work please?]

Phil — BTW, I just tried clicking on your Facebook profile, it’s down…

EC2 and S3 are great so long as you’re ‘funded’ or have a deep pocket. But if if I’ve written a fun app to share with a few buddies and it suddenly takes off I might get a 4 or 5-figure charge. Sure, that ‘might’ be offset if I’ve set up an ad-deal. Maybe this means there is an oppy for an an ad network to provide a scaling solution…

“After further analysis I am pretty convinced that Facebook inadvertently launched a Denial of Service attack on our servers.”

er… besides being hard to do inadvertently, dos and ddos attacks are illegal, so whatever the actual problem I doubt that was it.

Nothing about FB really works except that Vista renders the ulgy websites as best they can be.

http://fakesteveballmer.blogspot.com

Well said thoughts about platforms. But i feel platforms have an important role to play for a better feature of the consumer software industry (which is bugger than web). I have posted my thoughts regarding that on http://jatspeak.com/blog/?p=38

@ David Fox (#34) — MyBlogLog was serving nearly a billion profile pics per month out of S3 and I don’t think we had a charge in excess of $3K. AWS is incredibly price effective.

Mark, I agree with you that scalability seems to be the main issue. The practical reality is today is that Facebook developers whose apps catch on need a way to scale. Some are doing it well — such as BoozeMail — and Amazon AWS is an ideal solution for the reasons you list.

From our view at RightScale, it’s not inefficient to work with both AWS and us — quite the opposite. Compared to the alternatives that used to be available, getting a scalable Facebook app deployed in the cloud is relatively painless and inexpensive.

Thanks for covering this issue — we think it’s just the beginning of a trend that will become increasingly common.

It’s not just “Facebook applications are useless if they don’t work” but “most facebook applications are useless”

As a user I’d love to see some more serious applications coming forward, not just those things which suck up my time (yes, I know, a bit more self-discipline wouldn’t hurt!). Something to help with marketing or searching….. ?

S

lol when you have 18 years old in their underpants called “developers”, that’s exactly what’s gonna happen. Unsecured and broken applications which are not even worth a penny are gonna plague the net.

I’ve been developing Facebook applications since July. My background is in corporate development with about 14 years of programming experience. I’ve become addicted to the Facebook Developers’ board, so I’m extremely familiar with what’s going on here.

The majority of outages are not due to scalability issues. Sometimes it’s a Facebook problem and sometimes it’s a developer issue. One of my applications, “My Card” (http://apps.facebook.com/wmmycard), allows you to add a business card to your profile. It’s actually quite useful, but you’re right, it’s useless if it’s unavailable. My Card has been up since the First of August and has only been down when Facebook makes changes on their end. Depending on the week and the number of changes being implemented by Facebook, outages occur. I don’t like it. It makes me look bad, but I’ve been trying to live with the uncertainty of the situation.

Facebook is definitely not ready to take on support for externally developed applications. They have a lot of internal issues to address first. The best solution might be for Facebook to partner with a strong web host. That would allow Facebook to concentrate on its own issues while providing service through a stable third-party.

As a developer of Facebook apps, I can tell you Mark that Phil is correct — most often the issue is not with scaling but rather with Facebook’s often flaky API. In fact, many of the Facebook API features make it much *easier* to scale - access to their crazy memcache setup through tags, for example. The error page that you quote in your post is the bane of Facebook platform developers - it’s a frustating place to work right now as Facebook works the kinks out of their system.

Mark, as a “journalist” it was careless and wrong to make the asumption that you did.

Ben

I am also a Facebook app developer, and I can say that BOTH Mark and Phil are correct. =)

My app is not ready if a thousand+ users pick it up, in fact I would say that it would strain at the 200 mark. That is my fault, because it’s true, although I am a professional developer, Facebook gave me an opportunity to build something rapidly; I succumbed to a “Hey, this is cool!” mentality.

But at the same time, even though I only have about 70 users, and maybe between 10 and 30 active daily, my page continues to fail to load on a regular basis. I have read a number of different methods for working around these issues, but often the workaround discovered by the Facebook Developer community leads to other issues.

With that being said, this is a new platform, it is in its infancy and there are going to be some major growing pains. Perhaps if Facebook had thought to put the word “BETA” in big text across the top of the apps page we wouldn’t even be having this discussion.

At the time of this writing, many Facebook apps are throwing up ugly message boxes when users try to click on links. This happens in any application that uses Facebooks AJAX API. This has been happening for at least the last 3-4 hours and also happened for at least an hour or two last night as well. Facebook is aware of the issue and has made a post in their developer forum letting developers know they are working on the issue. In our office we’ve started to refer to these types of issues as the “Daily Facebook Outage”. It’s not quite every day that something is broken… but it’s close. And it’s never the same thing. In my opinion, the developers who need to learn about building enterprise scale applications are the developers at Facebook. And scalability is only part of the issue. They also have a lot to learn in terms of software testing, code release processes, and just plain communication with both app developers and end users.

This is a major source of pain for lots of Facebook developers. Unfortunately Mark, I think you’ve accidentally stepped into a bit of a hornets nest. When developers have spent that last few months of their lives dealing with Facebook frustrations and then someone comes along and blames the developers for the problem… I guess it’s enough to make some of us snap. I’m not sure this is something you could have known about, so I’m not blaming you. But like I said, I think you’ve stepped into a bit of a hornets nest.

Jerome,

My apologies for not being clear enough. I won’t detail the entire situation, but a problem with Facebook caused users to repeatedly request pages that our servers were returning but Facebook wasn’t displaying. I understand that this is not technically a DOS attack, but the net effect was the same.

I hope this helps, clarify. Keep your shirt on, I wasn’t saying Facebook actually launched a DOS attack.

As others have mentioned, it’s Facebook that has the scalability issues, not the amateur developers hosting on insufficient hardware. Hardware is powerful enough these days that really bad php code can still handle 10,000 users. I see that error message all the time, usually later in the day and on an app I am developing that has 1 or 2 users at most. I also have a decent amount of experience on how to scale applications with hundreds of millions of records.
Facebook has a couple of servers that share the load. When that error message comes up, people usually notice in their logs that they are no longer getting traffic from 1 or more of those Facebook servers/ip addresses. Which is also why it tends to affect applications geographically. So depending on where you server is, you may or may not be affected by a Facebook server issue. Which is why it seems to be an application problem, not a Facebook problem, since it’s not universal.
When you next get that error message, go to the developer discussion board an monitor. You’re likely to suddenly see lots of people complaining about there apps suddenly not working.
I think it’s great what Facebook has done, but there is definitely young cowboy mentality there.

Facebook should provide a CDN solution too.

facebook

I am having issues with facebook…. it will not load anything on my computer… that is load and make it worth my while to see… i have friends sending me aps and messages and i can not even send a reply back it read always… errors while loading… and it just shows ERROR! at the bottom of my screen… what do I do? If i can get an email responce I would be extremely grateful..

Thankyou for your time.

Jessica Chretien