The next-gen web is starting to gather pace, as this week MySpace integrated Google Gears, Yahoo! announced their new BrowserPlus product and Google launched a browser-based edition of their 3D Earth product. Technologies and formats such as AIR, Silverlight, JavaFX, Gears, XUL, Web Applications 1.0 (DOM5, HTML5 etc.) allow developers to accelerate beyond AJAX and towards a new generation of web applications with better performance, more functionality and tighter desktop integration.
Developers and users are now presented with more web technology choice then ever before; “DLL hell” has been superseded by “plug-in hell”, as a variety of companies present their versions of what the next-gen web will look like. But on the web, such choice can come at a cost to both users and developers. More than a decade has passed since the first battle over web formats, back then it was Microsoft, Netscape, Apple, AOL and others laying different foundations in the form of browsers, scripting languages, web servers and more. The legacy of that battle is still being felt today, as Javascript developers rely on whole libraries to assist them in developing cross-browser code and CSS developers depend on a catalog of hacks so that their sites can look consistent across different browsers.
With the new rich web application technologies still in the development phase, there is an opportunity to not repeat the mistakes of the past and instead take a standards-based approach. Thankfully during the course of the previous decade companies such as Microsoft became more receptive to open standards, data portability and cross-platform support. Having broad support for open standards simplifies technology for both users and developers, but it is obvious that not all of the currently announced technologies, such as those listed above, will
During the course of a series of posts here on Techcrunch we will look at various elements that make up the next-gen web and evaluate the options available, current support as well as how well standards are being adopted. In light of the recent announcement from MySpace that they are using Google Gears in their application, in this first post we will evaluate browser-based local storage cache’s.
Browser-based Local Storage
As web applications became more popular there was a general demand for an ability to run web-based applications offline. The first such solutions that could work without requiring a browser plugin or separate application were those that relied on the caching headers within HTTP to store objects within the browsers cache. Javascript libraries such as Dojo implemented support for offline web applications using the same principals, but applications were very limited in scope as there was no easy way to store structured data on the browser (Dojo now also abstracts a variety of other storage engines including Gears - tip: Dylan) .
In May of 2007, Google launched Google Gears, a browser plugin that allows web applications to synchronize data into a local data store and then allow web applications to function offline. At the launch of Gears, Google Reader was adapted to support it, and the emphasis of the pitch for Gears was about offline application access. What was less known is that Gears is a lot more than just offline access, as it provides three primary functions:
- Caching of resources (HTML pages, images etc.)
- Structured data storage in a database
- Asynchronous background worker threads
The part of this we will focus on here is the local object and structured data storage. Gears provides these functions via a Javascript API, which can be accessed by any web application. The structured storage is provided by Sqlite, a popular lightweight RDBMS. With the local database, the developer can not only perform queries and inserts to record new data, but also more complex SQL like joining between tables etc. Although you can have multiple applications using Gears, each app runs in a sanboxed environment with a domain-based security model (similar to cookies and Ajax requests). Sqlite has been built into Firefox since version 2.0, but its API is only accessible from an add-on or a core Firefox component. The Gears plugin bridges that gap and makes it available within the client scripting environment.
Before Gears was launched, the Web Hypertext Application Technology Workgroup (WHATWG) had begun work on its Web Applications 1.0 draft spec, which included structured data storage as part of HTML5. The current draft spec from the working group includes definitions for a Database object for accessing and querying a local data store. The details of the implementation are left up to the vendor, but the API is detailed in the spec. Firefox will be implementing parts of the same storage API from the WHATW spec in version 3.0 of the browser, which is currently available as a preview release. The key components of the WHATWG spec are:
- ApplicationCache - for storing objects in the local browser cache (and checking them)
- navigator.onLine - check if the browser is online or not (and use cache plus local data store if required)
- Storage interface and events - used for storing name and value pairs via the sessionStorage DOM attribute.
- Database interface - used for connecting to the local database. Supports SQL (or a subset thereof, depending on the server used), versioning, error events via callback
- Threading and Callbacks - so that multiple requests can be sent to the local data store asynchronously.
Implementing local storage, caching and offline access are relatively easy. The application can first check to see if these functions are supported, and then setup the local cache by synchronizing the users data in background processes. While a thread is running, either uploading or downloading, you can query it to check on its status to provide the user with feedback (eg. a progress bar). Once the data is local, by running database queries on the local machine developers are able to drastically improve performance. Currently many web applications use the browser as only a presentation layer, for eg. a spreadsheet application may do a round-trip back to the server to work out even elementary calculations such as =1+1. By utilizing the local data store and client-side code, the developer is able to offload processing and storage to the client and provide a much smoother, desktop-like experience at the same time.
Current And Future Support
The issue is that the majority of the WHATW specification was written after Gears was released, so the Database and LocalServer objects used in Gears are not compatible with WHATW - for now. The good news is that Google have come out and fully backed the storage portions of the WHATWG HTML5 spec, so developers with apps running on Firefox 3 with Gears installed will have a choice to use either the native implementation, or the Google implementation. Google go on to say that they will likely offer extra features as an incentive for developers to continue to target Gears over-and-above the HTML5 implementation (features such as desktop shortcuts, etc.).
Other alternatives for local data storage, such as Flash local storage, are completely incompatible with the WHATW specification. The developers at WebKit were very quick to announce that they have started implementing the storage portions of the HTML5 spec also. It is currently available in nightly builds, so in the near future we will see support in both Konquror and Safari. Opera have also announced similar plans and they are actually leading everybody when it comes to implementing HTML5 and Web Forms. Yahoo! BrowserPlus was only announced yesterday, and it is currently unclear wether their local storage support is compatible with the specification as laid out by the working group.
Local storage is a major new feature of the new web API, and developers will not only have consistent support across browsers but will also have the option of Google Gears (which is already available) as well as Yahoo! BrowserPlus (depending on how it works). There is just one browser maker missing in this discussion so far and that is Microsoft. Microsoft have released an early preview of IE8 and announced a raft of new features, a lot of which are based on open standards such as better CSS and Javascript support (with a more standardized object model). The big question is wether we will see consistent local storage support from IE8 following the same spec as the other browser vendors. The IE team have announced that IE8 will support DOM Storage, but that is only part of the overall local storage spec (ie the Storage object described above only).
Overview of Current And Future Support
| Gears |
BrowserPlus |
Firefox | IE8 |
Webkit |
|
| ApplicationCache |
soon |
|
|
|
|
|
detect onLine |
|
|
|
|
|
|
LocalServer |
|
|
|
|
|
|
Storage |
|
|
|
|
|
|
Database |
|
|
|
|
|
|
Threading |
|
|
|
|
|
|
SQL |
|
|
|
|
|
Notes: Once details of BrowserPlus are known the table will be filled in. Gears has pledged to adapt the standard. IE8 has only announced partial support. WebKit have most functionality available in their nightly builds. Both Flash and Silverlight have some form of local storage but not the HTML5 standard API.
It is very rare that a new technology such as local browser storage is so broadly advocated and supported mostly under a single spec. While Microsoft may not have yet announced full support, there is no doubt that they are heading in the right direction. It is also encouraging that implementations from Google Gears and Firefox 3 will follow the working group spec for HTML5. While the new versions of these browsers will not be available broadly for a while longer, Google Gears is available today and since all vendors are targeting the same API, developers can today confidently target and write for the Gears storage API with confidence - something that was almost impossible to do not so long ago.
So with local browser storage and caching, so far the open standard is the winner. The alternate solutions are likely to fall by the wayside or will adapt to implementing the same API.
Essentially, back-to-the-desktop — but in the name of the browser, brilliant!!
Fantastic article Nik. Very well presented information from a developer’s viewpoint. Perhaps you could blog later about the same points from a business strategy view. (Microsoft playing nice with IE standards, but making the Silverlight–which has a low profile over at the EU–Strategic & proprietary even though parts are open and so on)
I also think that HTML 5 might end up with some type of GPS support. I only think it’s natural for GPS to be handled by the browser. Of course, it would have to pull it from the system API but it’s important for developers to be able to make content easily relevant based on location without location being a proprietary. If location becomes easily and equally accessible by all web developers, we will see a level of innovation unlike anything we’ve seen before, and new types of ads that we might actually click on.
Nik, what are your favorite IDE’s, technologies, interpreters, databases & libraries? That would make a grand post too!
Web 3.0 anyone?
Browser + web = OS (sort of)
Desktop 2.0
“Thankfully during the course of the previous decade companies such as Microsoft became more receptive to open standards, data portability and cross-platform support.”
B-B-B-B-B-BULLSHIT!!!
This is the second post today sucking up to microshit… How much did they pay you guys?
Jason: Thanks for the tips, there will be a lot coming up so suggestions welcome.
Aleksey: trying *really* hard to avoid that term
This is funny because I accidentally posted this early, I came back here to preview it and found 6 comments. Table will be fixed in a second.
Looking at your chart, Webkit & Gears look like the two winners here. IE8 has more no’s than a nun on prom night.
Please, please, how about an object-oriented version of JavaScript. I recall hearing that there is a proposal on the table which has not been implemented by anybody.
Nik, To supplement your notes about Dojo, Dojo Offline was updated when Gears was released as a wrapper for Gears, and with a simple API for developers. Dojo also supports storage through the other methods available to developers (HTML5 storage, Flash storage, large cookies, Adobe AIR, etc., depending on what is available in the browser being used). More details are available at http://www.sitepen.com/blog/20.....oolkit-11/ , http://google-opensource.blogs.....orage.html , and http://dojotoolkit.org/offline .
What about data update synchronization. Without a uniform, and easy to use framework our new data storage overlords may fall flat. Just my opinion. I also worry about privacy issues, what happens when a gears enabled browser is installed at an Internet Cafe, and it stores my personal data on the local PC when I visit a gears enabled site.
It is always interesting to see the evolution of new standards and how major tech companies endeavor to control the evolution. It is definitely not a case of survival of the fittest.
google is killing the internet
it is making its own web languages every 15 sec
youwould think they would improve python which they are build on instead of hacking away at other languages
Wow, what a good article…had to check the page again to make sure this was TechCrunch.
I feel the biggest roadblock to these plug-ins and the features they offer is the install requirement. There is a cost associated with getting your users to go through the download/install process.
The more features which become implemented natively by browsers, the better. In the meantime, use of Javascript + Flash (ie. as Dojo can use for offline storage) where applicable makes for a great “zero-install” experience given Flash’s ubiquity.
“companies such as M$ became more receptive to open standards”
HAHAHHAHHAA
They are being left behind and now they are being nice so we let them play with us (me kicks M$ in the face and grabs their marbles)
Embrace, extend, extinguish is their motto, M$ is sooooo lame!
“DLL hell” has been succeeded by “plug-in hell”.
This is terrible, do you need any qualifications at all to write for TechCruch!!?
I’m sure you meant:
“DLL hell” has been superseded by “plug-in hell”
Andy
The article focuses on browsers that store data locally, but Adobe AIR has similar functionality for applications that want to run outside the browser, on the desktop. I’m surprised that wasn’t mentioned in the article, or in the comparison chart. AIR would have had green check marks the entire way down.
Mike Potter
Adobe Platform Marketing Team
Great article but you should check for typos!
(”Team” is singular)
With all these browsers (plus all the mobile browsers) popping up it’s hard to even keep them all separate in my mind!
Thanks for the great article!
Mike: more of a focus on the browsers here in terms of who is implementing WA1 Storage, but ‘next-gen web’ is actually a series of articles and there is a lot more to cover, including AIR.
I am getting to the typo’s now, thanks for spotting them - I accidentally published this post early
It was Microsoft’s decision to mothball IE development for 6 years that retarded web development to the point where it became necessary for large companies interested in web development (Google, Yahoo, Adobe) to start developing proprietary extensions just to get things moving. As for MS’s newfound love of standards, that’s nothing more than lip service to ward off the attentions of the EU whilst they push Silverlight as the RIA framework of choice. Even if IE 8 were released tomorrow with full HTML 5, CSS 3 and SVG support it wouldn’t solve web developer’s problems, because it would take years until a large enough number of IE users upgraded to allow them to drop IE 6 and 7 compatibility. Like it or not, these sorts of browser extensions are the only hope we have to push the web forward.
We’ll have to see about the reliability of this model. My FF is full of add-ons, toolbars and sidebars of all sorts, and it seriously slows it down a lot.
I am not a strong believer of “have your browser take care of it”. there is still too much unorganized information on the Web, that’s the core problem.
it will be interesting to see how these plugins and applications handle user privacy. i know for a fact that Flash will allow *any* website to store up to 100K worth of crap on *your* computer by default, and I think that’s pretty shady, particularly since nobody knows that it’s going on or how to change it.
wEB 2.9?
What the? This is actually really excellent commentary and insight into the market. Nice job guys!
Agree!,This is definitely Web 3.0
Nat
http://www.workersinc.com
is offline support really that important still ?
@Nat & Bill
… no 2.9 or 3.0.. ..there will be a fall back to the dark ages of web 0.0.. its the reboot.. and its coming muahahahahahahaha muhahahaaha
@ Jon - “Even if IE 8 were released tomorrow with full HTML 5, CSS 3 and SVG support it wouldn’t solve web developer’s problems, because it would take years until a large enough number of IE users upgraded to allow them to drop IE 6 and 7 compatibility. ”
- exactly. We’ve got users still using Netscape
I would also like to ask you, guys, the same questions as Dennis did before me. What about data synchronization and security?
If I have the ability to work offline with my data, wouldn’t that create multiple copies/version of my files/data? Would these frameworks handle conflict resolution or will it be left to the user to decide (in a way of annoying pop up windows), which version will I want to keep? Conflict resolution, anybody …?
And finally, what about data security as mentioned in Dennis’ post? I would really like to know your views…..
Thanks
I think the main driver behind this trend is scalability. It’s funny how we went from the isolated desktop to the networked desktop to the fat server and thin client (browser) only to realize that the economics of the web don’t really support such a model. I still laugh every time someone mentions Oracle’s and Sun’s glorified UNIX terminals. Remember those?
Imagine having to implement something as simple as the email sorting function for MySpace, and you need to scale it to 2M concurrent users (1% of installed base). You need an entire data center for that. Enter Google Gears, and the data center is now made up of all these 2M underutilized PCs that do nothing more than render HTML and CSS most of the time.
Centralized models never worked. The future of all development (economy, technology, energy) is in decentralization.
Everyone was tripping balls over cookies in 1998, and now you want to host a full database accessible by web sites on my computer.
Just incase it’s not clear, Safari 3.1 has database storage now. There’s a nice demo at http://webkit.org/misc/DatabaseExample.html.
— Al
What was wrong with the desktop to begin with? After all these years, I still fail to understand this push to cram all this complexity into a huge industry to get applications to conform to the enormously complex technology stack required to hack an interactive experience into the HTTP protocol through a myriad of unstable uncooperative platforms. It must all be driven by money.
Anyway, a couple of comments regarding other comments made here:
1) CD: JavaScript is object-oriented. It just uses a prototypal inheritance scheme, not the classical inheritance scheme that may be more familiar to C++ and Java developers.
2) Andy: The author meant “succeeded” - it means to follow, or come after.
Nik, in addition to your mention of Adobe AIR, you might take a look at its closest competition, Curl and Curl Nitro:
http://www.curl.com
Curl’s been around since 1998, when it was developed at MIT. Since 2002 or so, increasing numbers of large Japanese firms have been using it for creating intranet applications; only in the last year or so has it been extensively marketed at developers in the US and elsewhere. It also uses SQLite, as do some other tools mentioned in your post.
This category of products is coming to be referred to as tools for creating “Rich Internet Applications” (RIAs), and structured desktop storage is one aspect of those tools. Thanks for this post focusing on that topic!
Excellent article Nik. Great writing skills and lots of useful links. Hoping to see more articles like this from you in the future.
Hi Nik,
Excellent start to what should be a very good series of articles. I would encourage you to take a close look at Curl which is a RIA platform that has offered desktop storage for years. Curl is both a browser RIA solution as well as a desktop runtime similar to Adobe AIR only a lot more secure and much, much faster. It’s also the second most mature RIA framework available today - only Java has been around longer.
Anyway, please check out Curl before writing other articles as I think you will find it really is an excellent platform.
All the best,
Richard Monson-Haefel
VP of Developer Relations
Curl, Inc.
(This second comment is about standards.)
My confidence in the ability of commercial vendors to properly support browser standards is pretty low. Just look at Ajax which consists of standards like JavaScript (ECMA), HTML/XHTML (W3C), DOM (W3C), and CSS (W3C). Ajax requires libraries to bridge the incompatibilities of these technologies as implemented among all of the browsers (e.g. IE, Firefox, Safari, Opera, etc.). It’s been several years since these technologies were introduced yet browser compatibility is still poor - better than the DHTML days but not much.
I believe in standards but only if the vendors and open source projects that implement them do so in a consistent manner. Without consistent implementations a standard has limited utility and terrible portability.
Products such as Java (Sun), Flex (Adobe), Silverlight (Microsoft), and Curl (Curl, Inc.) all use plug-ins defined and distributed by a single vendor because that seems to be the only model that works if you want to get a portability across browser vendors. We learned this the hard way with Java applets and DHTML in the mid to late 1990’s. I suspect that Web Applications 1.0, while it may be technically sound, will never be implemented consistently across browsers and won’t be the silver bullet many people think it will be.
I wish I was wrong about the failure of browser vendors to implement standards, but I’m not. They just can’t do it. You have to have a plug-in provided by a single vendor or open source project in order to get consistent behavior across browsers.
All the best,
Richard
VP of Developer Relations
Curl, Inc.
The spec has a section on security:
http://www.whatwg.org/specs/we.....#security7
but it deals more with application security (eg. SQL injection etc.).
In terms of the local security you guys have mentioned, the model is the same as what it is with cookies. In the same way that when you are at a net cafe you wouldn’t use a cookie to remember your logged in session, you also wouldn’t enable Gears (or it wouldn’t be installed) so you would revert back to a non-Geared app.
The database itself is stored within the users operating system profile (eg. on OSX or Vista/XP), so by default only you when you are logged in should have any access to it.
In saying that though, I do expect that with these new applications you will see a re-hash of the same security and privacy concerns that cookies experienced years ago. It might take 2-3 years for most users to be comftable with both web apps integrated with the desktop, and also with placing their files and content up in the cloud.
App developers just need to be wary of these concerns and not ‘force’ Gears as a requirement for using a web app. They will also need to allow users to not use Gears even if it is installed (in the same way as when you login to a site, you can choose ‘remember me’ or not..). I expect that a whole set of best practices in this regard will start to form and the terminology (eg. ‘enable desktop’ or something like that, on login) will begin to form.
If you look at how Google Reader uses Gears, they have some of this. Interestingly, I went back to MySpace and they don’t let you switch it off again- which could be a concern as per what has been said above.
How do most people feel about having something like Gears store their data?
Thanks for the suggestions Aron, Richard et al I wanted to take a look at the browsers and plugins that were implementing WA1 Storage here, but definitely going to be extending into other platforms as well. You could probably tell already that I am putting an emphasis on open standards and if/where/how they are implemented only because I believe its the only way to avoid the mess that took place with the early web
Richard: I also think that standards adoptions has been very good nor quick enough. The whole reason Google built Gears was because they were frustrated with browser development and wanted to leap ahead to HTML5. So thankfuly we might see plugins for browsers to fill the gaps where a standard implementation is missing.
Also at least Firefox, Opera and Webkit seem to be reading from the same song sheet now, which is a solid start.
I will check out Curl shortly - I have heard of it but thanks for the pointer and your comment.
What is amusing is that what this is all about is a move back to client / server computing. Call is “client/server 2.0″. Plan HTML / Web apps have lousy UI characteristics and so we started using AJAX to provide asynchronous client/server communicaitons. Then we realized we needed to run on and offline and added local data stores and local db caching of server data.
Back to the future, with Javascript is the C/S programming language (but terrible to debug with) and the browser as the C/S programming host (as opposed to the Windows class libs or VB Forms stuff).
Firefox always rocks…!!!nothing can match its features…
@P Cause: I think you raise an excellent point. There was a lot to like about client/server architecture. One of the main things not to like was that updating client/server applications was a mess, involving distributing install CDs, platform configuration dependencies, etc. This really added to the total cost of ownership of a client/server app.
The real promise of RIA is to provide the advantages of client/server architecture with added reach and without the ownership costs, because you just update your app on a Web server and everybody is automatically running the new version. So then the next frontier is getting a powerful enough client-side platform to be able to run a client/server-class app. AJAX is looking a bit ragged around the edges for this purpose, and Adobe, Microsoft (Silverlight), and Curl are all competing to be used as this platform.
Note that on the Application server side, BEA has made a version that runs in a VM image as a native OS. They added storage and lightweight services that do everything that BEA needs, without an OS!
Now, on the client, how far are we from eschewing the need for a heavy OS if the most popular browsers can add these common services, while the VM vendors provide bootload and hypervisors and minimal console.
I anticipated this trend a few years ago, and definitely welcome it. The more freedom you have to choose how fat or thin your client is going to be, the more creative you can be when designing your next application.
For another open source effort in this direction, check out:
http://jnext.com
YES! The web browser IS the client operating system for client-server 2.0. Even better when it doesn’t require a plugin. I’m so excited that others are embracing true web clients. Server-side HTML generation is just lipstick on CGI.
http://www.appcelerant.com/lipstick-on-cgi.html
That is impressive news, I didn’t realize Firefox was so on the ball with getting up to speed with local storage. I was at Ajax World a year ago and this was a hot topic. To provide functionality to a user that is off the network, this will be big time for companies that have agents in the field, seamless synchronization is key.
thank you
Firefox always rocks…!!!