Google launched Gears last May, and for the first year of its release it was considered a minor, niche product that a few developers and users may take advantage of to allow offline access to web applications. You can probably recall the arguments at the time: who needs offline access, connectivity is everywhere anyway, not enough apps will support this etc. It wasn’t until a year later and only a few weeks ago, that Google revealed its ace card: Gears-powered messaging for MySpace that is super-accelerated. Google had entered the race to provide the new web API, and for a year almost nobody had noticed.

The browser of the future is likely to become the virtual machine that hosts almost all applications. In this scenario the operating system becomes transparent, so Microsoft has something to protect (the source of its profits), as does Adobe, who currently provides the most common and consistent web virtual machine with Flash. Google has made no secret of its plans to target and harm Microsoft, and they know that the best way to go about that is to make the operating system irrelevant by moving up a layer and turning the browser into a standard, but powerful, virtual machine for applications.

It is hard to convey in a review how Gears can change and accelerate the functions of a web application. With browser-based Javascript, functions in MySpace such as listing and sorting emails or filtering through a list of friends felt very slow; the loading bar would freeze as the hourglass spins while your browser makes multiple requests back to the server. With a quick install of gears, a click on a confirmation box and a couple of seconds of loading time, the same functions that would previously almost drive a user insane now feel like they are part of the browser itself. What Google showed us Gears could do with the MySpace integration woke almost everybody up to the true intention of the product: this was no longer about offline browsing, but a shot aimed directly at Adobe and Microsoft.
Read More

Last week we looked at how some browsers and plug-ins were adopting storage-related API’s that are a part of the new HTML5 draft specification. While Gears, Opera and Webkit have implemented structured storage API’s, the remainder of the HTML5 spec currently remains mostly unimplemented and also in a state of flux. HTML5 is a super-sized effort to bring all the browsers under a single, standard markup language and set of API’s - but with Microsoft, Adobe and others racing ahead with their own next-gen web technologies, will we ever see a real HTML5 standard?

Learning From History

In terms of the scope and effort, the HTML5 effort has an earlier historical analogy in the HTML 3.0 spec. Back in April of 1995, the HTML 3.0 spec was drafted as a backwards-compatible way of adding new features (such as tables) to HTML 2.0. The W3C had only just formed, and HTML 3.0 was one of the first specs to be produced by the new working group. At the time the browser wars were just around the corner, as Navigator had been out for only five months and had already built up 80% market share. Microsoft had taken notice and were rushing out Internet Explorer 1.0 which would be released a few short months later.

As it remains today, in 1995 the different browsers all supported a different set of markup. With their new 1.1 release, Netscape had raced ahead and implemented tables, floating images, and other navigational elements (such as visited links). IE 1 was a complete hack of a browser that had an approach of rendering at all cost, meaning that if it couldn’t work out what the user had intended with the HTML, it would do its best to have a guess and present anything. This resulted in issues such as being able to mix tags (eg. <b><p>Header</b></p>) which allowed developers to be lazier as IE would compensate for mistakes.

With the market share of Internet Explorer steadily rising, and with frequent point releases and updates from both Netscape and Microsoft, the two browsers steadily diverged further as the market was also segmented into two firm camps. The HTML specification effort, which had previously taken the form of RFC’s, was supposed to re-unite the browsers and formalize new features that browsers had already introduced. There was often significant tension amongst contributors to the spec about which browser, Netscape or Explorer, had a better implementation of each new feature. For example, Netscape and Explorer had very different approaches to image maps, where they were not compatible with one another. Microsoft were also responsible for making up random HTML tags, such as <top> and <bottom> to define static areas of a page (which would later become the very unfriendly frameset tags thanks to Netscape).

The problem was not that these new features were already out in the wild, but that there were two fiercely competitive products each implementing their own version of the web in order to either protect their market share or to gain control of more of it. Eventually both Netscape and Microsoft gave up on implementing a proper HTML 3.0 spec, for example from Netscape:

Netscape remains committed to supporting HTML 3.0. To that end, we’ve gone ahead and implemented several of the more stable proposals, in expectation that they will be approved. We believe that Netscape Navigator 2.0 supports more of the HTML 3.0 specifications than any other commercial client.

In addition, we’ve also added several new areas of HTML functionality to Netscape Navigator that are not currently in the HTML 3.0 specification. We think they belong there, and as part of the standards process, we are proposing them for inclusion

and Microsoft were left playing catchup in terms of supporting HTML:

Netscape has enjoyed a virtual monopoly of the browser market (about 90% according to some estimates), and this has allowed it to consolidate its position still further by introducing unofficial or ‘extended’ HTML tags. As a result, the Web is littered with pages that only work effectively if viewed in Navigator. By the time other browsers catch up, Netscape has made even more additions.

but that didn’t last long and Microsoft tired of playing that game. Further releases didn’t even mention HTML anymore and instead talked about a web built on Microsoft technology:

Microsoft Internet Explorer 3.0 is the first Internet client to integrate ActiveXTM technologies, which enable developers to create highly interactive applications and content for the Internet. These technologies allow a World Wide Web site to be as rich and interactive as an action game, a multimedia encyclopedia or a productivity application. For the first time, a Web site will be limited only by its author’s imagination, not by the limitations of the technology.

In a very quick year the browser wars had progressed from fighting over HTML tag support and towards the formats and languages that would produce richer client-side applications. The battle between Javascript (the Netscape proprietary client-side scripting language) and ActiveX (the Microsoft proprietary object container) was just around the corner with the release of Internet Explorer 3.0 in August of 1996.

The rest of the story where Microsoft wins, and more importantly, how they won, the browser war is common history. The web had fractured in a big way, with repercussions that would last for over a decade as thousands of developer hours go to waste producing cross-browser hacks and libraries. Despite Microsoft gaining dominance in the browser market and promoting multiple tiers of proprietary technology for building web applications, somehow simple HTML, Javascript and CSS eventually won over and Web 2.0 wasn’t built on ActiveX.

Fast Forward Ten Years

While Netscape has disappeared and been replaced with Firefox, the battle for the web today is not only between browsers but also one between new web platforms and technologies. The market share of Internet Explorer has by some estimates been notched down to 78% (from a high in 2004 of 95%), with Firefox at 16% and Safari, Opera and others making up the remaining 6%.  HTML 4.01 was published in December of 1999 and went on to become an ISO standard as the major browsers built in support for the spec. HTML 4.01 still remains the most widely and best supported HTML standard, but the problems today have migrated to other parts of the web technology stack, specifically with CSS and DOM access.

In what is now referred to as Web 2.0, thousands of rich web applications have been developed using HTML, CSS and XML - more commonly referred to as Ajax (ironically the a and x parts of Ajax started as a proprietary add-on to Internet Explorer in the form of xmlhttprequest). Ajax applications quickly reached limitations of what can be done with current technologies, but they had shortened the gap between desktop and web applications. A number of vendor-backed web client platforms such as Flash from Adobe and Silverlight from Microsoft have been released as a layer above the browser, presenting developers with a very rich desktop-like development environment for web applications. These new platforms work by extending existing browsers through plugins, and while these commercial solutions have already launched there is currently no suitable open source and open standards based alternative that extends beyond Ajax.

Frustrated by the lack of progress with HTML5 at the W3, a group of browser developers split off and formed WHATWG to further develop the specification. The primary mission of HTML5 was to recognize that the web has changed since the original HTML specs, as web applications were now capable of presenting very complex user interfaces and could make use of more advanced system functions (for the interface, Silverlight uses XAML while Flex/Flash uses MXML). The spec began as Web Applications 1.0, which was an umbrella term to describe not only the new HTML5 spec but other associated specifications such as CSS2, DOM5, ECMAv4 and new API calls (such as local browser storage).

The WHATWG working group spec was eventually (after 4 years) folded back into W3, and Microsoft joined the effort again. In the interim, developers searching for a rich web app platform beyond Ajax had little option other than to join either the Microsoft or Adobe universe. Progress on implementing the HTML5 spec was still very slow, until Google recognised the threat of a Microsoft or Adobe dominated web and stepped in by creating Gears. Gears is Google’s way of hurrying up implementation of HTML5 features in browsers, and they have backed it at each step by having their own applications such as Gmail and Reader support the new API calls.

Apple is another company who are fully backing the open, HTML5 alternative for rich internet applications. It was only a few years ago that a visitor to the Apple homepage would find a page dominated by Flash and PDF files. Today Apple have their own open-standards based browser with Safari and back the Webkit open source project. They have also backed up their support for both the free and open alternative by re-engineering their websites and applications to use Ajax over proprietary alternatives such as Flash.

We are back in 1996 again and HTML5 is the new HTML 3.0, but instead of two major browser manufacturers today there are numerous parties with interest in determining what the new web API and virtual machine will look like. In the 1990’s version of events, the open standards eventually won over - which both Microsoft and Adobe have recognized as they have released source code and API details for some parts of their platforms.

Web history teaches us that there is usually a single winner, as all users steadily migrate to the single winning solution which imposes itself as a standard (recall that many of today’s ’standards’ began life as proprietary technologies). There is a big difference though between a standard such as the Windows operating system, and an open standard such as HTML5 - and a repeat dose of the former is the biggest threat that companies such as Google and Apple currently face.

You can read the previous Next-Gen Web post about local browser storage here.

Our sister publication Techcrunch UK noticed that a Location services API had been added to Google Gears. The developers behind Gears have been plotting out future API additions for a while, and those plans have included having Geo-data available to mobile app developers (see the spec here). We found out today that Google is backing up their Location API with a large effort to map out cell-phone towers and wifi hotspots, so that a user’s location can be pin-pointed more precisely.

While some cell-phones have an internal GPS, the data is inaccurate indoors and not available on all devices. The other non-GPS method for accurate location data is to use the location of cell towers. Google can store the lat and long of a particular cell tower in their database, and when their software in the future sees that cell tower on a phone, they know exactly where the phone is. To boot-strap the database, both Google and Apple have been using a company called Skyhook, who drive around pin-pointing the location of cell towers. By using this method Google bypasses the need to have deals in place with network providers for positioning data. In addition to cell-phone towers, Google is also mapping out Wifi locations to form a large rogue base station almanac, which is used for both additional accuracy in location calculations, and also to point users to the nearest available access point.

Once the database has been boot-strapped with initial data and launched to developers via an API, users of the service will further refine and improve the service by having devices submit information on towers and signal strength (along with location) back to Google. This means that over time, the service improves itself and will be able to work almost anywhere in the world, regardless of local regulations, network providers or restrictions.

It is expected that the service and associated data will be made available for free to developers using Google Gears (specifically the new Windows Mobile version). For developers of mobile applications, it means that they now have a very accurate way of not only calculating a users position, but also an easy way to pinpoint other locations as a basis for a location-based service. There is also an effort to develop and define a standard API for accessing Location data and services in the browser. As with local browser storage, Google are leading the way here by implementing first and then working with other browser developers on a standard.

The next-gen web is starting to gather pace, as this week MySpace integrated Google Gears, Yahoo! announced their new BrowserPlus product and Google launched a browser-based edition of their 3D Earth product. Technologies and formats such as AIR, Silverlight, JavaFX, Gears, XUL, Web Applications 1.0 (DOM5, HTML5 etc.) allow developers to accelerate beyond AJAX and towards a new generation of web applications with better performance, more functionality and tighter desktop integration.

Developers and users are now presented with more web technology choice then ever before; “DLL hell” has been superseded by “plug-in hell”, as a variety of companies present their versions of what the next-gen web will look like. But on the web, such choice can come at a cost to both users and developers. More than a decade has passed since the first battle over web formats, back then it was Microsoft, Netscape, Apple, AOL and others laying different foundations in the form of browsers, scripting languages, web servers and more. The legacy of that battle is still being felt today, as Javascript developers rely on whole libraries to assist them in developing cross-browser code and CSS developers depend on a catalog of hacks so that their sites can look consistent across different browsers.

With the new rich web application technologies still in the development phase, there is an opportunity to not repeat the mistakes of the past and instead take a standards-based approach. Thankfully during the course of the previous decade companies such as Microsoft became more receptive to open standards, data portability and cross-platform support. Having broad support for open standards simplifies technology for both users and developers, but it is obvious that not all of the currently announced technologies, such as those listed above, will

During the course of a series of posts here on Techcrunch we will look at various elements that make up the next-gen web and evaluate the options available, current support as well as how well standards are being adopted. In light of the recent announcement from MySpace that they are using Google Gears in their application, in this first post we will evaluate browser-based local storage cache’s.

Browser-based Local Storage

As web applications became more popular there was a general demand for an ability to run web-based applications offline. The first such solutions that could work without requiring a browser plugin or separate application were those that relied on the caching headers within HTTP to store objects within the browsers cache. Javascript libraries such as Dojo implemented support for offline web applications using the same principals, but applications were very limited in scope as there was no easy way to store structured data on the browser (Dojo now also abstracts a variety of other storage engines including Gears - tip: Dylan) .

In May of 2007, Google launched Google Gears, a browser plugin that allows web applications to synchronize data into a local data store and then allow web applications to function offline. At the launch of Gears, Google Reader was adapted to support it, and the emphasis of the pitch for Gears was about offline application access. What was less known is that Gears is a lot more than just offline access, as it provides three primary functions:

• Caching of resources (HTML pages, images etc.)
• Structured data storage in a database
• Asynchronous background worker threads

The part of this we will focus on here is the local object and structured data storage. Gears provides these functions via a Javascript API, which can be accessed by any web application. The structured storage is provided by Sqlite, a popular lightweight RDBMS. With the local database, the developer can not only perform queries and inserts to record new data, but also more complex SQL like joining between tables etc. Although you can have multiple applications using Gears, each app runs in a sanboxed environment with a domain-based security model (similar to cookies and Ajax requests). Sqlite has been built into Firefox since version 2.0, but its API is only accessible from an add-on or a core Firefox component. The Gears plugin bridges that gap and makes it available within the client scripting environment.

Before Gears was launched, the Web Hypertext Application Technology Workgroup (WHATWG) had begun work on its Web Applications 1.0 draft spec, which included structured data storage as part of HTML5. The current draft spec from the working group includes definitions for a Database object for accessing and querying a local data store. The details of the implementation are left up to the vendor, but the API is detailed in the spec. Firefox will be implementing parts of the same storage API from the WHATW spec in version 3.0 of the browser, which is currently available as a preview release. The key components of the WHATWG spec are:

• ApplicationCache - for storing objects in the local browser cache (and checking them)
• navigator.onLine - check if the browser is online or not (and use cache plus local data store if required)
• Storage interface and events - used for storing name and value pairs via the sessionStorage DOM attribute.
• Database interface - used for connecting to the local database. Supports SQL (or a subset thereof, depending on the server used), versioning, error events via callback
• Threading and Callbacks - so that multiple requests can be sent to the local data store asynchronously.

Implementing local storage, caching and offline access are relatively easy. The application can first check to see if these functions are supported, and then setup the local cache by synchronizing the users data in background processes. While a thread is running, either uploading or downloading, you can query it to check on its status to provide the user with feedback (eg. a progress bar). Once the data is local, by running database queries on the local machine developers are able to drastically improve performance. Currently many web applications use the browser as only a presentation layer, for eg. a spreadsheet application may do a round-trip back to the server to work out even elementary calculations such as =1+1. By utilizing the local data store and client-side code, the developer is able to offload processing and storage to the client and provide a much smoother, desktop-like experience at the same time.

Current And Future Support

The issue is that the majority of the WHATW specification was written after Gears was released, so the Database and LocalServer objects used in Gears are not compatible with WHATW - for now. The good news is that Google have come out and fully backed the storage portions of the WHATWG HTML5 spec, so developers with apps running on Firefox 3 with Gears installed will have a choice to use either the native implementation, or the Google implementation. Google go on to say that they will likely offer extra features as an incentive for developers to continue to target Gears over-and-above the HTML5 implementation (features such as desktop shortcuts, etc.).

Other alternatives for local data storage, such as Flash local storage, are completely incompatible with the WHATW specification. The developers at WebKit were very quick to announce that they have started implementing the storage portions of the HTML5 spec also. It is currently available in nightly builds, so in the near future we will see support in both Konquror and Safari. Opera have also announced similar plans and they are actually leading everybody when it comes to implementing HTML5 and Web Forms. Yahoo! BrowserPlus was only announced yesterday, and it is currently unclear wether their local storage support is compatible with the specification as laid out by the working group.

Local storage is a major new feature of the new web API, and developers will not only have consistent support across browsers but will also have the option of Google Gears (which is already available) as well as Yahoo! BrowserPlus (depending on how it works). There is just one browser maker missing in this discussion so far and that is Microsoft. Microsoft have released an early preview of IE8 and announced a raft of new features, a lot of which are based on open standards such as better CSS and Javascript support (with a more standardized object model). The big question is wether we will see consistent local storage support from IE8 following the same spec as the other browser vendors. The IE team have announced that IE8 will support DOM Storage, but that is only part of the overall local storage spec (ie the Storage object described above only).

Overview of Current And Future Support

Notes: Once details of BrowserPlus are known the table will be filled in. Gears has pledged to adapt the standard. IE8 has only announced partial support. WebKit have most functionality available in their nightly builds. Both Flash and Silverlight have some form of local storage but not the HTML5 standard API.

It is very rare that a new technology such as local browser storage is so broadly advocated and supported mostly under a single spec. While Microsoft may not have yet announced full support, there is no doubt that they are heading in the right direction. It is also encouraging that implementations from Google Gears and Firefox 3 will follow the working group spec for HTML5. While the new versions of these browsers will not be available broadly for a while longer, Google Gears is available today and since all vendors are targeting the same API, developers can today confidently target and write for the Gears storage API with confidence - something that was almost impossible to do not so long ago.

So with local browser storage and caching, so far the open standard is the winner. The alternate solutions are likely to fall by the wayside or will adapt to implementing the same API.