In July of last year, I wrote about The New Apple Walled Garden. The post was about the irony of developers and advocates who were otherwise open standards and open source champions being absolutely pro-iPhone, a platform that is closed and proprietary in every sense. Since that post, the horror that was foreshadowed by some has been realized – rejected apps, rejected apps, rejected apps. We documented the troubles here at Techcrunch and the overall response was nothing more than long comment threads, complaints, and a few wise people changing their minds. The complaints to date are from some bloggers and a small number of application developers, incidents that Apple are able to write-off as being minor, as they have a dedicated fan base and growing market share to fall back on. That was, until yesterday.

Yesterday, a high-profile iPhone developer became fed up with the nature of the platform and decided it was time to call it quits. Joe Hewitt of Facebook not only pronounced that it was time for him to move onto ‘other projects’, but had the courage to state that his reason was because of the closed nature of the iPhone platform and his frustration with the approval process. Joe is not just the guy who wrote the Facebook application, within 12 hours of the first iPhone launching he released a library for app developers to create iPhone-like applications. This was back in the first generation, when iPhone ‘applications’ were nothing more than websites. Without any documentation from Apple, and with sheer enthusiasm for the new-born platform, Joe created a library for other developers that would help them build applications that would mimic native iPhone applications built by Apple.

As somebody who downloaded the very early releases of Joe’s library, I could immediately see that most, if not all, of the first iPhone applications were built on, or at least inspired by, the iUI library he released. The credibility that Joe has and the work that he did not only inspired developers, but it gave them an easy path to developing the first generation of software for the iPhone. With the statements that Joe made yesterday, Apple has not only lost another developer that it can write-off, but has lost somebody who was an early adopter of their platform and an impetus for others.

Most iPhone and Apple fans would retort that “Apple make great products, and it is winning in a market where the consumer has free choice”. I agree that they make great products, I am writing this post on a Macbook. I was beside myself with excitement when I found out about Rhapsody, about OS X, about the new Mach kernel, about FreeBSD code being used for userland (my code is in there, somewhere). I was so enthusiastic about the second coming of Jobs that I had an email exchange with him about incorporating OpenSSL, amongst other things, when the early dev previews were out. I was totally sold, because an operating system was being built and released that combined the best of UNIX with the best of great interfaces. Finally, the open source on desktops conundrum had been solved, I cheered. The biggest non-Microsoft company had adopted what we knew was good, as a way to compete against the standard. It validated my belief in the BSD license, and I was completely spellbound and a fan (although not in the more recent fanboi sense).

It was not until the iPhone was released that I felt let down. I felt betrayed. I wanted to hack, and I wanted to do so standing on the shoulder of a giant who was gaining market, a giant who was my old friend. I hold a very strong belief in the open market, a concept which at a theoretical level is difficult to argue against. The iPhone took advantage of a market where the competition was completely clueless. It took an intelligent and smart outsider to recognize that. What has shaken my belief in the open market is that an otherwise good company can enter a market, show them how it is done – but do it in a bad way for the overall ecosystem, and at the same time win the support of people who would otherwise philosophically disagree with them, completely on the basis of that company being not-Microsoft and, well, being sexy.

I never believed that Microsoft were evil, first because as a user and developer I had a choice. Second, Microsoft gave me free tools to learn how to code. And last, despite the position Microsoft were in on the desktop they never asked me to send them my code so that they could test it against their black-box of what is ‘compliant’. Microsoft never sent me a letter to say that speech bubbles can not be used in my application. Microsoft platforms let me run whatever-the-hell voice provider I wanted. Microsoft, as far as I can recall, also never told me that I could not have a sense of humor (the ironic 1984 reference has already been done, thanks Jon). Developers today also have a choice with mobile applications, and the sooner more developers raise their blinkers and realize that the popularity of the iPhone is built on the applications they are building, the sooner we can either get rid of this mess and see Apple change, or see a new more open alternative thrive.

Hewitt’s statements, as a model iPhone developer from a large company, can be the tipping point. The only thing holding this back right now are Facebook themselves, who seem keen on preserving a business relationship and casting Hewitt off as a rogue. Facebook came out today, and in a more official capacity (ie. somebody with ‘communications’ in their title, as opposed to ‘developer’), said that “Facebook’s relationship with Apple and our commitment to the iPhone platform remain strong”, and that “There’s been a fair amount of confusion and speculation about Joe’s comments” (chuckle, chuckle) and that “Facebook has a great team of engineers taking over iPhone related development”. Joe is probably taking some heat from his employer right now, and he probably knew he would before he made any comment. Facebook could have simply shifted Joe to another project (Android, I hope), and many wouldn’t have noticed – but he stood up for what he believes in, and what many have been thinking, and he deserves the full support and credit from everybody who believes in transparency and free opinion, regardless of which side of the iPhone debate your opinions may reside.

If it comes down to Facebook vs iPhone, Facebook wins. If Apple hold to their position on being the gatekeeper for everything on their platform, we only win if the developers say no. An iPhone platform with applications only from Apple and no third-parties is no longer a viable platform, and no longer a device that consumers will purchase — because they are making decisions based on applications and access, not on the brand or suburb engraved on the back of it (I hope).

Facebook should recognize this and back Joe all the way. If they do, it will show that that interest of what they want to do takes precedence over what a handset manufacturer wants to do. Apple can squash small developers, but if a big developer were to set aside short-term business interest for a moment, they will win in the longer term. If only we could all do that and not be blinded, perhaps, well, the free market could work again.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

PHP was developed further and commercialized by Zend, but Lerdorf has maintained an ongoing involvement with the open source project. Lerdorf has worked at a number of companies since first developing PHP, but has spent a large part of his professional career with Yahoo and he had a strong association with the company. Lerdorf is one of a number of star engineers and developers who have left Yahoo in recent times, and the stable of notable and high-profile engineers at the company has whittled out.

Lerdorf has been more recently noted for his blog posts, such as his outline on his philosophy to developing PHP applications: The no-framework PHP framework.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

I got the first alert as I was stepping towards the door to leave (it is always like that), and when I got back to my seat found that half the web seemed to be talking about it. The main Techcrunch site was still serving pages to most, due to our super-aggressive-mega-cache, but it seemed that the entire Dallas NOC was being rebooted.

From the status blog:

As of 12:35AM CST Rackspace Cloud engineers are seeing intermittent connectivity to our WC2 cluster in our Dallas – Fort Worth (DFW) and data center. We are working to resolve the issue as quickly as possible and will update the status post accordingly.

If you have any questions or concerns please contact our support via live chat or at 1-877-934-0407 international +1.210.581.040.

UPDATE: As of 1:15am CST, Rackspace Cloud engineers are still working to address the current connectivity issues. We are making significant progress and we will post another update here shortly.

UPDATE: As of 1:30am CST, service has been restored to the majority of our technology clusters in our WC2 cluster. Some sites may still be having performance issues, We are continuing to monitor and address the situation. Additional updates to follow.

From slicehost (who actually mention power outage):

DFW Interruption
November 3rd, 2009 @ 01:14 AM

UPDATE 1:16AM CDT: Power has been restored, however, we’re working to check all our systems and make sure everything comes back up correctly. Slices have not yet been restarted. We’ll try to keep you updated as much as possible.

We are currently experiencing a service interruption in our Dallas data center. Our engineers are currently working to restore connectivity. We will send an update as soon as information becomes available.

And from Scoble, on Twitter:

(the list he pointed to is actually a good one to follow if you are a Rackspace customer).

This will likely lead to many cursing the cloud, when in essence there is nothing about this problem that seems unique to being a ‘cloud problem’. What is more concerning is that the NOC seems to have run out of power (almost unimaginable) and then took so long to come back online.

So – how did you all spend the downtime? It seems most admins and devs from Rackspace hosted companies were just hanging out on Hacker News and IRC bitching about RS (first time I noticed that he shares initials with his employer).

As soon as we know what happen etc. or any more, we will be posting updates here

Update From Rackspace: from their site:

Rackspace has experienced a service interruption during tonight’s scheduled maintenance on UPS Cluster G. We were testing phase rotation on a Power Distribution Unit (PDU) when a short occurred and caused us to lose the PDUs behind this Cluster. The phase rotation allows us to verify synchronization of power between primary and secondary sources.

All power has been restored and devices are being brought back online. The PDUs were down for a total of about 5 minutes. We have aborted the maintenance for the remainder of the evening and will reschedule this for another date.

Service to Cloud sites has been restored and we are continuing to work with Cloud sites customers to bring them online.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Unlike completely elastic hosted DB services, which abstract a large-scale cluster into a shared environment for customers, the Amazon model is to step up or down through tiers of service based on requirements. The tiers of service (with names that seem to be inspired by a fast food restaurant menu) and pricing are:

You also have to provision a set amount of storage, which is charged at $0.10 per GB-month (pre-provisioning means that you can run out of disk space, it wont grow out). Requests are charged at an additional $0.10 per million requests.

Backups are available (full, snapshots etc.) and backup space equivelant to the provisioned storage space is available for free. Additional space is $0.15 per month. Data transfer is charged at the standard AWS rates, with no charge for data transfers between AWS services (ie. if you have your web server at one host, and the DB with AWS, you will be charged for all the traffic between the web server and the DB).

AWS offer a large range of services, and full RDBMS hosting seemed like an obvious service to offer. AWS has the existing SimpleDB service, which is a key-value based data store.

My initial take on the new RDS service is that it seems that it involves pre-defined and pre-configured EC2 instances with MySQL running. This makes the task of creating and starting new DB instances easier, but does not mean that your resource allocation will automatically grow and scale with resource requirements. There are existing third-party services, such as Fathom, that are built on AWS and use EC2 to create and manage DB instances.

Your application will have to recognize that more resources are required, and make the appropriate API calls to either step up or down along the tiers of instances available. RDS, like most AWS services, provides building blocks for developers to use.

Update: Amazon has now officially announced the service on the AWS blog.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

We received a number of tips early this morning that the majority of web servers at Twitter was exposing server and load-balancer status information to the public. The status page, which are an (often default) option in the open source Apache web server dump an output of all connections and state information for a particular server. The information is used by administrators to monitor servers, and the pages are often either removed entirely or locked down to prevent the information from being used for nefarious purposes.

At some point in the past 24 hours (I would more accurately guess 22 hours 28 minutes and 4 seconds ago, based on the status page itself), the Twitter web servers introduced a misconfiguration to expose this information to the public. The page includes overall server statistics along with every HTTP requests currently being handled by that server, with the full request URL. The server status page is usually accessed by requesting /server-status for a web server. In the case of Twitter, this exposure allows anybody to see requests that sometimes rely on being secret to remain secure, such as oAuth keys, which are used to authorize applications to access Twitter accounts.

News of the pages being open spread quickly through Twitter, with some calling it “great transparency” while others recognizing it for what it is – a little too much transparency, and unintentional. Twitter were very quick to respond and blocked all access to the page, and the vast majority of the information found is purely informational and can be deduced through other means. Your Twitter account is probably safe again, but that doesn’t mean we can’t geek out while we get a sneak peak at what Twitter looks like behind the curtain.

Screenshot of one such page below with some of the information cut out.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

Update: Yammer is back! In a blog post the company explains that the issue was with a master switch.

Yammer, the Twitter-like short messaging service for business users, has been experiencing a prolonged period of downtime today due to DNS issues. The service first went down over 12 hours ago, was alive for a short period tonight, and then became unresponsive again a few hours ago. The issue is also affecting sister company Geni, who share the same DNS servers.

A look at the whois records for the domains shows that there are only two name servers assigned, and they are ns1.geni.com and ns2.geni.com – and they are both down. I wasn’t able to locate an IP address for the Yammer servers to test if the actual service is still there, but a query to the root servers shows that the IP addresses for the two name servers used by the domain are on the same netblock and are both down at the moment.

DNS is very fault tolerant, since it is possible to setup secondary servers that know where to find the answer to a query, and query responses are heavily cached all the way down to the local machine performing the lookup. There are a number of commercial services available that offer distributed DNS hosting along with advanced features, such as EasyDNS, who we use at Techcrunch (Disclosure: we use them).

We use and love Yammer at Techcrunch, and the product won the 2008 Techcrunch50 conference. We have become very accustomed to using Yammer as a replacement for a lot of internal email and Skype group chat, so we are almost lost at the moment without it. We can definitely sympathize with other Yammer users flooding Twitter with questions and complaints (Yammer has been very responsive to queries over Twitter, although has not confirmed a firm ETA on the service being back up).

Yammer is an enterprise service, being used my a number of corporations for internal communication. When Twitter goes down, we can moan about it and make do with not knowing what our friends are having for lunch. But when Yammer is down it has an effect on those businesses using it as a communication tool.

There are a number of emerging services taking aim at the corporate short-messaging market, least not Google with Wave and TC50 demopit winner Socialwok. For all of these services, factors such as availability and reliability are far more critical than with consumer oriented sites, and with the enterprise market these factors often take precedence over features or nice design.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

A chronological system for indexing information breaks down quickly once the amount of information received reaches a certain critical point. Active users of email constantly moan about the information overload they experience, and the information is only a load because it is difficult to sort through and manage in modern systems. According to the cognitive theory of choice complexity, that feeling of load multiplies with each incremental increase in choices and decisions having to be made. In the email world this leads to a complete breakdown, and the trend of email bankruptcy (deleting all email and starting again).

Chronological order became more common on the web as social networks, such as the Facebook, blogs, feeds, feed readers, FriendFeed and services such as Twitter designed around the same paradigm – leading to most recent being most important. Some call it real-time, others call it information overload.

A default view of chronological order presents a natural barrier to the number of information sources that can be managed effectively (Scoble somehow broke the barrier, he is an exception). With only a few dozen feeds, a hundred or so emails a day and following one hundred or so people on Twitter, I find myself constantly behind and not being able to manage. When I am reading these sources, I find myself simply scanning for what is most relevant and most important – for eg. I will quickly reply to an email from a co-worker, while leaving others to slowly creep into the abyss of my archive.

Chronological order needs to be abandoned in favor of relevance. Without relevance, our ability to manage large sets of information is inefficient. The technology for relevance exist today, for eg. spam filters are able to tell us what we definitely don’t want to read. Real world information retrieval and organization is based on relevance, either what somebody else believes is relevant to us, or what we decide is relevant. Newspaper stories are not laid out in the order that events took place and libraries do not catalog their books in the order they were published.

Web applications that present relevance over chronological have proven to be popular. Techmeme hacked RSS, and instead of reading 50 feeds I can have Techmeme read 20,000 for me. Community-powered sites such as HackerNews are similar, they float up the latest content based on what a like-minded community finds interesting. The TiVo hacked television by taking chronological out of the picture and applying relevance.

Email applications have attempted to hack what is essentially relevance into the traditional chronological order. Old desktop email clients introduced folders and filters. Gmail introduced labels, adding a star to a thread and grouping multiple emails into a thread. Yahoo Mail attempts to highlight emails that it believes are from people close or important to you.

I hand over a lot of information to the applications that I use every day, but I am getting nothing in return (other than ads that creep me out). Every time I click a ‘like’, or I re-tweet, or I bookmark a page, or I spend time reading a post, that information can be stored somewhere and used to figure out what information is most important to me. I would happily exchange that part of my privacy for the ability to save a few hours each day and the pain of having to personally sort through all this information.

The ingredients for a personalized aggregator of all information exist today. A working solution would allow me to funnel far more data into my stream, and to not only discover more, but become more efficient. The second by second and minute by minute chronological order paradigm is broken, and like QWERTY, is a legacy from a world where systems were not smart enough to determine relevancy and real networks did not exist.

Original backwards post here

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

The big story today is about Microsoft subsidiary Danger losing all T-Mobile Sidekick customer data from their servers. Danger is the company noted for the T-Mobile Sidekick, the revolution in cloud mobile, and most memorably, almost everybody living in 90210 having to get new phone numbers because of Paris Hilton. Valued T-Mobile Sidekick customers received a notice today from the company updating them on the “data disruption” problem. The good news is that data is no longer being disrupted. The bad news is that there is no data left to be disrupted.

This latest large-scale publicized data loss will surely lead to managers everywhere forwarding a link to the story to their IT departments asking “what are we doing so that this doesn’t happen to us.” It will lead to the issue of data loss and backups being written about ad naseum by technology pundits. Research companies will rub their hands together as they prepare new 80 page whitepapers with titles such as “How Companies Who Pay Us Money Can Prevent Your Data Being Lost” (complete with FDA “may cause drowsiness” warning label on the cover). Consultants will flock to their customers, pat them on the head, and reassure them that everything is ok because their project specification powerpoint shows that they included two of everything (and charged for it).

Backups are a hard sell. Most of us don’t want to think about things going wrong (or put more colloquially, shit hitting the fan). Spending your Saturday afternoon staring at a progress meter that seems to be moving backwards is the polar opposite of fun. If there was a brainwave study of people in the process of backing up data, it would probably show no activity at all (but they could use the results to help calibrate the machines). Furthering the point of no interest, Google trends shows that while the volume of news stories about backups and data loss is increasing over time, volume from people searching about it is proportionately decreasing. We are only shaken out of this slumber briefly when there is an incident such as the one at Danger this week.

Like the death of a celebrity from a drug overdose, publicized data loss incidents remind us that we should probably do something about taking better care of our data. But we usually don’t, because we quickly remind ourselves that backups are boring as hell, and that it’s shark week on Discovery. Our previously well thought out backup and recovery plans are expunged as we scan the perimeter of the clinic for the shortest fence to jump over and bolt back to freedom.

Those who are organized and backup their data usually discover the later, larger, part of the problem – restoring from a backup: Where did I put the backup? It’s an old copy. That file I was just working on isn’t there. It was never actually backing up. No software I use can read this stupid fucking format, etc. For most of us, by backing up, we are only setting ourselves up for a bigger failure down the road.

If you read almost any technology website or newspaper, you could be forgiven for thinking that “The Cloud” solves everything. When “The Cloud” is proposed as a solution to any problem most nod in agreement, not wanting to appear out of the loop by asking what the hell it even means. It certainly isn’t a solution to backups – as Sidekick users found out today, and ironically, as 7,500 users of online backup provider Carbonite found out after the company lost their backups (Carbonite can take some comfort in that they now rank very well for ‘data loss’ in search engines because of the incident. What do they say about bad publicity?).

In the Danger case, it appears from initial speculation that the data was lost because they attempted to upgrade a storage array without backing it up first. Here is a case of smart and rational people who do this for a living at one of the best companies in the world, and they didn’t even bother making a backup – so what hope do we have? Relying on the cloud as a backup didn’t work, because somebody forgot to backup the backup. Roman poet Juvenal foreshadowed this very problem when he wrote “Quis custodiet ipsos custodes?” (at least I think he did, hard to tell because there was no word for “backup” back then).

Storage technology does a reasonable job of keeping data intact, considering that it is only a spilt Red Bull away from not functioning at all. The methods used to store data are vulnerable to simple things such as a magnet, and we live on one of those (hint: The Earth). We have become far too reliant on something that is inherently unreliable.

Every systems administrator has at some point in their life experienced the sickening feeling of realizing that they have lost data – and do not have a backup. It is so common that Eminem even wrote a song about it (Lose Yourself, about a sysadmin who when realizing he didn’t have a backup decides it is time for another career (replace ‘music’ with ‘man tar‘ in the lyrics for the full effect)). The sick feeling that all sysadmins have felt after losing data is because of the pressure and responsibility of the situation, sysadmins run the technology, and we expect technology to solve this problem.

The solution may be to do nothing, certainly not to panic. The biggest problem is that we hoard data. We produce more data and information than we ever have, and we are all vain enough to believe that the data we create is so fantastic that it should live on for eternity. Losing the contact list on your phone shouldn’t be a problem – you should know who your friends are anyway. If you are losing sleep because you can’t find an old email you wrote, you likely have deeper issues to address.

Technology has spoiled us to the point where we feel nostalgic when we lose data that didn’t really matter in the first place. If it did matter, a primal instinct would have driven us to do more to preserve, rather than rely on a sleep deprived sysadmin on the other side of the country. If you didn’t care enough to take care of it yourself, then you didn’t really need it. It is our misguided expectation of technology that causes us to panic when we lose data. The only people who have a larger incentive to preserve your data are those who are using it to target an advertisement at you, or sell you something.

Not only is a lot of this data not important, but do we really want to keep it? I certainly would not want a full account of everything I did in my youth sitting on a server somewhere. I am also certain that we do not want the record of our as a society time being documented and discovered by future civilizations based on Twitter messages.

Data experiences its own form of natural selection. What is important will survive, the remainder will thankfully fade away.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

Google made a very minor but significant change to their search homepage earlier this week. While everybody else was distracted by the barcode logo, a few Chrome and Safari users may have noticed that the search buttons now have a certain zing to them, a new and pretty look, with slightly rounded corners, a border around them and a cool looking gradient.

Now, before you think or say, “baa baa techcrunch why is this a story Google change their button baa baa iphone twitter” (or something like that), what is important here is not what they did, it is how they did it.

To achieve sexy buttons, Google has implemented CSS features that are currently not part of any standard and are only supported by Webkit based browsers (ie. Chrome and Safari). To experience sexy buttons on Google, you will need the Google browser (or that other one). The two specific features that are being used to enable sexy buttons are -webkit-border-radius and -webkit-gradient. Both were implemented by Webkit developers as new CSS features over a year ago – and it is hard to argue against their usefulness (where would we be today without rounded corners?).

Browser developers are resorting to going their own way with implementing new features because the standardization process is nothing short of a clusterfuck. In 1996, back in the wild west days of the web, the CSS1 recommendation was published – but the two major browser maker at the time were at each others throats and didn’t pay much attention to it. It only took another two years to get the next version, CSS2, to recommendation status. It was this version, and more importantly, support from the browser makers, that spurred the widespread adoption of CSS and the separation of markup from style on the websites. The first drafts for CSS3 was published in 2001 – and today, a full 8 years later, it is still a work in progress as nobody seems to be able to agree to anything again.

This has spurred the various browser developers to press ahead with their own plans, some of which, such as gradients and rounded corners, can be found in browsers today. These browser-specific bleeding-edge features have always been there, for example, XMLHttpRequest, the core component to Ajax, started a a proprietary extension to IE 5.0. Some of these features live on, some die, and some become part of the standard (or more correctly, a standard).

By implementing currently non-standard features on their homepage, Google are sending out a strong message on what they believe the new standard features should be, and not coincidently, it is the features that their own browser implements and supports. This is not the first time Google has sent a wrecking-ball into the standards process. Google Gears was launched long before Chrome as a way to implement proposed HTML5 standards, such as offline caching, into browsers (see my NextGenWeb series from last year). It was born out of frustration for the slow and beurocratic standardization process – something that Google couldn’t afford to wait for as their web applications could not advance further without a non-aligned platform to build them on.

A large part of the anti-trust case against Microsoft was that with combined desktop, browser and server market dominance the company could abuse that position to make the web a Microsoft web by implementing Microsoft-only features. Google are using their dominance to force an issue that has been stalled for far too long – but the difference is that they are using their force for potentially a greater good (I hope). The theoretical Microsoft web would have been “this website only supports Internet Explorer”, whereas with Google so far we have “this website is a lot better, and has sexy buttons, if you use Chrome (which btw is open source)”.

Update: I originally referred to -webkit-rounded-corners which should be -webkit-border-radius. I blame Web 2.0.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

A large number of web services are geographically restricted, such as Hulu, Pandora and Spotify. The reasons are usually to do with content licensing restrictions, or because US visitors (or visitors from other advanced economies) are of a higher value from a monetization perspective. A web application can only guess at the location of a visitor based on an IP address and other information, such as browser language and regional settings.

IP addresses are mapped to countries (and in some instances, further to states and cities) using large commercial datasets such as GeoIP from Maxmind, which is a ‘best guess’ database based on data it has collected (how, I would rather not know). The system is accurate enough to enable services to block on a country level, but often fail at a more local level.

But the nature of the web means that geographically restricting web services is next to impossible, because those who are technically adept have known how to find and use proxy servers (both open and private) and VPN services to masquerade as being from another country.

The demand for such services has become so popular that more apps are being released that make this process almost as easy as installing any other application – one-click VPN/Proxy install and then pick a country you want to be surfing from (default USA). Even better, there are now VPN solutions available for free – some of which are outright free, others which are ad supported.

If you find yourself outside of the USA and wanting to watch Hulu, outside of the UK and wanting to checkout the BBC, or wanting to rig a web poll, here are some tips:

Proxy Servers

Easy to find, easy to setup. Some sites have become smart enough now to check if the IP address you are coming in from is an open proxy server and will attempt to deny it – but this is most often the easiest solution. The key is to find an open proxy server that everybody else, or even worse, Eastern European crime syndicates, are also not using.

The best source if you are a blogger is to check your spam comments. Most of those IP addresses will not only be open proxy servers (you just have to work out the port – or if you host your own blog, start logging the port), but will be virgin proxy servers.

Otherwise there are a ton of lists available online, often updated each minute, as well as services where you can test your proxy.

FoxyProxy is a Firefox plugin that allows you to easily switch between proxy servers (many Chinese web users are very familiar with having to juggle proxy servers and use such plugins, or browsers that have similar features built-in)

VPN Servers

Similar to a proxy, except that a VPN is an encrypted link to a server that will route all of your network traffic (your computer, in effect, becomes part of the network).

FreeVPN – thefreevpn.com – A completely free VPN client and service for Windows machines. No ads, and a fast service. Not sure what the business model is, which is why I wouldn’t trust it with any personal or private information and restrict it to just movie watching or poll rigging. Best free VPN service and super easy to install (see review here)

Feeedur - www.freedur.com – A commercial VPN/anonymizing service that works well.

HotSpotShield – hotspotshield.com – Another free VPN service, but forces you to click on an ad. Working with Hulu again.

UltraVPN – www.ultravpn.fr – cross platform (OS X support). Both free and anonymous.

The Web Is Flat

Using a proxy or a VPN to bypass geographic restrictions or to preserve anonymity online has been known and used by more advanced users for years. More modern services and tools are making it easier for the average internet user to take advantage of the same techniques.

There are entire business models that depend on geographic targeting, so there is a constant cat-and-mouse game between providers of these services and those seeking to bypass the set restrictions. Those who are seeking to access content are winning though, and they will continue to win, as the very nature of the Internet and web make it near impossible to detect where somebody actually is if they refuse to let you know.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

The big problem with Twitter is assymetric following without limitations on the number of connections, which means that a single account can theoretically have a number of followers limited only by the total number of Twitter users. This adds massive complexity to the system. Other services solve the problem by forcing both sides to agree to friendship, a one-to-one relationship. Others, like Facebook, limit the connections to 5,000 as well. But Twitter has no limits on complexity. And since they are a centralized, bottlenecked system, it is both hard to scale and easy to attack.

The short messaging format is popular, and it is now part of the web. It should thus be designed and implemented as a decentralized service like most other core web services (email, DNS, blogging etc.). The Internet was built to withstand a nuclear attack, and it is a platform that can’t be owned, attempting to completely centralize a new core service has never worked.

As Twitter grows, it needs to be architected more like the Internet.

New Twitter COO Dick Costolo says that he believes Twitter can scale in a centralized way, meaning the status quo will continue. But he acknowledges that it is a theoretical debate at this point, and he says that he hasn’t ruled out decentralizing Twitter.

We believe decentralizing Twitter solves two problems – it will help the service scale infinitely. And it is potentially a very lucrative source of revenue.

Email Is A Business – The Microsoft Exchange Model (Get Your Customers To Pay You And Do The Heavy Lifting, Too):

Twitter should look at how email, and commercial email servers such as Microsoft Exchange Server, developed. The business generates $2 billion or more in revenue for Microsoft, and powers the majority of corporate office functions (email, calendar, etc.). Businesses pay a few hundred dollars for Exchange, plust $50 or so per year per user. Plus, the businesses handle all the infrastructure costs (servers, bandwidth, etc.).

Twitter should sell Twitter Server just like Microsoft sells Exchange Server. They’d then run their own Twitter node on their own hardware.

Twitter likely couldn’t get $50/user/year out of Twitter Server, but they could certainly get more than the zero they are charging now. And they’d move the burden of scaling Twitter to businesses that want a highly stable solution. And users could still go to Twitter.com to create accounts for free, too. They just wouldn’t have the benefit of controlling the data on their own servers, and having the peace of mind knowing that their uptime was conditioned only on their own infrastructure, something under their control.

There would be some issues to work out, like the namespace and messaging between parties (If we had our own Twitter server, my user name would have to be something like @nik.techcrunch, or we could just use the existing global namespace – email). Twitter could build and sell a kick-ass Twitter server for corporations and those who wish to control their own messaging and their own brand.

But the benefits would be huge. Possibly hundreds of millions of dollars in revenue. And a partially decentralized service that would stay live even if Twitter.com went down.

So there are the benefits – revenue, lower operational costs, higher uptime. And there’s one more benefit, too. A decentralized Twitter would suck the air out of the idea that Twitter needs a decentralized competitor. Twitter could own the micro-messaging protocols and core service for the long term. Twitter owns the protocol, the users, the format, the trademarks, the brand and the name – why does it also need to host the whole damn thing?

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Squidoo founder and author Seth Godin has backed down on creating company pages by default as part of their new ‘Brands In Public’ service that launched a few days ago. The idea behind the new service is that brands are able to track feedback from customers on a public ‘lens’ (aka. a web page).

Feedback is aggregated from multiple sources, but mostly twitter and mostly by matching against the brand name. The concept itself is not an evil one, but Squidoo setup feedback pages for over 200 brands at launch without the express permission from the vast majority of them. The hitch was that if a brand wanted to control the lens and the feedback, they would have to pay Squidoo $400 a month – and it was that part of the deal that made a large number of people rightfully angry.

Godin says in a blog post today that they will remove the brands they created by default, and instead make the program opt-in. This is a big step back from yesterday where he left a comment on an excellent blog post by Lisa Barone, who criticized the product as being ‘brandjacking’, by saying:

I’m not sure it’s brandjacking any more than a Google search or a Twitter search is brandjacking. I guess the difference is that we’re making it really easy for the brand to show up next to the stream of comments.

Godin has built a reputation, on the back of his books, as being a marketing and community guru. He must have read some of his own work overnight because today on his blog he says the policy has changed to:

When a brand wants a page, we’ll build it, they’ll run it and we’ll both have achieved our goals.

Godin opens his post today with:

The response from the brands we’ve shared it with has been terrific, but other people didn’t like elements of it. And they were direct in letting me know.

Well we know he didn’t hear that ‘direct feedback’ using Squidoo’s own ‘Brands in Public’ page, which during the storm yesterday conspicuously didn’t mention a single point of negative feedback about the campaign.

Godin also does not have comments enabled on his blog, but the launch of the new Squidoo service just happen to time with the launch of Google Sidewiki – which allows users to leave notes on a website. Many flocked to Sidewiki out of frustration, including SearchEngineLand editor Danny Sullivan, and left constructive and well thought out arguments against ‘Brands In Public’. It is ironic that the ‘customer feedback’ for a product that is meant to aggregate just that all came from other sources such as sidewiki, blog posts, twitter and comments on blogs.

We were going to reach out to Godin yesterday, but instead figured we could write this story by aggregating what everybody in the world thinks of Squidoo, and then asking him to pay us $400 to remove the parts he may not agree with.

Crunch Network: CrunchBase the free database of technology companies, people, and investors

Moments ago Microsoft launched WebsiteSpark, a new program to provide web developers and designers free copies of Microsoft development tools, applications and server licenses for a period of three years. The program is the third and latest launch as part of the ’spark’ series of outreach and support programs designed to engage communities with new Microsoft products. The initial programs to launch were BizSpark, for startups, and DreamSpark, for students.

The WebsiteSpark program announced today provides eligible individuals or organizations with 3-year licenses of Visual Studio 2008, Expression Studio, Expression Web (also part of studio), Windows Web Server, SQL Server and DotNetPanel. To be eligible, an organization or individual developer must be in the business of building web applications or websites for others (ie. clients) and also have no more than 10 employees.

I recall as a young developer constantly bitching about the cost of development tools, which was a real barrier of entry. Microsoft bundled QBASIC with DOS, which spawned a whole generation of developers, but for those who were looking to learn further there was a real commercial barrier because of the price of good compilers and tools. Most of us ended up ripping these tools off by downloading them – which meant that we all became familiar with certain tools (like the old VC++) and then ended up getting real jobs where we would use them. Microsoft have obviously caught on and have realized that they need to lower the barrier for some parts of the market (as with academic discounts) in order to bring Microsoft tools, and in-turn platforms/servers (and services!), into development shops and to developers.

There is a broader motive here – Microsoft want to eventually sell you on the entire platform. But who cares, because frankly, their developer tools have long been the best available (queue flame war). I could never have imagined such programs coming out of Microsoft all that long ago, especially combined with support for more open source (PHP), supporting an open implementation of the entire .NET platform and executives like Scott Guthrie who are not only blogging, put publishing their email addresses so that anybody who has a problem signing up with the program can email him (it is scottgu@microsoft.com, btw).

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

A Russian security group has posted a detailed blog post (translation here) about how they managed to extract the source code to over 3,300 websites. The group found that some of the largest and best known domains on the web, such as apache.org and php.net, amongst others, are vulnerable to an elementary information leak that exposes the structure and source of website files. A web surfer is able to extract this information by requesting the hidden metadata directories that popular version control tool Subversion creates.

The actual ‘exploit’ itself has been well known for a long time. It is the fault of the server administrator or developer, rather than the fault of a particular application, since the working metadata directories in Subversion are only required for working copies of code. What is surprising is just how prevalent the problem is – and who it affects. Finding version control metadata directories is as simple as looking for ‘.svn’ or ‘.cvs’ folders within web paths, for example: http://www.test.com/.svn/.

The metadata directories are used for development purposes to keep track of development changes to a set of source code before it is committed back to a central repository (and vice-versa). When code is rolled to a live server from a repository, it is supposed to be done as an export rather than as a local working copy, and hence this problem.

Most web servers are configured by default to disallow access to directories that begin with a period (the traditional prefix for a hidden file or folder in UNIX) – which makes this problem more embarrassing for the affected sites as not only have they mismanaged their version control, but have somehow managed to disable the standard safeguard in webservers meant to prevent hidden files and folders from being returned to users.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

When Tweets are published there is an additional layer of information below the main message that says when the message was posted, and how it was posted. Here’s an example message Michael Arrington just posted from the Seesmic Twitter web app. If you click on “Seesmic” in that Tweet it takes you to Seesmic.com.

But there’s a problem. Twitter’s API allows developers to register any application name, and Twitter messages posted from that third party application will show that name and will link to anything the developer wants. Only names that contain “twitter” or “tweet” are filtered out. Everything else is fair game.

Robert Robb from TweetBorder emailed us about this, and show this test Twitter message that was posted from “Windows.” He also registered the Microsoft name but deleted it to avoid any legal trouble.

We’ve checked, and the TechCrunch name has already been taken by someone.

This isn’t a big issue yet, but we expect to become one shortly. And if you want to avoid the hassle of trying to get your name back from the Twitter API, we recommend you take steps to register your name and application now.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

RSSCloud is a new format specification for feeds that solves polling and notification issues. It works by adding a cloud element to a feed which describes the path to a cloud server that should be notified when a feed is updated. The cloud server, in-turn, will send the updated feed content to all subscribers and aggregators. There is a description of this process on the RSSCloud website.

The protocol was designed by Dave Winer, who also drafted the original RSS specification and pioneered the use of feeds as a way to aggregate content. RSSCloud allows feeds to be more responsive and real-time. Rather than a polling model (’are we there yet, are we there yet’), it pushes updates and update notifications down to subscribers via a cloud server and API.

The new protocol took a big step forward today as Wordpress.com enabled the cloud tag on all post feeds (comment feeds will be enabled at some later point). Winer tweeted about it today, and Automattic’s Matt Mullenweg has since confirmed in an email that all Wordpress.com blog feeds now support the tag. If you view the source of a feed on Wordpress, such as this one, you can see the new tag:

<cloud domain='rsscloud.wordpress.com' port='80' path='/?rsscloud=notify' registerProcedure='' protocol='http-post' />

A cloud notification server is defined for each channel in the feed. This now means that client tools that support the new protocol will be pushed updates whenever there is a new post on a Wordpress.com blog that the user has subscribed to.

This could also mean the beginning of a new format war for the real-time web, reminiscent of the old RSS vs Atom battles. Another groups of developers, lead by Brad Fitzpatrick, published a format and cloud hub known as pubsubhubbub, which is now being supported by Google Reader. There is sure to be much discussion of Wordpress.com falling into the RSSCloud camp, and which protocol/format/method etc. is better than the other (a debate we will engage in on this blog, no doubt).

Services such as Twitter and Friendfeed centralize real-time data and updates. RSSCloud and broader support of such a protocol is a step in the direction of decentralizing such services.

Update: The Wordpress.com blog now has a post about the update

Crunch Network: CrunchBase the free database of technology companies, people, and investors

We wrote this morning about Gmail suffering some turbulence, but it appears now that it has completely crashed and disappeared. Both Apps For Domain and the usual consumer Gmail service are down completely. Google seem to be going backwards on fixing the problem, this morning they sent out an alert saying:

September 1, 2009 8:18:00 AM PDT
Google Mail service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change.

I use Apps For Domain for everything – my contacts, my email, my todo list, my chat, my documents and more recently, my phone. As soon as it went down, I noticed in less than a second. I am now completely stuck, after a few months of being impressed by how I was able to run my entire life on Google.

It is not just the frontend that is down, but also the backend IMAP and POP servers (Update: they are up, but slow). This is a huge fail for Google, considering how admired they are for all the technology they have built internally to scale out their applications.

Update: The Google App Status dashboard says that there is currently a ’service disruption’ with email.

Update: The outage immediately became a trending topic on Twitter, with thousands of tweets from users noticing and complaining about the outage. The outage that we reported this morning was not as widespread, but could point to a potential originating cause.

Update: Still down. I wonder if the paid Apps for Domain users, who have an SLA, are also down?

Update: New status message:

September 1, 2009 12:53:00 PM PDT
We're aware of a problem with Google Mail affecting a majority of users. The affected users are unable to access Google Mail. We will provide an update by September 1, 2009 1:53:00 PM PDT detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change.

They will be back in an hour (the engineers must have been out at lunch).

Update: Apparently IMAP/POP are up for some. Setting up IMAP …

Update: New message from the Google Twitter account:

We’re aware that people are having trouble accessing Gmail. We’re working on fixing it. Apologies for the inconvenience

Update: For those of you who use the web interface who want to also grab their email with IMAP or POP, instructions courtesy or Rajeev. Only works if you had IMAP/POP enabled before this downtime.

SMTP: smtp.google.com
(TLS, port 557, enable authentication)

IMAP: imap.gmail.com
(Enable SSL, port 993)

login: user@domain.com

Update: Downloading my mail now with IMAP. Slow, but sorta working.

Update (2:06PM PST): New update message. Still down, and now no ETA on being back up:

September 1, 2009 1:02:00 PM PDT
We are continuing to investigate this issue. We will provide an update by September 1, 2009 2:16:00 PM PDT detailing when we expect to resolve the problem.

Update: Google has posted to their blog:

We know many of you are having trouble accessing Gmail right now — we are too, and we definitely feel your pain. We don’t usually post about minor issues here (the Apps status dashboard and the Gmail Help Center are usually where this kind of information goes). Because this is impacting so many of you, we wanted to let you know we’re currently looking into the issue and hope to have more info to share here shortly. If you have IMAP or POP set up already, you should be able to access your mail that way in the meantime. We’re terribly sorry for the inconvenience and will get Gmail back up and running as soon as possible.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

The Twitter document leak fiasco started with a simple story that personal accounts of Twitter employees were hacked. Twitter CEO Evan Williams commented on that story, saying that Twitter itself was mostly unaffected. No personal accounts were compromised, and “most of the sensitive information was personal rather than company-related,” he said. The individual behind the attacks, known as Hacker Croll, wasn’t happy with that response. Lots of Twitter corporate information was compromised, and he wanted the world to know about it. So he sent us all of the documents that he obtained, some 310 of them, and the story developed from there.

This post isn’t about the confidential information taken from Twitter. It’s about exactly how Hacker Croll was able to get such deep access to Twitter in the first place.

It’s clear that Twitter was completely unaware of how deeply they were affected as a company – when Williams said that most of the information wasn’t company related he believed it. It wasn’t until later that he realized just how much and what kind of information was taken. It included things like financial projections and executive meeting notes that contained highly confidential information.

We’ve already said a lot about all of this and the related “server password = password” story that was discovered by another individual last week. But we’ve got two more stories to tell. The first, this post, is exactly how the hacks took place, based on information gathered from hours of conversations with Hacker Croll. The second is what was happening behind he scenes with Twitter as the story unfolded. We’ll post that later this week.

When the story first broke the true scope of what had taken place and how it occurred was not understood. Various bloggers speculated about the cause of the attack – with some placing the blame on Google while others blaming the rising trend of hosting documents in the cloud.

We immediately informed Twitter of the information we had in our possession (and forwarded it to them), and at the same time reached out to the attacker. With some convincing, the attacker responsible for the intrusion at Twitter began a dialog with us. I spent days communicating with the attacker in an effort to gain insight into how the attack took place, what the true scope of it was and how we could learn from it.

We’ve waited to post exactly what happened until Twitter had time to close all of these security holes.

Some Background

In the security industry there is a generally accepted philosophy that no system or network is completely secure – a competent attacker with enough time, patience and resources will eventually find a way into a target. Some of the more famous information security breaches have relied on nothing more than elementary issues exploited by an attacker with enough time and patience at hand to see their goal through. A classic example is the case of Gary McKinnon, a self-confessed “bumbling computer nerd” who while usually drunk and high on cannabis would spend days randomly dialing or attempting to login to government servers using default passwords. His efforts led to the compromise of almost 100 servers within a number of government departments. After McKinnon spent a number of years trawling through servers looking for evidence of alien life (long story), somebody within the government finally wised up to his activities which lead to not only the arrest and attempted extradition of McKinnon from the United Kingdom, but a massive re-evaluation of the security methods employed to protect government information.

A more recent example is the case of Kendall Myers, who after being recruited to work for the Cuban government by an anonymous stranger they met while on holiday in that country, set out to obtain a high ranking position within the State Department specifically to obtain access to US government secrets. Kendall dedicated his entire life to obtaining state secrets, and up until he was recently caught by the FBI had successfully passed on secret information and internal documents to the Cuban government for 30 years. He relied only on his memory, his education credentials and sheer dedication.

The Twitter Attack: How The Ecosystem Failed

Like other successful attacks, Hacker Croll used the same combination of patience, sheer determination and somewhat elementary methods to gain access to a frightening number of accounts and services related to Twitter and Twitter employees. The list of services affected either directly, or indirectly, are some of the most popular web applications and services in use today – Gmail, Google Apps, GoDaddy, MobileMe, AT&T, Amazon, Hotmail, Paypal and iTunes . Taken individually, most of these services have reasonable security precautions against intrusion. But there are huge weaknesses when they are looked at together, as an ecosystem. Like dominoes, once one fell (Gmail was the first to go), the others all tumbled as well. The end result was chaos, and raises important questions about how private corporate and personal information is managed and secured in a time when the trend is towards more data, applications and entire user identities being hosted on the web and ‘in the cloud’.

“Hacker Croll” is a Frenchman in his early 20’s. He currently resides in a European country and first discovered his interest in web security over two years ago. Currently in between jobs, he has made use of the additional time he now has, along with his acquired skillset, to break into both corporate and personal accounts across the web. His knowledge of web security has been attained through a combination of materials available to the public and from within a tight-knit group of fellow crackers who exchange details of new, and sometimes unknown, techniques and vulnerabilities. Despite the significance and impact a successful attack has, the cracker claims that his primary motivation is a combination of curiosity, exploration and an interest in web security. There is almost a voyeuristic tendency amongst these individuals, as they revel in the thought of gaining privileged access to information about the inner lives of individuals and corporations. The “high” of access and gaining unauthorized knowledge must be big enough to carry a cracker’s motivation through the long hours, days and months of effort it may take to hit the next pot of gold.

For Hacker Croll, his first port of call in setting out to gain access to a target network is to make use of public search engines and public information to build a profile of a company or individual. In the case of the Twitter attacks, this public information allowed him to create a rich catalog of data that included a list of employee names, their associated email addresses and their roles within the company. Information like birth dates, names of pets and other seemingly innocent pieces of data were also found and logged. This dragnet across the millions of pages on the web picked up both work and personal information on each of the names that were discovered. Public information on the web has no concept of, or ability to, distinguish between the work and personal details of a person’s identity – so from the perspective of a cracker on a research mission, having both the business and personal aspects of a target’s digital life intertwined only serves to provide additional potential entry points.

With his target mapped out, Hacker Croll knew that he likely only needed a single entry point in any one of the business or personal accounts in his list in order to penetrate the network and then spread into other accounts and other parts of the business. This is because the web was designed at a time where there was implicit trust between its participants – requiring no central or formal identification mechanism. In order to keep private data private, modern web applications have built out their own systems and policies that require a user to register and then manage their identities separately with each app. The identifier that most applications use is an email address, and it is this common factor that creates a de facto trust relationship between a user’s applications. The second factor is a password: a random string that only the user knows, is unique to each application, and in theory should take even a computer months or years to figure out if it started guessing. These two elements would work well enough for most cases, were it not for what is often the single weakest factor: human habit.

Look at the front page of almost any web application and you will see hints at just how hopeless and helpless we are in managing our digital lives: “forgot my password”, “forgot my username”, “keep me logged in”, “do not keep me logged in”, “forgot my name”, “who am i?”. Features that were designed and built as a compromise since we are often unable to remember and recall a single four-digit PIN number, let alone a unique password for every application we ever sign up for. Each new service that a user signs up for creates a management overhead that collapses quickly into a common dirty habit of using simple passwords, everywhere. At that point, the security of that user’s entire online identity is only as strong as the weakest application they use – which often is to say, very weak.

Now going back to Hacker Croll and his list of Twitter employees and other information. Twitter just happens to be one of a number of a new breed of companies where almost the entire business exists online. Each of these employees, as part of their work, share data with other employees – be it through a feature of a particular application or simply through email. As these users become interwoven, it adds a whole new attack vector whereby the weak point in the chain is no longer just the weakest application – it is the weakest application used by the weakest user. For an attacker such as Hacker Croll looking to exploit the combination of bad user habit, poorly implemented features and users mixing their personal and business data – his chances of success just got exponentially greater. Companies that are heavily web based rely largely on users being able to manage themselves – the odds are not only stacked against Twitter, they are stacked against most companies adopting this model.

Unfortunately for Twitter, Hacker Croll found such a weak point. An employee who has online habits that are probably no different than those of 98% of other web users. It began with the personal Gmail account of this employee. As with most other web applications, the personal edition of Gmail has a password recovery feature that presents a user with a number of challenges to prove their identity so that their password can be reset. It likely wasn’t the first account from a Twitter employee that Hacker Croll had attempted to access – but in the case of this particular account he discovered a kink in the armor that gave him the big first step. On requesting to recover the password, Gmail informed him that an email had been sent to the user’s secondary email account. In an effort to balance usability with security, Gmail offered a hint as to which account the email to reset the password was being sent to, in case the user required a gentle reminder. In this case the obfuscated pointer to the location of the secondary email account was ******@h******.com. The natural best guess was that the secondary email account was hosted at hotmail.com.

At Hotmail, Hacker Croll again attempted the password recovery procedure – making an educated guess of what the username would be based on what he already knew. This is the point where the chain of trust broke down, as the attacker discovered that the account specified as a secondary for Gmail, and hosted at Hotmail was no longer active. This is due to a policy at Hotmail where old and dormant accounts are removed and recycled. He registered the account, re-requested the password recovery feature at Gmail and within a few moments had access to the personal Gmail account of a Twitter employee. The first domino had fallen.

Well designed web applications will never just give a user their password if they forget it, they will force the user to pick a new one. Hacker Croll had access to the account, but with a password he had specified. To not alert the account owner that their account had been compromised, he had to somehow find out what the old Gmail password was and to set it back. He now had a bevy of information at his fingertips, a complete mailbox and control of an email account. It wasn’t long before he found an email that would have looked something like this:

To: Lazy User
From: Super Duper Web Service
Subject: Thank you for signing up to Super Duper Web Service

Dear Lazy User,

Thank you for signing up to Super Duper Web Service. For the benefit of our support department (and anybody else who is reading this), please find your account information below:

username: LazyUser
password: funsticks

To reset your password please follow the link to.. ahh forget it, nobody does this anyway.

Regards,

Super Duper Web Service

Bad human habit #1: Using the same passwords everywhere. We are all guilty of it. Search your own inbox for a password of your own. Hacker Croll reset the password of the Gmail account to the password he found associated with some random web service the user had subscribed to and that sent a confirmation with the password in clear text (and he found the same password more than once). He then waited, to check that the user was still able to access their account. Not too long later there was obvious activity in the email account from the account owner – incoming email read, replies sent and new messages drafted. The account owner never would have noticed that a complete stranger was lurking in the background. The second domino falls.

From here it was easy.

Hacker Croll now sifts through the new set of information he has access to – using the emails from this user’s personal Gmail account to further fill in his information map of his target. He extends his access out to all the other services he finds that this user has signed up for. In some instances, the password is again the same – that led Croll into this user’s work email account, hosted on Google Apps for Domains. It turns out that this employee (and in fact most/all Twitter employees and everyone else) used the same password for their Google Apps email (the Twitter email account) as he did with his personal Gmail account. With other sites, where the original password may not work – he takes advantage of a feature many sites have implemented to help users recover passwords: the notorious “secret question”.

Fork the story here for a moment because there is a real issue here with the “secret question” (from here on abbreviated more appropriately as just “secret ?”). For some strange reason, some sites refer to the “secret ?” as an additional layer of security – when it is often the complete opposite. In the story of Hacker Croll and Twitter, the internal documents that we now all know about were only a few steps away from the first account he gained access to. In addition to that, this attacker, and certainly others just like him, have been able to demonstrate that some of the biggest and most popular applications on the web contain fundamental weaknesses that alone might seem harmless, but in combination with other factors can cause an attacker to completely tear through the accounts of users, even those who maintain good password policy.

This is not the first time that the issue of “secret ?” being used in password recovery systems has been raised. Last September, US Republican Vice Presidential candidate and former governor of Alaska, Sarah Palin, had screenshots of her personal Yahoo mail account published to Wikileaks. A hacker or group known only as ‘Anonymous’ claimed credit for the hack, which was carried out by the attacker making an educated guess in response to the security question used to recover passwords. In early 2005, celebrity Paris Hilton suffered a similar incident when her T-Mobile sidekick account was broken into, and the details of her call log, messages (some with private pictures of Hilton) and contact list were leaked to the media. The culprit, again, was “secret ?”.

Giving the user an option to guess the name of a pet in lieu of actually knowing a password is just dramatically shortening the odds for the attacker. The service is essentially telling the attacker: “we understand that guessing passwords is hard, so let us help you narrow it down from potentially millions of combinations to around a dozen, or even better, if you know how to Google, just one”. The problem is not the concept of having an additional authorization token, such as mothers maiden name, that can be used to authenticate in addition to a password, the problem arises when it is relied on alone, when the answer is stored in the clear in account settings, and when users end up using the same question and answer combination on all of their accounts.

From this point, with a single personal account as a starting point, the intrusion spread like a virus – infecting a number of accounts on a number of different services both inside and outside of Twitter. Once Hacker Croll had access to the employee’s Twitter email account hosted by Google, he was able to download attachments to email that included lots of sensitive information, including more passwords and usernames. He quickly took over the accounts of at least three senior execs, including Evan Williams and Biz Stone. Perusing their email attachments led to lots more sensitive data being downloaded.

He then spidered out and accessed AT&T for phone logs, Amazon for purchasing history, MobileMe for more personal emails and iTunes for full credit card information (iTunes has a security hole that shows credit card information in clear text – we’ve notified Apple but have not heard back, so we won’t publish the still-open exploit now).

Basically, when he was done, Hacker Croll had enough personal and work information on key Twitter executives to make their lives a living hell.

Just to summarize the attack:

1. HC accessed Gmail for a Twitter employee by using the password recovery feature that sends a reset link to a secondary email. In this case the secondary email was an expired Hotmail account, he simply registered it, clicked the link and reset the password. Gmail was then owned.
2. HC then read emails to guess what the original Gmail password was successfully and reset the password so the Twitter employee would not notice the account had changed.
3. HC then used the same password to access the employee’s Twitter email on Google Apps for your domain, getting access to a gold mine of sensitive company information from emails and, particularly, email attachments.
4. HC then used this information along with additional password guesses and resets to take control of other Twitter employee personal and work emails.
5. HC then used the same username/password combinations and password reset features to access AT&T, MobileMe, Amazon and iTunes, among other services. A security hole in iTunes gave HC access to full credit card information in clear text. HC now also had control of Twitter’s domain names at GoDaddy.
6. Even at this point, Twitter had absolutely no idea they had been compromised.

What could have happened next is that Hacker Croll could have used or sold this information for profit. He didn’t do that, and says he never intended to. All he wanted to do, he says, was to highlight the weaknesses in Twitter’s data security policies and get them and other startups to consider more robust security measures.

He also says he’s sorry for causing Twitter so much trouble. We asked Hacker Croll if he had any message he wants to deliver to Twitter, and he sent me the following:

Je tiens à présenter toutes mes excuses au personnel de Twitter. Je trouve que cette société a beaucoup d’avenir devant elle.

J’ai fait cela dans un but non lucratif. La sécurité est un domaine qui me passionne depuis de longues années et je voudrais en faire mon métier. Dans mon quotidien, il m’arrive d’aider des gens à se prémunir contre les dangers de l’internet. Je leur apprend les règles de base.. Par exemple : Faire attention où on clique, les fichiers que l’on télécharge et ce que l’on tape au clavier. S’assurer que l’ordinateur est équipé d’une protection efficace contre les virus, attaques extérieures, spam, phishing… Mettre à jour le système d’exploitation, les logiciels fréquemment utilisés… Penser à utiliser des mots de passe sans aucune similitude entre eux. Penser à les changer régulièrement… Ne jamais stocker d’informations confidentielles sur l’ordinateur…

J’espère que mes interventions répétées auront permis de montrer à quel point il peut être facile à une personne mal intentionnée d’accéder à des informations sensibles sans trop de connaissances.

Hacker Croll.

This roughly translates to:

I would like to offer my personal apology to Twitter. I think this company has a great future ahead of it.

I did not do this to profit from the information. Security is an area that fascinated me for many years and I want to do my job. In my everyday life, I help people to guard against the dangers of the Internet. I learned the basic rules .. For example: Be careful where you click the files that you download and what you type on the keyboard. Ensure that the computer is equipped with effective protection against viruses, external attacks, spam, phishing … Upgrading the operating system, software commonly used … Remember to use passwords without any similarity between them. Remember to change them regularly … Never store confidential information on the computer …

I hope that my intervention will be repeated to show how easy it can be for a malicious person to gain access to sensitive information without too much knowledge.

Croll hacker.

What’s the takeaway from all this? Cloud services are convenient and cheap, and can help a company grow more quickly. But security infrastructure is still nascent. And while any single service can be fairly secure, the important thing is that the ecosystem most certainly is not. Combine the fact that so much personal information about individuals is so easily findable on the web with the reality that most people have merged their work and personal identities and you’ve got the seed of a problem. A single Gmail account falls, and soon the security integrity of an entire startup crumbles. So for a start, reset those passwords and don’t use the same passwords for different services. Don’t use password recovery questions that can easily be answered with a simple web search (an easy solution is to answer those questions falsely). And just in general be paranoid about data security. You may be happy you were.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

Previously EC2 instances were able to access temporary storage as part of the compute instance, or persistent storage only on S3 – the Amazon online storage service. The difference between EBS and S3 is that EBS allows block-level access, so that it can be mounted just like any other local storage device from within EC2 and can be accessed across servers and between instances. S3 is accessed as a web service, so performance for latency sensitive applications was never optimal (such as running a database store). EBS provides a much higher level of performance comparable to high-grade local storage in terms of both access times and availability.

Persistant block-level storage for EC2 is perhaps long overdue, as one of the criticisms of EC2 when it first launched was the inability to run a fast data store across snapshots, which made running databases or other data-intensive applications slightly more complicated. Services such as RightScale have built products around helping developers scale and manage MySQL instances on EC2. Other cloud-based computing services such as Mosso or virtual servers from providers such as MediaTemple have had persistent storage options, although what Amazon have developed with the combination of EC2, S3 and now EBS is a tiered approach which provides more flexibility to developers.

Read the rest of this entry at TechCrunchIT.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

iPhone application development house taptaptap has published sales figures for the first month of sales for their two AppStore applications, bringing further insight into overall sales volume and figures for the online store. The two applications developed by the company are WhereTo, an application that provides a more general GPS interface to the iPhone with location-based services, and Tipulator, a simple tip calculator.

WhereTo retails for $2.99 in the store and 24,094 copies were sold in the first month – netting the company just over $50,000 in revenue after Apple took their cut (it currently ranks #69 on the top paid application list). Tipulator retails for 99 cents, and sold 3,168 copies which resulted in just over $2,200 of revenue (it is currently unranked). The table below outlines overall sales volumes and revenues for each application:

taptaptap AppStore sales and revenue numbers for US sales, month 1

The resulting net profit and sales figures are good for a small company that has developed one application that is relatively sophisticated, and another that is very straight forward and simple but yet still brings in $2,000 a month. There is definitely great revenue potential for developers of iPhone applications, as users of the AppStore and the iPhone in general are more likely to pay for applications. Integrating with iTunes makes the process simple for the user, but for the developer poses a challenge as all applications must be submitted to Apple and must meet their approval.

We should also note that while both of these applications have done well, their download figures unsurprisingly pale in comparison to those of Facebook and Tap Tap Revenge, both of which have over 1 million users. The real money in the App Store may well lie in monetizing these free applications, be it through integrated advertising or downloadable content (though it remains to be seen what restrictions Apple will place on this kind of strategy).

Crunch Network: CrunchBase the free database of technology companies, people, and investors

As a Facebook user you can help us protect you by doing the following things:

* Report any spam message or posting you see. The more reports we get, the easier it is for us to respond decisively.

* Never share your Facebook password with anyone. Never. No Facebook employee will ever ask for it, and no one else should know it. If you are ever prompted to log in to Facebook, make sure it’s from a legitimate Facebook web address. If something looks or feels off, go directly to www.facebook.com to log in.

Never entering your credentials on a non-Facebook site is very good advice, which most users should know by now and should adhere to. The problem is that Facebook do not seem to support these same principals when it comes to a users credentials from other sites, such as a users Google username and password, which Facebook requests when a user imports their contacts. The screenshot below is from Facebook, its the feature where a user can login to their Google, Hotmail or Yahoo account, from within the Facebook site, to retrieve their contacts.

This very feature directly contravenes what Facebook has stated in its own good security advice. While the message below the box does state that they do not store passwords, the point is more that the practice of users directly entering credentials from another site is a very poor design decision and generally very poor practice. Each one of the sites that Facebook integrates with supports oAuth or a similar authentication protocol that does not require the user to enter both their username and password. Better yet, most of those services also provide an API where the user can grant permission to Facebook to only access their address book, and not their whole email and certainly not every other service tied into it.

The Facebook security team have stated what is good practice on their blog, perhaps its time for them to direct their energies internally and evangelize support for oAuth and other open data formats as both a more secure and conveniant mechanism for data exchange.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

Over a year has passed since Google completed the acquisition of feed massaging and hosting service Feedburner, and today some users now finally have their feeds hosted on what appears to be Google’s servers and infrastructure. At Techcrunch we have always been big fans of Feedburner, and their widgets and RSS subscriber counts have adorned almost all of our sites since their first days. At some point in the past 12 hours, the feed URL at feeds.feedburner.com began to redirect to feedproxy.google.com. Our subscriber count widget dropped to displaying a zero count for a few hours while the domain change took place.

It appears that only select feeds have been migrated, mostly those with higher subscriber counts. This would indicate that Feedburner has turned to Google to assist with serving the load on high-traffic feeds. Over at TechcrunchIT I recently wrote about the problems that some acquired companies have experienced at Google. The proprietary software and hosting stack at Google can often lead to a slowdown in development, an often long migration phase and in some cases death for the acquired company or product. Feedburner has avoided these problems by remaining largely independent of Google since the acquisition, but at some point they have turned to pappa bear for assistance with handling load and we are seeing the results of that today.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.