In July of last year, I wrote about The New Apple Walled Garden. The post was about the irony of developers and advocates who were otherwise open standards and open source champions being absolutely pro-iPhone, a platform that is closed and proprietary in every sense. Since that post, the horror that was foreshadowed by some has been realized – rejected apps, rejected apps, rejected apps. We documented the troubles here at Techcrunch and the overall response was nothing more than long comment threads, complaints, and a few wise people changing their minds. The complaints to date are from some bloggers and a small number of application developers, incidents that Apple are able to write-off as being minor, as they have a dedicated fan base and growing market share to fall back on. That was, until yesterday.

Yesterday, a high-profile iPhone developer became fed up with the nature of the platform and decided it was time to call it quits. Joe Hewitt of Facebook not only pronounced that it was time for him to move onto ‘other projects’, but had the courage to state that his reason was because of the closed nature of the iPhone platform and his frustration with the approval process.

PHP founder Rasmus Lerdorf has left his long-held position at Yahoo, according to his Twitter account. Lerdorf joined Yahoo in 2002 and has worked for the company as an engineer since. Lerdorf is most notable for creating the original PHP engine, and for being a notable open source developer, speaker and author. Lerdorf developed PHP in 1995 after building up a collection of C macros that he was using in web application development. The original meaning of the anagram is ‘Personal HomePage’, and the language and environment are still the most popular in use on the web today.

A large number of customers of Rackspace Cloud, including Techcrunch, have been experiencing downtime for the past 1h 20m or so. The status blog reports that the service was degraded, and other reports state that it is due to a power outage at the Dallas network operations center. Customers of both Rackspace Cloud and Slicehost are affected, putting services such as Posterous, Dailybooth and others out of commission.

A new generation of database products and companies is beginning to emerge, and one of the more interesting examples is Swedish-based Neo Technology, the developer and vendor of the neo4j graph based database (graph in the data structure sense). The neo4j product has been in development for over 8 years, and Neo Technology are today announcing a new $2.5M round of funding. The company has been developing the neo4j project as a commercial product, and is now taking it to market with a dual-license model.

Amazon has launched a hosted relational database service, Amazon RDS, as part of the suite available at AWS. The new service is a hosted MySQL database instance with the full capabilities and access rights as a normal self-hosted DB. As a hosted solution, the service has an ability to scale out across computational, memory and storage requirements while still being treated as a single db instance by the end user. Pricing stars at $0.11c per hour for the smallest scale specification, and is available now on the AWS site.

We received a number of tips early this morning that the majority of web servers at Twitter was exposing server and load-balancer status information to the public. The status page, which are an (often default) option in the open source Apache web server dump an output of all connections and state information for a particular server. The information is used by administrators to monitor servers, and the pages are often either removed entirely or locked down to prevent the information from being used for nefarious purposes.

Yammer, the Twitter-like short messaging service for business users, has been experiencing a prolonged period of downtime today due to DNS issues. The service first went down over 12 hours ago, was alive for a short period tonight, and then became unresponsive again a few hours ago. The issue is also affecting sister company Geni, who share the same DNS servers.

When email was first created in 1965 it was used as a method to communicate between time-shared mainframe computers. Email has rapidly evolved since then, with the evolution of rich desktop clients, corporate email systems and webmail. Despite the evolution in the core messaging system, and despite the explosion in use of email, the default method for accessing and viewing communications has remained the same: chronological order.

The first webmail imitated earlier mail clients by displaying messages in chronological order. The desktop computing paradigm was folders and files, sorted alphabetically. The web paradigm for accessing information has in most cases become chronological order, mostly because of the email and webmail legacy

The big story today is about Microsoft subsidiary Danger losing all T-Mobile Sidekick customer data from their servers. Danger is the company noted for the T-Mobile Sidekick, the revolution in cloud mobile, and most memorably, almost everybody living in 90210 having to get new phone numbers because of Paris Hilton.

Valued T-Mobile Sidekick customers received a notice today from the company updating them on the “data disruption” problem. The good news is that data is no longer being disrupted. The bad news is that there is no data left to be disrupted.

Google made a very minor but significant change to their search homepage earlier this week. While everybody else was distracted by the barcode logo, a few Chrome and Safari users may have noticed that the search buttons now have a certain zing to them, a new and pretty look, with slightly rounded corners, a border around them and a cool looking gradient.

Now, before you think or say, “baa baa techcrunch why is this a story Google change their button baa baa iphone twitter” (or something like that), what is important here is not what they did, it is how they did it.

A large number of web services are geographically restricted, such as Hulu, Pandora and Spotify. The reasons are usually to do with content licensing restrictions, or because US visitors (or visitors from other advanced economies) are of a higher value from a monetization perspective. A web application can only guess at the location of a visitor based on an IP address and other information, such as browser language and regional settings.

IP addresses are mapped to countries (and in some instances, further to states and cities) using large commercial datasets such as GeoIP from Maxmind, which is a ‘best guess’ database based on data it has collected (how, I would rather not know). The system is accurate enough to enable services to block on a country level, but often fail at a more local level.

The background debate about whether or not Twitter can actually scale has intensified. More than a year ago I asked “Twitter At Scale: Will It Work?” Today Twitter is far, far bigger. And the uptime woes continue.

The big problem with Twitter is asyncronous following without limitations on the number of connections, which means that a single account can theoretically have a number of followers limited only by the total number of Twitter users. This adds massive complexity to the system. Other services solve the problem by forcing both sides to agree to friendship. Others, like Facebook, limit the connections to 5,000 as well. But Twitter has no limits on complexity. And since they are a centralized, bottlenecked system, it is both hard to scale and easy to attack.

There’s a reason why the Internet is decentralized. It has to be. If the Internet were built like Twitter, every bit of data would have to pass through a single node. If that node went down, the Internet would go down.

As Twitter grows, it needs to be architected more like the Internet.

Squidoo founder and author Seth Godin has backed down on creating company pages by default as part of their new ‘Brands In Public’ service that launched a few days ago. The idea behind the new service is that brands are able to track feedback from customers on a public ‘lense’ (aka. a web page).

Feedback is aggregated from multiple sources, but mostly twitter and mostly by matching against the brand name. The concept itself is not an evil one, but Squidoo setup feedback pages for over 200 brands at launch without the express permission from the vast majority of them. The hitch was that if a brand wanted to control the lense and the feedback, they would have to pay Squidoo $400 a month – and it was that part of the deal that made a large number of people rightfully angry.

Moments ago Microsoft launched WebsiteSpark, a new program to provide web developers and designers free copies of Microsoft development tools, applications and server licenses for a period of three years. The program is the third and latest launch as part of the ’spark’ series of outreach and support programs designed to engage communities with new Microsoft products. The initial programs to launch were BizSpark, for startups, and DreamSpark, for students.

The WebsiteSpark program announced today provides eligible individuals or organizations with 3-year licenses of Visual Studio 2008, Expression Studio, Expression Web (also part of studio), Windows Web Server, SQL Server and DotNetPanel. To be eligible, an organization or individual developer must be in the business of building web applications or websites for others (ie. clients) and also have no more than 10 employees.

A Russian security group has posted a detailed blog post (translation here) about how they managed to extract the source code to over 3,300 websites. The group found that some of the largest and best known domains on the web, such as apache.org and php.net, amongst others, are vulnerable to an elementary information leak that exposes the structure and source of website files. A web surfer is able to extract this information by requesting the hidden metadata directories that popular version control tool Subversion creates.

Twitter continues to work through username squatting issues by reassigning trademarked and even non-trademarked user names to their more appropriate owners. It’s a manual process that sometimes takes weeks, but with Twitter’s growing importance more and more brands are trying to lock up their usernames. Now, though, Twitter has a new headache, and poor organization and planning around Twitter’s third party developer platform is to blame.

When Tweets are published there is an additional layer of information below the main message that says when the message was posted, and how it was posted. Here’s an example message Michael Arrington just posted from the Seesmic Twitter web app. If you click on “Seesmic” in that Tweet it takes you to Seesmic.com.

But there’s a problem. Twitter’s API allows developers to register any application name, and Twitter messages posted from that third party application will show that name and will link to anything the developer wants. Only names that contain “twitter” or “tweet” are filtered out. Everything else is fair game.

RSSCloud is a new format specification for feeds that solves polling and notification issues. It works by adding a cloud element to a feed which describes the path to a cloud server that should be notified when a feed is updated. The cloud server, in-turn, will send the updated feed content to all subscribers and aggregators. There is a description of this process on the RSSCloud website.

The protocol was designed by Dave Winer, who also drafted the original RSS specification and pioneered the use of feeds as a way to aggregate content. RSSCloud allows feeds to be more responsive and real-time. Rather than a polling model (’are we there yet, are we there yet’), it pushes updates and update notifications down to subscribers via a cloud server and API.

It was only three days ago that I wrote about the almost hopeless challenge of web security, specifically around new vectors with cross-site scripting attacks. Today came news that an XSS vulnerability had been found in the RubyOnRails development framework – and that applications built on the framework, such as Twitter and Basecamp, were vulnerable to XSS attacks.

The vulnerability was discovered by Brian Masterbrook. He probed Twitter with some Unicode characters and found it vulnerable, tried the same thing on Basecamp and found it vulnerable, and then deduced that it must be a problem with RubyOnRails. He has an excellent and detailed write-up on his site about the process he went through. If you are running RubyOnRails anywhere, stop now and read his post as well as the security notice from the Rails developers and get your servers updated (the patch is in the notice, it will be in the release branch ‘today or tomorrow’).

Today we are trusting the web with our most personal and important data, from private photos and social graphs to finances and key work documents. Our hesitation to share such information has dropped over the years as our trust in our favorite services grows. Yet all the while, the web is actually growing less secure, as sites are left open to new attacks that can spread easily and leave users totally unaware when they’ve been compromised.

Looking back on the history of the web, classic security protection involved patching servers to assure latest versions were running, monitoring advisories from vendors, and maintaining some level of filtering and firewall to keep basic attacks out. Simple moves on the part of an admin or developer could protect sites from 99% of automated scripts. But a few years ago, a new security can-of-worms was opened, as new exploits that took advantage of simple oversights within web applications were being used to steal large amounts of user data.

The Twitter document leak fiasco started with a simple story that personal accounts of Twitter employees were hacked. Twitter CEO Evan Williams commented on that story, saying that Twitter itself was mostly unaffected. No personal accounts were compromised, and “most of the sensitive information was personal rather than company-related,” he said. The individual behind the attacks, known as Hacker Croll, wasn’t happy with that response. Lots of Twitter corporate information was compromised, and he wanted the world to know about it. So he sent us all of the documents that he obtained, some 310 of them, and the story developed from there.

This post isn’t about the confidential information taken from Twitter. It’s about exactly how Hacker Croll was able to get such deep access to Twitter in the first place.

It’s clear that Twitter was completely unaware of how deeply they were affected as a company – when Williams said that most of the information wasn’t company related he believed it. It wasn’t until later that he realized just how much and what kind of information was taken. It included things like financial projections and executive meeting notes that contained highly confidential information.

We’ve already said a lot about all of this and the related “server password = password” story that was discovered by another individual last week. But we’ve got two more stories to tell. The first, this post, is exactly how the hacks took place, based on information gathered from hours of conversations with Hacker Croll. The second is what was happening behind he scenes with Twitter as the story unfolded. We’ll post that later this week.

When the story first broke the true scope of what had taken place and how it occurred was not understood. Various bloggers speculated about the cause of the attack – with some placing the blame on Google while others blaming the rising trend of hosting documents in the cloud.

We immediately informed Twitter of the information we had in our possession (and forwarded it to them), and at the same time reached out to the attacker. With some convincing, the attacker responsible for the intrusion at Twitter began a dialog with us. I spent days communicating with the attacker in an effort to gain insight into how the attack took place, what the true scope of it was and how we could learn from it.