Twitter Search is easily the most promising aspect of Twitter. People talk about mundane updates, or connecting with companies, or following celebrities — but that’s all small scale. The real power of Twitter lies in its aggregate data. Why do you think Google and every other company out there is interested in them? It’s not just because they are the hot ticket in town right now, despite what some would have you believe. It’s all about the data. And Twitter knows that too — and is apparently on the verge of some interesting moves with Twitter Search that will better highlight that.
Speaking on a panel today, Santosh Jayaram, Twitter’s new VP of Operations, had some very interesting things to say, Webware’s Rafe Needleman who moderated it, reports. The most interesting thing is that Twitter Search will soon begin crawling the links that people tweet out and indexing them. This immediately takes Twitter Search, which is still a very basic service, to the next level. This means that no longer will it just be a stream of textual tweets, but it will include millions of web pages as well — web pages that are more or less already curated by the individuals who tweet them out. Sure, there will be some spam, maybe even a lot of it, but this user curation should help real good content from around the web bubble up.
Apparently, Twitter Search will index the content of these pages as well. Yes, this is what Google does. So it should be no surprise when I say that Jayaram formerly worked on the search quality team for Google. [Update: We previously stated that Jayaram was a VP at Google, which is not the case.]
Of course there is no way Twitter Search will index as many pages as Google, but that’s not the point. Twitter Search isn’t meant to replace Google, that’d be dumb. At this point, no one is going to beat Google at its own game (you hear that Microsoft?). Twitter Search is meant to be a different kind of powerful search engine in its own right. A smaller, potentially curated, real-time search engine.
Twitter’s biggest trump card here is the real-time factor. It’s not entirely real-time right now, and there are often delays, but it’s faster than Google — mainly thanks to the nature of tweets (fast to send) versus the nature of webpages (slow to build). And so it’s not yet clear how indexing these linked pages would impact that aspect of Twitter Search. You’d have to assume it would slow it down, but really there’s not much point in assuming anything because it’s not yet really clear how Twitter would use this webpage data in search results.
One thing that is more clear is that Twitter is also looking at ways to better tailor search results. As I mentioned, right now the results are a very basic collection of tweets with key terms. Jayaram says that the company wants to add some sort of reputation filtering to offer better results. Now before everyone gets in a tizzy about the word “reputation” just like when Loic Le Meur brought up the idea of filtering Twitter Search by “authority,” Jayaram says Twitter’s engineers are still determining the best way to calculate this reputation. That seems to indicate that it wouldn’t be something silly like just being based on the number of followers you have.
Things should get very interesting in this space over the next several months. Unless, of course, Google buys Twitter first.
Great post MG. What most people out there don’t understand are the values behind data aggregation and content. Most question Twitter’s ability to make money, but their users are extremely valuable. Just look at Facebook. Gifts/ads are a great revenue model, but each person’s data and commitment to the site is worth well over hundreds now.
keep twitter search simple, stupid!
I think http://www.wallpipe.com will surpass twitter and facebook in no time after its launch. I have been using their alpha version and it’s just great.
Get in line coz it is launching soon ….
We wish you much luck. Cool name!!!. We are demoing first then launching ourselves. See you in the universe.
well the important thing i think is indexing the actual links and pages, rather than just the tweets alone. How valuable is indexing a bunch of , bit.ly links… the value is in the actual links such as ..2009/05/07/google-twitter-to-start-indexing-links-for-search/
which is more relevant
ok, after reading this article my eyes got open, cuz i was wondering why da heck everyone wants to by twitter when in my opinion twitter is d peace of cr4p
booz helps me get down the stairs with style
http://www.epic.../load/8-1-0-309
Are you kidding? There was a website with 200 million unique users 10 years ago, it was called Hotmail. They still can’t monetize those users. And look at facebook? They are bleeding cash so bad it’s not even funny. 512 million and are still no profit at all. When will these companies wake up? Users aren’t worth a dime unless you can monetize them, which so far the giant open free sites can’t. Google still only makes 10% of revenue off of their own sites, the other 90% from external Adsense sites.
I find it very amusing techcrunch is such a twitter fanboy. Why don’t you stick you companies that have a real product?
Good point. Google losts 1 million per day just owning Youtube. They want to buy twitter and lose more?
In response to @annoyed, you have to consider that Twitter is still in it’s infancy. While I agree that one of the most valuable aspects of it is it’s aggregate data, I believe the true genius behind the tool is that it’s direction is being driven by the community. Visit a Twitter application directory such as http://www.twitdom.com or the Twitter Fan Wiki and notice how new applications to aggregate the data are being released by the day by third party companies (representing the community). Be patient, for it will quickly become apparent how the first real time, all accessible, community driven application will change all of our lives.
“Twitter Search will soon begin crawling the links that people tweet out and indexing them”
What happens when those shortened links expire?
yes that will be a problem. …until Twitter buys Bit.ly
long term, they’ll be okay. haven’t started crawling the links yet, so they have time to get their URL shortener.
Dear MG Siegler
This comment is off topic but I have noticed that the exec tweets ads are eating up everything surrounding it.
This happens repeatedly on my Firefox version.
But in Opera TechCrunch looks okay.
I’ll send you a screenshot if you want…
Yesterday the exec tweets ads another box, ate up a whole lot of other ads, covering up the posts.
It’s a Microsoft Ad, and I suspect something fishy…
Maybe they are using a Firefox vuln against Mozilla and also TechCrunch… who knows…
I found the solution:
This is what you find in your leaderboard:
The align=”middle” should be align=”right”
Actually the ExecTweets ads on techcrunch are fine with my firefox.
my version is 3.0.6
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2008121621 Ubuntu/8.04 (hardy) Firefox 2.5
maybe a bug for only this version… so it is a firefox bug???
Screen Shot:
http://s3.amazo...dJwWXg6j2Km0%3D
This is irrelevant, they know the full URL as soon as they follow a shortened link
yes i think the same thing.
“until Twitter buys Bit.ly” wat will to happen to the links of TinyURL ? why cant they just run a bot that will replace all the bit n tiny stuff with ther own URL shortening service.?? i m not very sure how this will go abut technically but this could be a good option.
They’ll but them too! They’ll buy everything with all that revenue coming in.
I can only HOPE they buy bit.ly and place it on the front page so I can easily create a short link before tweeting — I HATE having a 1,000 mile long link with no room to add personal comments concerning what I’m trying to get others to look at. If I could shorten a link on the home page BEFORE tweeting that would be perfect.
Paul…Download Tweetdeck. It has bit.ly already incorporated within it and even has the column views that you are suggesting. You will love it.
That should be irrelevant. By the time they expire they would no longer have a prominent position(if any) in the stream.
Good for Twitter. I like their approach, it should be powerful. Now this is a good story, and thats coming from a non twitter user.
On the issue of
“It’s not just because they are the hot ticket in town right now, despite what some would have you believe. It’s all about the data”
Dude this is the type of Twitter stories we want to hear. I and others who complain about Twitter overload, is because there is far too much irrelevant news on or about Twitter.
Are you folks gonna start covering Twitter when it burbs, farts, sleep etc. 90% of your Twitter coverage is not necessary. Hence the overload.
This is story right here is good stuff.
Well yes, but just like like most other big trends that emerge, there’s silly news with real news. Some people have different tastes, etc. No problem is covering it all, as I see it, read what you like, etc.
TwitterCrunch.
Ad articles. Ads on bars. Seriously this whole blog is an ad.
This is the way why blogs should rule over mainstream media Arrington?
I will sit at my home. Get some articles ordered and sent as PR, charge for them as well from ads from sidebars and live a happy life? Get some hype for companies even without getting out of the room.
See this is why mainstream will always rule. These people actually work their asses off to get information and confirmation. They don’t post nonsense or stupid stuff because then their reputation would fall.
You guys are taking nice care of TechCrunch’s reputation. It’s like rollercoaster going downhill
Cls, you are absolutely right!
Usually I don’t read comments, but for this article I click on them, to see If there is anybody who also hate Twitter like articles in this TwitterCruch, and there is!!!
In this tech blog the 80% of the entries are about this annoying site.
in fact twitter’s power lies in the search, you can already smell that from google itself, when you do a search on google twitter results are becoming increasingly useful.
for example a google search for webmilker shows twitter status message and profile on 3rd. Its great Twitter knows its strength – the search.
How often, exactly, are people likely to search for “webmilker”? What possible value is there in that?
This is great news, and im so excited! however,
Why dont they use google’s massive search power? google already owns the 2 biggest search engines (google & youtube)
The founders are former google employees and google has faith in the microblogging site (why else would they offer to buy it?)
im confused at why they want to build a whole new thing when they have experts on speed dial?
Power idea = âI want to search only in the pages that have been recommended from my friends/trusted peopleâ and not the garbage web anymoreâ¦.
The human comment layer determines if a content has the slightest value or not. If no human on earth want to share a content⦠it is maybe that this content has zero value compare to all the other content that has been shared recommended elsewhere
Metacognition search
was this content in the brain of at least 1 person that wanted to share it with other humans?
A cool idea, that will utterly ruined by a spam overload.
the human sharing marker/metadata. “is this content valuable enough to have been recommended by at least one other human genuinely (vs spam)? How many times?” and the more it has been recommended RTed the more it may seem relevant… and then one can create a “my preferred humans” filtered Web.
Twitter has a great future. That’s why they don’t want to get acquired for nothing.
Excellent idea to include this, however, it would be great if they could find a way to exclude the “RSS tweeter accounts” out of the search results. I don’t wanna see what a bot recommended, I have Google for that.
@Friendfeed could eventually become the "metacognition search" winner. one can search what was in the brain of the other by searching his linked pages. What is important is the "human desire to share" metadata layer and if yes how many times #of RT of the link, post in blogs, emailed etc…
Bang on MG…..I’m not a fan of the Twitter will save the world hype but search is one of the real and valuable uses for Twitter.
Just wanted to point out that we developed an algorithm to calculate “authority” and to show search results based on it. Moreover, tweets which include “links” are shown first.
http://www.tweefind.com
Check it out and let me know what you think. It got much coverage a few weeks ago (not from Techcrunch, though).
Luca
Real time search at the scale of Twitter and its links is easily done by Google [they already update heap of pages on a 5-minute or so basis] and by other search engines.
Twitter’s edge is not technology nor data. Twitter is valuable as a destination site, because of the user base and because it is changing behavior of users into new patterns that might be exploited in the future [but we don't know how yet].
Andraz Tori, Zemanta
awesome post ! This is getting real intresting… keeping my fingures crossed.
Let the games (and gaming) begin!!!
though this doesn’t sound like a bad idea, I think this will prompt too much of the linkpost activity and destroy/harm something that made twitter special and will mark the beginning of the end
I don’t know about this. I like things simple. Maybe it’s me trying to honour the initial creation of this, or perhaps I’m just not ready to make this upward move; though I’m thinking that it’s more along the line of this doesn’t seem realistic. Perhaps it’s just me….Well, as I read some of the comments above, maybe it’s not JUST me.
If they are going to filter search by reputation/authority I hope they stay true to the original format as well. Displaying both authority and population search for trending topics!
We’ve been using semantic tools in addition to crawling links to add value to our job search engine for twitter.
Would love to hear any feedback: http://twitterjobsearch.com
This is even more reason that Google will end up backing up the brinks truck to Twitters door. What on earth would they get accomplished at AOL? Even better get one hell of a tax attorney.
they first have to start bastardizing their client devs; then they’ll have something
Spam will be the problem.
Spammers will know the algorithm and will misuse it for their own benefit.
Good move for Twitter. They can only really “win” by changing the game. Real-time changes that.
Does this also mean that Twitter, at some point in the future, is looking at using their search service to generate revenue by serving ads along side search results? Seems like building a robust and sophisticated search engine for their real time data would potentially lead to such a scenario. I find that possibility interesting because it would means that twitter is simply rehashing an existing monetization policy to their benefit and not coming up with a fundamentally new business plan to monetize micro-blogging, as many expect it to do. As long as it works for them they ought to do it but the bigger question would remain unanswered.
the issue is, who would use this kind of search. if i am searching for ’swine flu’ it’s because i heard it somewhere (on twitter). If i’ve never heard of it how am i going to search for it in the first place? a dictionary of trending topics is more important here (twitter should buy one of these startups).
If this ends up being just a list of trending web pages, it will be rather worthless, as superficial sites will make it to the top. spammers will also race to the top in no time
It would be infinitely helpful if Twitter indexed short URL’s as their ultimate end point – or translated the URL on the fly. That way if I searched for ‘techcrunch’ and someone tweeted a link to ‘http://tcrn.ch/1S3′ the search would be intelligent enough to find that tweet. I imagine that indexing the page content would ultimately solve this though, no?
This is a natural (and welcome) result of how people have been using social media: we’d rather know which backup service, restaurant, USB drive or whatever those we’ve connected to prefer than Google. Great example of a company watching what their "customers" are doing and making it easier for them to do.
The interesting thing is I’ve heard this bantered about for FriendFeed to do as well….index all the links (and subsequent conversations) and provide in-line links to “Other discussion on this topic”
Yes Twitter might be all about search. But FriendFeed is going to massively overtake twitter in this regard. They have better architecture, better scalability, and they know what their community is doing, because they participate in it.
Twitter can keep trying to expand services, but they’re going to miss out in the end. They’ve continual failed to attend to their core service of delivering tweets. Without that there’s nothing to index.
did you ever see someone is saying “good night FriendFeed” or “Good morning Facebook”?
On Twitter people do. Twiter impact on people is enormous and many are not aware of that yet.
There is some magical relationship between people and Twitter which can’t be explained.
I would like Twitter to stay independent and take on big guys. It can.
MG, this sounds to me like a much more robust version of DIGG. With the hashtags substituting for the DIGG categories. Thoughts?
This feature will just make more and more spam on twitter. In the long run, the search results will be irrelevant and full of advertising and sales. As a matter of fact, we already have Tweetmeme and Tweetlinks as our links search on twitter. I don’t think this link indexing will be necessary to Twitter at all. Twitter’s business model seems to be even more vague and all over the place. No wonder, the number of active users are decreasing.
Honestly I am not a fan of that… in my opinion we need to keep twitter and website search separate… be way to tempting for spammers.
We do not need another search engine, Google spends millions on optimizing theirs to be the best, why waste time and effort to duplicate something that they will not be as good at.
Unless of course to make it more $$ attractive to amazon (alexa), yahoo or msn
I love Twitter search. I’m a big time ICEROCKET.COM user. I love their built in Twitter search. They even have a blog search and you can ping them after every post.
Does anybody know how much Twitter pays for each outgoing SMS?
“So it should be no surprise when I say that Jayaram was the former VP of Search Quality for Google.” It’s a surprise to me. Two seconds of Googling would have found you this: “Sr. Manager – Google Search Quality, Mountain View.” (http://www.link...com/in/santojay)
This is the best post you’ve written by far.
If they plan on doing this, to me it’s clear they will buy bit.ly: They just switched to bit.ly for links, and bit.ly already stores all the pages people link to (the first step in making them searchable). I bet in a few weeks they announce this.
Twitter can definitely pull this off. It’s brilliant, really. Spam-identification, authority, reputation are all things they have already had to excel at. It will be trivial for them aside from the massive IT infrastructure costs, which also won’t be a problem for them.
i can see it now the twitter seo companies starting up promising to get you to the first page of twitter search. just kiss
This is a good move for Twitter.
Data – tick
General Twitter rage in these comments – boo hiss
Ad Based Business Model – please God no, not another, when I can block ads I don’t see anyway, when will this cease to be a viable business model
Spam – given I get Followed by about 10 fake accounts daily (’we’re sorry the user you’re looking for is no longer here’) I reckon this idea has a big troll like problem.
Lap Dancing – why do I get followed by Texan lapdancers, born again Christians and evangelists, and when did ‘evangelist’ become a job title?
Keep up the good work MG, I admire anyone who’s first name contains no vowel. Might I suggest you put a ‘c’ in between the M and G? McG, like the film director?
A small correction: Jayaram was never the VP of Search Quality for Google.
Twitter should split the screen and drop the idea of having ‘reputation’ anything… just have what they have now, users talking about keywords on the left, and link results on the right.
heck, make it a 3 column – users talking on left, search links in the middle, paid adword type ads on the right.
Wow, a search engine that indexes the world’s biggest chat log.
Maybe AOL should make their AIM searchable? It’ll be just as useful.
Get real, people.
Definitely good move Twitter
“Unless, of course, Google buys Twitter first.”
Shouldn’t Microsoft be thinking about buying Twitter instead of Yahoo? Oh, right, when Microsoft walks outside in the morning, it leaves its brain at home.
Pictures from the TiEcon Tweetup and more, here:
http://blog.tie...tiecon-tweetup/
Am hoping to meet Reid, Biz and Ev at TiEcon next week. Reid’s got a keynote.
I can’t figure out why this is such a big deal.
Are there that many webpages linked to on Twitter that aren’t linked anywhere else on the web? Probably not. So, if the link is on Twitter, it’s probably already in Google’s database (either web or news).
All Twitter’s going to produce is a smaller search database than Google. That’s unlikely to be a serious Google challenger.
No wonder Google wants to purchase twitter, they know the secret…!!