One of the hottest areas of search right now is real time search, which attempts to find results based on what is happening right now. Twitter’s search engine fast becoming one of the key ways to navigate the service and discover what people are thinking about any subject at any given moment. Facebook is testing out ways to let you search your personal stream. Google is waking up to the challenge as well (Larry Page is particularly concerned with keeping up).
Every week, it seems, a new startup launches tackling real time search from a different angle. (Collecta, One Riot, Scoopler, Topsy, Almost.at, Tweetmeme, CrowdEye, Omgili, to name a few). They are trying to apply real time search to all the different streams of information flowing over the Internet right now: Twitter, Facebook feeds, Digg submissions, blog comments, RSS feeds, Flickr photos, YouTube uploads, shared links on bit.ly and elsewhere. The list keeps getting longer every day.
There is something about human nature which makes us want to prioritize information by how recent it is, and that is the fundamental appeal of real time search. The difference between real time search and regular search didn’t really crystallize for me until I had a conversation with Edo Segal, who sold his real time search company Relegence to AOL a few years ago and holds three patents on the subject. “Real time taps into consciousness,” says Segal, “search taps into memory. That is why it so potent. You experience the world in real time.”
This raises an interesting dilemma. If real time data streams are akin to the living consciousness of the Web, how do you search them? How do you search consciousness? It is not the same as searching memory, which is what Google does when it looks at its indexed archive of the Web and how those pieces of information build up authority over time. The real time search dilemma centers precisely around how to rank results, and how to resolve the tension between recency and relevancy.
The default, or at least the starting point, for most real time search engines is simply to put the most recent results up top and then keep pushing then down in a free-flowing river of information as new results which match the query come in. That is what Twitter search does, for instance. It is a chronological stream of the most recent Tweets containing a particular set of keywords. Real time search startup Collecta also takes this approach of simply presenting the stream as it comes in, and letting you filter by source. Ranking results any other way would automatically reorder them and automatically make them less real-time.
Yet not being able to filter that stream generates too much noise. Other approaches attempt to add in other factors. OneRiot, for instance, is developing what it calls PulseRank, which takes into account the freshness of the information, the link authority of the Webpage where it is coming from, the authority of the person who is sharing the link, and the velocity with which the information is being passed around the Web. This seems like a reasonable approach, but it may not catch something important as fast as simply watching the unadulterated stream.
There are other approaches as well. You can look at what people on the Web are actually doing in real time or look for variations in the stream of mentions for any given keyword to notice spikes of activity. When everyone is talking about Michael Jackson or Iran above and beyond the normal level of chatter for those topics, that is when you want to know that you need to pay attention. So maybe real time search is more like an alert system.
Can you search consciousness, or can you only watch it pass by? We’ll be debating this at one of the panels on real time search at our Real Time Stream CrunchUp in July. But it is clear that in order to make sense of the stream, it needs to be ranked by order of importance as well as by time.
(Photo credit: Flickr/Andrew Sea)
i would still prefer real-search alongwith the older search…. how about a google page divided into two columns “Normal Search” and “Real-time Search”??? I think this would the best combination….
They are implementing it in another way.. You can sort your search by date.. When next you use Google, click on ’show options’ just before the results.
Try Yauba.
Google needs to focus on real-search not real-time-search. Because when it comes to relevant real-search Bing is kicking their ass. Talk about getting relegated.
I used to think Real time meant in the News. Just go to CNN or NYT
I agree Joe. I have a radio that I listen to while working on my computer or laptop. I get all the breaking news & info that are important in real-time. Also I get to enjoy the depth of coverage they provide. If I want to see the event on TV, then I switch on the telly and watch Fox News or CNN and again, it is real-time with depth. When I want archival information, then I simply use Google & Bing.
Radio or TV did not cover Iran election as good as Twitter., even though credibility is always an issue.
However, I do not understand the amount of investment in the real-time search when it is just a simple API call to Twitter or others. For AAfter search, we want to give credit to Twitter, and just link to them if you have a question that may get an answer from Twitter stream.
But Twitter stream may not always be reliable. Like the time when tweets about Payrick Swayze’s death hit Twitter while he was alive and vacationing. Twitter is good but not always reliable regarding news. Although they did a great job covering the elections and protests in Iran.
Try tweetnews
tweetnews.me
uses twitter as a signal for ranking authoritative yet fresh content so it balances it out
real time resembles our collective working memory, the ’static’ web resembles our collective long-term memory. although both are really semantic memory, as it text we are dealing with. you are misusing the term ‘consciousness’ though, you probably mean ’stream of thought’ or ‘attention’
“So maybe real time search is more like an alert system.”
Yes! And we’ve had alerting for years. We’re also seeing some of the challenges associated with alerting–it’s a lot harder to filter out spam or noise in real time. Indeed, by the the time you can assimilate a stream of “real time” content, others have probably done so to and blogged or written news about it.
More at
http://paidcont...h-is-overrated/
http://thenoisy...for-prime-time/
Daniel,
I don’t think you see it yet Daniel, let me offer another set of metaphors to help illuminate the distinction between what you refer to as “alert systems” and the unfolding frontier of real-time internet search.
The alert systems you refer to looked at an alert and a subscriber telling a publisher they care about something (Pub-Sub). It was a world analogous to a web witout a search engine, people only knew about the few sites they found on the yahoo directory. The potent issue with real-time search is NOT in the mere ability to obtain information about poor Michael Jackson’s demise. It is a porthole for us as a species to tap into what i have always felt was our “collective consciousness”. This is very different from our libraries and the amazing search technologies afforded us by the marvels of Google’s PageRanks watershed moment.
Think about how profoundly google changed the way we all experience life in the wired AND physical world. It has become an amazing tool for us to retrieve information acting as both a discovery engine for things that literally use to take years but also an extension of our memory. It is such a potent extension in fact that the young minds of children entering the school system as we speak are literally going to be wired differently in how they approach solving problems. It has profound long term cognitive development implications for us as a species.
The domain of real-time search is another such leap in our evolution. If standard search is creating our Alexandrian library for the digital age and altering next generations concept of memory then RTS is allowing us to connect as a species in ways never before possible. What is the internet about if not those ever accelerating connections. With both its benefits and minuses (ever shortening attention span).
The ultimate purpose of RTS is to divine insight from the TOTALITY of the human conversation in ways we have only begun to realize in REAL-TIME. We are all one in so many ways, the internet is a testament to that basic human need to connect and the new world of social media, another stage in mapping what we are as a species to the systems we build.
Twitter Trends is just the first shot, watch the space. You can be sure that the smart Ex-Summize people inside Twitter, the insightful team at AOL’s Relegence and companies across the different startups in the twitter eco-sphere will continue to push the envelope in ways we cant yet to imagine.
The relevance of RTS technology does not end with current events. It is part of the larger loop of how insight is gleaned from the infomation, knowledge and wisdom we create and put on the internet. As such as it evolves it will directly feed back into the world of what we know today as standard search.
Our children’s way of experiencing the world in real-time, in the moment, their consciousness is already fundamentally altered from the ever increasing connectivity and streams.
Imagine that every public discussion, post, video, event, are all being analyzed at once by an intelligent system, an evolving AI, take a god’s eye view, think of the opportunities. Its not about getting an alert for a stock you follow or a blog post, its about connecting to humanities collective consciousness as these technologies manifest as a real world vibrant avatar of us all.
Sorry if its a bit religious, but it was my religion for about a decade
I hope it’s helpful in illuminating the issue.
Thanks, Edo.
The collective consciousness stuff is powerful. Maybe calling this stuff search doesn’t do it justice.
You are not so much retrieving information as seeking out like-minded thoughts so you can connect with them. The trick is to do it all in a synchronous fashion or as synchronous as possible so that you can actually have a conversation.
I have a question regarding say Twitter, or a similar mechanism, and your thoughts. I wrote some comments prior to Twitter, that I would like to see an RSS like feed with the ability to tag it with contextual information. Part of the problem in my mind is we have “glean” the relevance and deeper implications of emotional information. DO you see any way to get at that information beyond an interactive process of “tagging?”
Incredibly interesting stuff… I have been reading some books on semantics and as a designer find the implicit meaning, or rhetorical frameworks, to be more interesting than the design itself. Such tools are certainly a ways off, but the implications and base mechanisms will probably be appearing soon on an increasing level.
Edo, you said it correctly when you stated that , Sorry if its a bit religious and your post was more religious than about technology. You used the term collective consciousness religiously. Did you mean the philosophical definition of consciousness or its IT catch phrase?
As I said, in my previous post above, if anyone wants realtime info, then grab a radio (even a headphone portable radio – commercially available) to carry it around. I got far more depth of real-time news about Michael Jackson’s death the other day from radio than any of my friends and family members who use Twitter or search for info on the net. It is a waste of time for anyone to monitor Twitter for more info about the sad death of Michael Jackson, just turn on bloody Fox News or CNN if anyone is interested in real-time news delivery.
I can only speak to what I did yesterday… Namely, I watched the Twitter stream for “michael jackson” unfold. I even added “cnn” at one point in an attempt to *confirm* the story.
What did I need? Perhaps the “michael jackson” stream, in chronological order, on the left. To the right of that, a list of the LINKS that were being passed around in that stream — unrepeated, fully qualified URLs with titles. (Not just the shortened URLs — I kept clicking through to TMZ, by accident.)
This amounts to equal parts consciousness and soon-to-be memory.
FWIW, I stopped watching Twitter the moment someone posted a link to the LA Times where the story was confirmed. In the end, I still needed old media.
This is probably the most interesting piece I’ve ever read on TC. Thanks.
Agreed, its always nice seeing thought provoking pieces like this one.
Let me tell you how this will play out:
You’ll use real time search to know what subject is in the news (MJ, for instance). After you know what the news subject is, you’ll want to tap into the “memory”
This is like watching a live interview with an author about his views on life versus reading a book by the same author about his views on life. There’s plenty of room for both, though the “memory” will win out over the “conscious” – the memory is invariably better organized and written than a stream of consciousness
Yauba already does this …
WANTED: COMBINATION OF REAL TIME (e.g. Twitter) versus ALGORITHMIC (e.g., Google) SEARCH
It’s refreshing to see that, for once, TC is casting a little doubt on the value of pure real time search.
I only work in the hear and now.
Realtime is where we are going like it or not. Already small upstarts are using these services to leapfrog larger companies including stars of the dot.com era.
Realtime is what made many Wall Street types billions and still does. It is also how the exchanges keep an eye on trades. It is how our nation keeps tabs on “evildoers”
It will be the way enterprises and SMBs will grow. Expect to see some astonishing changes over the next few years.
Courtney Benson your example is not relevant. In wall street, you’re dealing with risks where billion dollar trade could be wiped off its paper value in seconds, so the need to get access to real-time info is necessary. When Oprah posts something in Twitter, does it pose risk to investments where billion of dollars are changing hands in the market? The answer is no, there is no risk that involves whether someone want to search in realtime for what Oprah had just posted in her Twitter site or not. Realtime or not realtime to find out what Oprah is upto doesn’t pose any risk to the financial market traders.
Sorry if I did not elaborate enough. I could have said Google – my point was to provide some compass for what real time has offered and to suggest that much more will be done as we move forward
Realtime data is one of the next big thing and a real threat for Google. .tel infrastructure allows you to update in real-time your contact info as an individual or a business. It is probably the first structured information accessible on the web with a realtime update. For more info visit http://www.serv...rg/us/index.php
@courtney benson
.tel is perfect for SMBs who wish to publish real-time info on the web to their audience, improve indexing, and leverage mobile presence.
Mobile surfing requires realtime data as users often make the search according to a specific place and situation.
I do not believe that real-time search matters to mainstream. Not now anyway.
We get enough from CNN etc, crawling entire web in real time is both super hard and pretty useless.
What problem are we really trying to solve here? My sense is – news. And we already have a ton of good solutions, so it seems to me a non-issue.
Re: the question of searching memory vs. consciousness. The terms are being used somewhat loosely, but in the brain it is exact same structures that are used for both, so long or short term memory retrieval mechanisms are the same.
In the brain, the exact same structures are used for everything. Yet different parts of the brain serve completely different purposes. It is how you train these structures that count.
Deep memory is built over time by the same neural pathways being built up and reinforced over time (aka learning), similar to to how link authority is built up over time on the Web.
Consciousness is a different part of the brain. It is a working memory of everything you experience as it happens. Deep memory is more selective and permanent. Consciousness is more fleeting. The two are connected, but they are not the same.
Erik, I agree with Alex Iskold here. Can I ask if all you follow from the Michael Jackson’s death was from the internet (Twitter, Google) or you get mostly the detail from TV or radio? If all your info was from the internet, then I have to say that you waste a chunk of your time in using the net while you could have watched the depth of coverage on TV or on radio? Doesn’t Michael Arrington buy a TV for the lunch room at TechCrunch’s office? If there is one, did you watch the coverage on TV then? If so, why did you watch it on TV instead on wasting your time on typing for various searches on MJ’s death?
Reat Time Search is definitely where it is at. As I blogged some time ago a very viable sub-market would be Real Time Critique and one can imagine betting, gaming with real time search and other apps very easily.
Agree with prev comments best post from TC for a long while.
The announcements of all the real-time search engines didn’t mean much to me, but this little article ties it all together. The fact that your analysis made all the recent news make sense to me proves the point you are making in the post itself. Well done.
Like Technorati didn’t already exist years ago… duh.
Experience in real life is contextual and selective for people. We choose places to go/be (and some times places to avoid) because of our experience preferences. If we are going to make real time search more successful we will need to incorporate automatic personalization. Web 3.0 and all – I know. But it’s particularly relevant inthis context.
I have a bit of an issue with “searching the consciousness” – the Observer Effect – the very act of performing the search changes the results.
If you look at any given trending topic on twitter for example, most of the search results will be a mixture of spam and people asking why people are searching for that topic.
What’s needed is a *reliable* realtime search that isn’t susceptible to being manipulated, and I think we’re a long, long way off from anything that smart.
Agreed on most parts, except the “what’s needed” part, mainly the word “need”.
I can’t imagine a circumstance where I’d need real-time. Only in the event of a ‘home-hitting’ disaster would I want it. Although disasters seem to be what the gov’t is after, giving us crisis after crisis. It’s getting us used to the idea of a crisis which changes our behavior while we rely on borg mentaility to get our opinions about the world.
This is true but it may not be as far off as you think. There are examples of systems that have combatted real time manipulation. One example would be Cloudmark’s SpamNet which employs real time feedback and a trust system.
Great article and following discussion! I really like your quote from Edo Segal, “Real time taps into consciousness” and “search taps into memory.” Tapping into both these dimensions is experience and utilizing our collective experiences to make more informed timely decisions is what TipTop Technologies is all about.
TipTop’s real time search weighs the content by relevancy, prioritizes information by time, and sorts the messages into tip or pit like sentiment (positive-to-do or negative don’t do messages). On top of that, users can filter the content of the query results by concepts embedded in the pool of messages or view the messages in categories they belong to. All this natural language processing and analysis is of course above and beyond just filtering out all the spam or noise found in micro blogging platforms like Twitter. Try TipTop out for yourself at http://www.feeltiptop.com and read more in our blog at http://tiptopte....wordpress.com/.
TipTop is a platform for semantic analysis of natural language which provides live search results capturing individual and group sentiment, opinions, and experiences from content of various sorts including messages on Twitter.
Greg Martin
I’ve got a problem with part of your story there.
“There is something about human nature which makes us want to prioritize information by how recent it is…”
No, that’s the rise of the importance of technology in our lives that has given us all ADD. It has zip to do with ‘human nature’.
No one is doing general-purpose real-time search because being able to do so is predicated on finding a solution to a longstanding problem in computer science. All these “real-time” search companies are either so narrow as to be extremely limited or fakers who don’t understand this type of technology well enough to execute.
In short, we won’t have real-time search until we have a company that is doing awesome computer science that is applicable to far more than real-time search. One of the perennial embarrassments of Silicon Valley is how many companies get funded to do things that are not possible without solving major theoretical problems, apparently oblivious to the fact that this is the case. Venture capital has a pretty poor track record with technical due-diligence when the technology is any more complicated than selling pet food on the Internet. It suggests a distinct lack of substance.
What long-standing problem of computer science are you referring to?
I think only real-time search is Twitter. Topsy, Scoopler, OneRiot etc. are just superstructures
Not always having been a fan of Twitter, since I feel that it could do more, I do currently feel that Twitter is one of the few viable streams of consciousness available at the moment.
THe question is how we search or interact with it, and at the moment we are limited to the textual. “Michael Jackson” and “CNN” allow is to access a more specific stream, but what about context or perhaps images? How do I limit the news to how others feel about growing up with MJ or verification of the death?
Interesting stuff…
This is exactly where TipTop’s search comes in. Look at Greg Martin’s post above. You can see for yourself the answer to you question at http://feeltipt...l%20jackson%22/
The key is not simply to think of consciousness or memory; to me, it is to think of relevance. We are bombarded with millions of stimuli a day; those that we choose to react to generate relevance.
Is there a tool which currently exists which can match real time data flows which aggregate based on user-interaction?
This is a fascinating topic, but from a general business perspective I don’t get the “Twitter thing?” Perhaps I have turned into a Luddite after thirty years of being a geek. For certain folks, for example journalists, I definitely get it; they need to track (and write about) the latest info, that is how they get paid.
Same holds true for “real time search,” some small percentage of people may need from a business perspective, but most won’t. That said, there is probably billions to be made by feeding the curiosity of the masses, apart from any business use.
Great article and comments. Ladies and gentleman I strongly believe the next step is a Global Artificial Intelligence algorithm. With real time search merged and scrubbed against historical and current knowledge. Horn linear logic will be used as a comprehensive logical system capable of handling the typical AI problem of making a plan of the actions in a new search algorithm. Twitter may look silly as well as Facebook, Bebo etc. But they are needed for us to innovate the next step in search. It is similar to learning to walk and logical thought.
> I strongly believe the next step is
> a Global Artificial Intelligence algorithm.
I strongly believe that too. Although there will be no resemblance with existing AI systems. And it is not exactly going to be artificial
I also strongly beleive it’s not about search. It’s not about the memory/consciousness thing either. These technologies and methods are too old and too simple to make a project the next big thing in terms of money making. There’s no problem to be solved there.
> It is similar to learning to walk and
> logical thought.
I beleive the baby needs to learn about giving the appropriate structure to social data. Think entities.
We will hopefully see some projects doing exactly that at the upcoming TC50.
The future of search is a real-time / cached hybrid.
Fun stuff! The Minsh Fish floated by, the TweetWheel turned me around, Sweet Tweets left me in a fog, Tweet3D left me in a row, Twistori left me longing, Just Landed left me grounded, and Social Collider left me warped. Data visualizations of Twitter content is fun, but I prefer actually using TipTop search results to help me move beyond the http://www.feel...visualizations/ to actually be able to use all the great content people share on Twitter.
One real-time search engine that combines the relevancy and recency for ranking real-time streams is feedmil.com