Extracting meaning from the Web is huge project that is very difficult to do at large scale. Keyword search only skims the surface of meaning locked in Web pages. Various semantic search technologies try to go deeper by adding structured data to web pages so that the Web can be treated more like a database. But adding semantic metadata to the Web is laborious and time-consuming. Just look at Twine. It’s approach so far has been to add semantic data only to the Web pages members save to the service.

While it appeared like Twine was finally getting some traction earlier this year, it’s fallen by the wayside. Traffic is way down (see chart below), partly because it is no longer buying traffic with ads and partly because of changes to the way Google indexes the site. Bottom line is that is that beyond a hardcore following of about 250,000, Twine does not have broad appeal.

But CEO Nova Spivack and his team at Twine have been busy working on something else entirely, to the point that the current Twine service is pretty much on autopilot. In the video above, Spivack gives a sneak peak at what his team has been working on. Codenamed T2, it is complete departure from the navel-gazing approach of Twine 1.0. It is a big step towards creating a semantic search engine that might eventually scale across the Web—exactly the kind of swing for the fences type of idea we like to see at TechCrunch.

When T2 launches, hopefully by the end of the year, it will be a demonstration of what semantic search could be. T2 will have a semantic index of the top 50 to 100 sites across major categories such as food, health, sports, music, finance, television, politics, tech and movies. In those categories, T2 should provide really good guided search. If you search for “baseball” you will get a list of baseball players, along with categories on the side to refine the list such as by position or team name. When you type in “thai food,” you can select the Recipes tab and then filter by food site, rating, main ingredient, and so on. Or you can select the restaurant tab and drill down by city, hours of operation, etc.

You’ll find this type of guided search on Bing, with the categories changing based on the initial search term. But Twine does things differently.

What Twine has done, basically, is speed up the rate at which it can look at a raw Web page and create semantic metadata for it. Bing sometimes does this via natural language processing, through the technology it bought with Powerset. That takes a lot of computation. It also employs other methods. Twine’s approach is more to create a set of semantic tags for each page.

There are already standards for doing this, such as RDF and OWL, but most Webmasters don’t bother adding such tags to their sites. If they happen to be there, Twine can read them, but it can also make a good guess as to what is on the page and assign its own tags to the page. In order to try to make it easier for Web developers to tag their sites, Twine is also working on developer tools such as an Ontologies Editor. This lets anyone with domain expertise define the different concepts and tags which would characterize a page about a particular topic, such as a recipe or a baseball player or a car. For example, a recipe might be contain concepts such as ingredients, difficulty level, an author, and a a date. There are literally millions of potential properties that can be matched to different concepts. The collection of all of these together for a specific topic is an ontology.

There can literally be hundreds of thousands of ontologies for every conceivable topic. If Twine knows what ontology to apply to a given Web page, it can do a better job applying semantic tags to it and extracting data. Twine wants to create an open directory of these things, which will be like a SourceForge for ontologies where anyone can contribute and make them better. You can watch this video for more details.

All of this might seem a bit abstract, but if we could ever get to the point where the most important pages on the Web have semantic tags, it will be a lot easier for computers to know what they are about. And to the extent that data is locked in those pages, the Semantic Web will turn that data into something that can be computed. As these tags get applied to more and more information, they could eventually help filter stream data as well that everyone is increasingly drowning in.

Whether or not Twine will be the company to deliver any or all of this is a long shot, but it is definitely something worth swinging for.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

What is the best way to sift through a stream of information? The list view seems to be the most popular because it is information-dense and easy to scan, but it can be overwhelming. More visually appealing ways to manage data are needed. Twine, a site which lets you collect and subscribe to different interest feeds, just introduced a new way to wade through its streams.

The new Flash visualization presents your stream of shared links as a deck of headlines which you can shuffle through (see video below). A slider along the bottom, lets you cycle through the deck by time, and arrows underneath let you move sequentially, or you can just click on a deck in the background to move it forward. If you want to learn more, you can flip each deck to read a snippet and link to the full detail page. The semantic tags associated with each item also show up on the side and can be clicked on to navigate through the deck.

It feels like Coverflow on Apple’s iTunes, except that you navigate through the deck front to back instead of side to side. It also reminds me of the deck metaphor on the new Palm Pre. I can definitely see this as a good UI for mobile apps as well where screen size is more constrained. Don’t worry, though, the regular list view is still an option.

My only problem with the visualization is that it takes a while to load (Flash, why do you torture me?). But other than that, it makes sifting through each feed seem more like channel-surfing in the way that SearchMe does for search results and music search or StumbleVideo does for Web videos. But there must be a better way to wade through the stream. What is it?

Crunch Network: CrunchBase the free database of technology companies, people, and investors

What doesn’t kill you will only make you stronger. Adam Lindemann learned that the hard way with iMindi, a startup trying to create a “thought engine” that was skewered by our judges at last year’s TechCrunch50. “It almost destroyed us,” says Lindemann. But he and his team have completely redesigned the product, which creates a mind map of your thoughts based on semantic indexing technology, and lets you “merge” those thought maps with related ones created by other people.

It is still rough around the edges, but is a vast improvement over the original concept. Today, iMindi is launching in private beta, and we have 1,000 invites for TechCrunch readers (sign up here).

The drubbing iMindi received at TechCrunch50 last year was brutal. After Lindemann’s presentation (see video below), Mark Cuban, who was a judge, laid into him:

Maybe I’m missing something, but that just sounded like the biggest bunch of bullshit I’ve ever heard in my life. You want millions of people to create a virtual decision tree, and create a virtual mind meld, and then get advertisers to mine the virtual mind meld. Why would you want to invest the time?

Lindemann and his co-founder Galen Kaufman were devastated. Investors wanted to pull out. “It was a complete disaster,” says Lindemann, “but it was the best thing that could have happened because they were essentially right. If we would have launched in September, we would have failed.”

The main problem with the original design was that users were expected to manually connect their thought maps to other people’s thought maps. It was very labor intensive, and it wasn’t clear why anyone would invest the time. There are still some UI issues, but the connections are now automated, and it is easier to dump in data from other places on the Web.

Let me take a step back and explain what iMindi is today and what it hopes to become. When you create an account, you are encouraged to sign up for different “think tanks,” which are topic areas of interest. These include “Innovation & Technology,” “Wealth and Finance,” Fashion and Style,”"Travel and Adventure,” Sports,” “Pregnancy and Parenting,” and “Mind, Spirit, and Religion.” You can create your own think tank topics. Once you’ve signed up for a few think tanks, you are ready to enter your “thoughts” on that topic.

You can enter your thoughts directly, just like writing a blog post. Or you can cut and paste from blogs, articles, or other sources on the Web. (Soon it will be possible to ingest your existing thought stream from other sources, such as an RSS feed from your blog or your Twitter feed). Once you you are done entering your thought, iMindi runs it through its semantic index and creates tags for all the major concepts it recognizes. The concepts are hyperlinked in your post and presented as a thought map below. Clicking on any concept reveals other entries you have made with the same semantic tags. If they are truly related, you can “merge” the two thoughts and they will be connected on the mind map.

iMindi is still hit or miss in identifying and mapping the right concepts. For instance, I entered a post I wrote called “Jump Into The Stream” and it produced the mind map at right. It correctly identified the people mentioned in the post, a date, and the fact that the “stream” is a “metaphor” for information consumption related to another metaphor, “the page.” But other than “Yahoo” and “Google,” it failed to identify any of the smaller companies at the center of redefining teh Web in terms of information streams (Twitter, Facebook, FriendFeed, etc.). So its semantic index needs to get better, but it would be easy for iMindi to allow users to add their own tags or edit existing maps.

Where this becomes interesting is that you can also find other people whose posts/thoughts have created the same semantic tags, and merge your mind map with theirs. In this way, iMindi hopes to help you find like-minded people. Once you do find them, you can follow them and their thoughts in the think tanks where your interests overlap. The more people who merge their maps with yours, the greater your “mind rank.” You also can see everyone’s thoughts in a particular think tank as a stream when you explore that think tank. In this sense, iMindi shares an approach with Twine, which is also powered by a semantic index and lets you follow other people’s interest feeds.

Twine is much better funded and at a more mature stage of development. (Twine’s parent company, Radar Networks, has raised $18 million, whereas iMindi was built with only $500,000 so far). But iMindi’s focus on creating these mind melds is promising. We are increasingly drowning in people’s thought streams already (Twitter, Facebook, FreindFeed, you name it). iMindi today is still too much work. But if it follows through and lets you actually ingest these thought streams you are already creating and following elsewhere, it could be a valuable filter.

Imagine being able to map your entire Twitter stream, or the streams of individuals you are following, or just individual Tweets, and mapping those across time. Putting a semantic layer on top of Twitter, or any stream of content is a powerful way to explore related thoughts and concepts. The mind map connects related thoughts not by links, but by analyzing the underlying language used to express those thoughts. In a world of ever-larger information streams, we need better ways to navigate those streams. Semantic mind maps could be one way.

iMindi still has a lot of work to do before we get there. It needs to remove steps from the way it lets you create “thoughts” and link to other thoughts. Personally, I’d lose the think tanks concept or put it into the background. Right now, if you don’t look in the right think tank, you might miss a related thought if somebody entered it into another category. iMindi needs to let its thought engine do even more of the work in connecting thoughts together, or at least exposing them to users. Lindemann is aware of these limitations, and is keeping an open mind about how to make iMindi work. Sign up for the beta and give him your thoughts.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

It turns out that people are following more than just their friends online. Look at the comScore chart above comparing unique visitors in the U.S. to FriendFeed versus Twine. Yeah, I was shocked to see that Twine has more than three times as many unique monthly visitors as FriendFeed (714,000 vs. 188,000). On a worldwide basis, comScore shows FriendFeed still slightly ahead of Twine. ComScore doesn’t always do a great job with small sites, so I checked Compete, which shows Twine with 2.25 million monthly visitors in April versus 998,000 for FriendFeed (see embed below). Different numbers, same story.

While FriendFeed is organized around following feeds of your friends’ activities across the Web, Twine is organized around interest feeds. Essentially, Twine members create topic pages that others can follow. It requires more work than FriendFeed. You have to add items such as links,articles, videos, and notes. But once you do, Twine organizes them for you using an underlying semantic index and tagging technology combined with social inputs from the community. So in a sense it competes more with Mahalo or Squidoo in that it creates authoritative pages around topics, except that these pages are really constantly updated topic or interest feeds that anyone can add to. You can read more about the original concept here, which relaunched publicly in October, 2008. All the growth is from October.

I pinged Nova Spivack, CEO of Radar Networks, the company behind Twine, to ask what’s up. He says that both the Compete and comScore numbers are off, but the trend is right. The initial growth came simply from letting people in who had been on the waiting list. But even he is surprised by the growth rate. So far five million items have been bookmarked in Twine. There are now 200,000 registered users who have created Twines (its name for interest feeds) across 30,000 different interest groups. The rest of the traffic comes from people visiting those topic pages.

And it is not all SEO traffic. Spivack provides the following breakdown of traffic by source: 59 percent comes from people coming directly to Twine, 20 percent comes from search engines, and most of the rest comes from people who receive email notifications and daily digests tracking the interest feeds they’ve signed up for. About 2 percent of traffic comes from twitter, but that portion is “rising fast.”

Following interesting people is just a proxy for following your interests, and Twine lets you connect with like-minded people as well. It is the combination that is killer.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

A year after launching its beta, Twine opened up today to the general public with a completely redesigned site. The relaunch got lots of coverage. Maybe you read some of it. Even if you did, you probably still don’t know what Twine does. Some semantic shit, right?

Exactly. Twine’s marketing department made the video above as a joke for their staff meeting today. (Warning:Turn the volume down, NSFW). In the voiceover, Candice Nobles, director of marketing at Radar Networks (which operates Twine) explains:

You use Twine to collect, find some shit, and share that shit with people you know. Twine ties it all together by topic, so you can have that shit in one place and it is easy for you to find it. You know what I’m sayin’?

I think that is the best explanation I’ve heard yet of what Twine does.

Click here to see the original version of the video.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Radar Networks is the company behind Twine, a site in private beta that helps you organize the Web and your personal information by automatically tagging and cataloging everything you save to it. (For more, see our write-up).

Crunch Network: CrunchBase the free database of technology companies, people, and investors

CEO Nova Spivack took me through a demo. On the surface, Twine is a place to organize information you find or create on the Web—bookmarks, notes, videos, photos,contacts, tasks. (A Web browser plug-in makes it easy to save stuff to your Twine wherever you may find it on the Web). You can also share that information with a private group or publicly. Once you ingest in all the information you want to organize, Twine applies a semantic analysis to it that creates tags for each document or video or photo. The tags match up to concepts that Twine’s algorithms associate with each piece of content, regardless of whether that concept is specifically mentioned in the Web page or other content being tagged. For example, you might bookmark this post and Twine would create tags for all the people mentioned in it (Nova Spivack, Paul Allen, Peter Rip, and Ron Conway). It would also create tags for the organizations related to the post, such as Radar Networks and DARPA, but also Paul Allen’s venture firm Vulcan Capital—even if Vulcan was never mentioned in the post.

What Twine does is automatically generate smart tags and connect them together. There is also a social element. If you share a Twine with others, each piece of content that someone brings into that online space is associated with that person. So when you do a search, the results that come back are influenced not just by the tags, but also by who put the information into the Twine in the first place. “It’s the wisdom of crowds plus the wisdom of computers working together,” says Spivack. The more closely related that person is to you, the higher the relevance. At the same time, Twine is creating a very detailed profile of your interests which it hopes to run highly targeted ads against.

Twine is putting structure onto all of this unstructured data that is out there by analyzing it and adding tags to it that are connected together. The network of links between these tags is something that Spivack calls the “semantic graph,” which includes the “social graph” that is made up only of those tags categorized as people. Bu the semantic graph is bigger than that, comprising other tags such as organizations, places, and other categories.

Rather than create a semantic index of the entire Web, which would be a huge undertaking, Spivack is starting with just those parts of the Web people feel are important enough to save in their collections. Then he applies natural language processing and semantic indexing to just that data. “If you just sucked in the whole Web,” he says, “you would get stuff people didn’t want. Here we are looking at who thought it was important and why.” It’s also cheaper to do it this way, since it’s a more limited set of data that needs to be run through Twine’s semantic engine.

Everything in Twine will become widgetizable and exportable elsewhere. There will also be a full set of APIs. All the data will be able to be taken in and out. Other search engines will be able to index anything in a public Twine, along with the smart tags that have been appended to the information there. “When you put stuff into Twine,” says Spivack, “Twine enriches it, but you can take it out.” Of course, all of those enriched tags will point right back to Twine. “We’re the only place that can even see the connections between things,” says Spivack. Well, not quite yet. People have to start using Twine first.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0