Since its inception, one of the biggest problems with Digg has been that users often submit the same content over and over again. This makes it harder for cool content to become popular because some users digg one submitted story, while some digg another. Today, Digg is releasing “several major updates” to its duplicate (known as a “dupe”) detection system.

The solution sounds fairly intensive. “To better understand the nature of the problem, we analyzed the types of duplicate stories being submitted. Most common are the same stories from the same site, but with different URLs. Our R&D team came up with a solution that identifies these types of duplicates by using a document similarity algorithm,” Digg’s Director of Product Chris Howard writes in a blog post. He goes on to say that there will be a follow-up more technical post to explain a bit more about how this actually works, but says that it has proven to be a reliable system so far.

URL shortener and analytics service Bit.ly has been working on a new set of products, being referred to as “Bit.ly Now” internally, which will define the next stage of the company’s growth. The company confirmed these plans to us today. The services will include both a destination website as well as a distributed service via expansions to the Bit.ly API.

The core Bit.ly service, which lets users shorten web URLs into something suitable for Twitter and other services with limits on characters per post, has continued to grow quickly. 7 million URLs are shortened via the service each day, the company says, and 2-3 million of those are unique URLs Bit.ly has not seen before. Those Bit.ly URLs are clicked on 150 million times per week across a wide range of services - Twitter, Facebook, instant messaging, email, etc. Twitter itself now uses Bit.ly for URl shortening, and the service has quickly taken the lead in their market.

The magic behind Bit.ly are the stats that the service makes available on the underlying domains being clicked. Investor John Borthwick explained it all to investors in an email we obtained earlier this month:

Late last year we wrote about an experimental advertising product that Digg was developing:

One experiment Digg is working on, says one source close to the company, is a self service advertising product that will be somewhat similar to Google Adwords, but with a twist. The product would insert advertisements into the Digg news stream (presumably clearly marked). Where those ads end up, and how much an advertiser pays per click, would be based on user feedback.

So users would have the ability to vote on advertisements in the same way they vote on stories. The better ads, as determined by Digg users, will get more prominent placement and a lower cost-per-click.

Compare that to the blog post from Digg a few minutes ago announcing a new advertising product:

High profile advertising network Federated Media’s Chief Revenue Officer Chas Edwards has resigned, we’ve confirmed, and will shortly be taking a job at Digg with the same title. Thomas Shin, who Digg stole from Yahoo earlier this year, will report to Edwards.

Mike Maser, currently Digg’s Chief Revenue and Strategy Officer, will change his title to Chief Strategy Officer. He controls Digg’s marketing, business development, corporate development and community management groups.

While OpenTable had a very successful IPO yesterday, the IPO market might not open up for other tech startups for at least another year, says prolific Silicon Valley investor Ron Conway. He made the assessment in a video interview with Vator.tv’s Bambi Francisco on mobile startups, republished below.

Questioned on when he sees the IPO market open up again, Conway responds that he thinks we are at least one year away. He’s more bullish on the M&A market picking up again, expressing the hope that this is only six months away.

How quickly they grow. Remember when Twitter was just a little pipsqueek, with less than 10 million monthly unique visitors to its site worldwide? That was back in February, 2009. Fast-forward to April, and Twitter’s U.S. visitors alone reached 17 million. Now comScore has released its worldwide numbers and it estimates Twitter’s global unique visitors in April, 2009 was a whopping 32 million, up from 19 million in March, 2009.

To put that in growth into perspective, Twitter has just passed Digg (23 million), LinkedIn (16 million), and the NYTimes.com (17.5 million) in monthly unique visitors, as counted by comScore. And comScore only measures the number of people who visit Twitter’s Website, not the millions more who send and read tweets via their phones, desktop apps, or other Websites. Twitter.com is also now bigger than Bebo and Freindster, for what it is worth. Who will it pass next?

Like everyone else, Digg has a serious case of Twitter envy. And they’re doing something about it.

In an interview last month Digg founder Kevin Rose told me that the company was working on an overhaul of the Digg service, calling it a “completely new direction” and referring to the new Digg search as an indication of what direction they’re going in. He didn’t (and still won’t) give many details, except to say that Digg needs to to “a living and breathing site” and “a little bit more real-time in nature” (”real-time” is secret code for Twitter, and has been used so much lately that people are mocking it).

Earlier in the interview Rose talked about wanting more user participation on the site, with top stories getting 50,000 or more votes (most top stories get a few thousand now). The new product is designed to encourage more user engagement.

We’ve taken the relevant clips from the original interview below. Whatever it is, it’s coming sometime soon.

There’s a popular story on Digg right now about a 104-year-old British woman who uses Twitter. It’s an obvious headline: “World’s oldest Tweeter talks cuppas and casserole.” It’s Digg bait. But what’s worse is that if you examine the story closely, it looks like the UK press may have gotten the poor old lady to sign up for Twitter just for their story!

The story is about how Ivy Bean uses the hot social network to post mundane updates about her 104-year-old life. But take a look at the picture in the story. On the screen next to Bean, you’ll see her Twitter page with a whopping two updates. These two tweets were sent out at the same time, the day before the story ran in a number of UK publications. In other words, Bean signed up and sent her first two tweets at the time all these guys were writing their stories. Or, to put it more clearly, this whole story was staged.

Digg has done a lot of talking about integrating Facebook Connect, but not much in the way of actual implementing. That changes today, we’ve heard from a source close to the company, and Digg will go live with the service some time today. Facebook gets another big name partner on its increasingly popular Facebook Connect platform.

Digg CEO Jay Adelson has suggested that Facebook Connect is the future of Digg, and that Facebook’s massive userbase will both spur Digg usage and provide a collaborative filter on stories.

With Facebook Connect users will be able to cross-post, share Digg activities on Facebook and auto-follow Facebook friends.

The future of media is algorithmic aggregation, at least that is the approach Future US is taking. The U.S. subsidiary of Future PLC, and publisher of such niche magazines as Nintendo Power, Guitar World, and Maximum PC, is adopting a different approach online than simply putting its print articles on the Web. Instead, it has launched dozens of news aggregation and discovery sites called “Blips” that are a combination of Techmeme and Digg. The Blips collect topical stories from across the Web and present the headlines in discussion clusters like you’d see on Techmeme, but stories can also then be voted up the page like on Digg.

There are about 40 different Blips on various topics, including TechBlips, EarthBlips, and WrestlingBlips. All of them are accessible from Future’s online portal, DailyRadar (which also houses the magazine content under games, music, tech, entertainment, and sports tags). Future has been launching Blips quietly since last summer, and they now account for 9.3 million of DailyRadar’s 27 million monthly unique visitors (which is up from about 12 million uniques a year ago before the Blips sites were added).

Mixx, a Digg-like site that lets users vote to push news stories and other bookmarked content to the home page, is experimenting with an innovative new advertising platform called Mixx Sifter. Frankly, and we’ve written this before with other Mixx features, this is something Digg should have done.

The idea is to get the Mixx community to give direct feedback on advertising. The better ads will get impressions - not based on how much is paid, but simply on how positive the feedback is from users. Last year we’d heard that Digg was thinking of doing something along these lines. Digg founder Kevin Rose refused to speculate on it, though, in a recent interview.

Here’s how Mixx Sifter works - an advertiser uploads five different ads, in virtually any format. Mixx then invites its power users (elite Mixx users who have spent hundreds of hours on the site) to review those ads, rate them and provide direct private feedback. The users get karma points and a chance to win a gift certificate or computer in return. The most popular ad unit is then run on the site.

Last week I sat down with Digg founder Kevin Rose to get the Digg “State of the Union.” The company, now more than four years old, has continued to grow incredibly since launching in late 2004 and regularly innovates with new products. They are rumored to be approaching profitability after making headcount cuts earlier this year. But much of the hype around Digg is now in the past. From 2006 onwards there were regular rumors of Digg being acquired. So many, in fact, that we called on CEO Jay Adelson to “Just sell Digg already” in late 2007.

And Jay certainly tried to. They were very (so very) close to closing a deal with Google in mid 2008, but the deal fell apart at the 11th hour. Digg was quite literally left standing at the altar. And we believe that the company made the decision at that point to stop looking for buyers and to focus quietly on the core business for a long while. They raised more money late last year.

I talk to Kevin about the history of Digg for about half of the 30 minute interview. There’s a lot of good content about the early attempts to buy Digg, since it’s far enough in the past that he feels comfortable talking about it. But the forward looking stuff is easily the most interesting.

I asked Kevin if he felt that any of the numerous competitors (AOL Propeller, Yahoo Buzz, Mixx, Reddit, Hacker News) were a true threat to Digg. He shook his head no before I even finished the question (skip to the 29 minute mark), and said something surprising: “I just feel that we’re heading in a different direction than them.” When pressed for more details, he says that they’re launching a product to address the real-time threat that Twitter, Facebook and others are focused on, and that the new Digg search was the first hint of what they’re thinking of.

Click through for the video and full transcript.

Digg is putting an end to its exclusive ad selling relationship with Microsoft after two years, which is one year earlier than the deal was originally set to expire. Starting this Summer, the social news service will begin to rely heavily on its own internal sales force, which will be responsible for selling the majority of its ad inventory, reports Clickz.

Microsoft gets the leftovers, i.e. remnant inventory.

The partnership between Digg and Microsoft was initially supposed to last until mid-2010, but according to Mike Maser, Digg’s chief revenue and strategy officer, the two always had an understanding that Digg would at some point start selling the majority of its own ads. He added that the company’s internal sales efforts will focus on custom, non-IAB inventory combined with standardized banner ads.

URL shortening services are a dime a dozen, and despite wishes for them to vanish (with good reason) they’re here to stay and more popular than ever given the abundance of social services that thrive on short messages and links. TinyURL and bit.ly appear to be the more popular of the bunch, but we’re seeing other services use their own custom URL shortening services at an increasing rate. To name but a few, Digg uses its top domain for the DiggBar and associated links, Posterous uses post.ly to trim down links when they distribute them to other networks and Twitter toolbox HootSuite uses ow.ly.

Update: shame on me for forgetting that TechCrunch also has its ‘own’ short URL (tcrn.ch), which you can view under every post and which we use for our Twitter account. We use awe.sm for this.

The latest startup to add a URL shortening feature to its service is UnHub, and we’re also taking a look at the new Lnk.by.

The DiggBar, Digg’s browser-based toolbar for digging and sharing content, has seen a tidal wave of controversy since its release last week. The problem for many site publishers is that it frames content within the DiggBar itself, meaning that users aren’t actually on your site, but are looking at a frame of your site while staying on Digg’s servers. And the DiggBar also acts as a URL shortener, but does so using a 200 code rather than the preferred 301 permanent redirect that other shorteners use to ensure the destination sites get the traffic. Now Digg is getting ready to slightly compromise.

Digg’s VP of engineering John Quinn, writes in a post today that Digg will change the behavior of shortened URLs. For anyone not logged into Digg, they will now get a 301 redirect to the site being pointed to sans the DiggBar. Users who are logged into Digg and have not opted-out of using the DiggBar, will still get the 200 variety, meaning they will technically reside on Digg’s servers.

The new Digg search, which we wrote about last week with a screenshot, just went live. Small feature change, but it’ll have a big impact on Digg usability. From our earlier post:

Digg may have 19 million unique visitors and nearly 85 million pageviews (Comscore worldwide, Feb ‘09), but one thing it’s never had is a decent search engine. There’s no rhyme or reason to the ordering of results. A lot of stuff just seems to disappear. And the filters are overly complicated.

That’ll change in the near future, the company noted in a blog post today. And since we’re so curious about what the new Digg search will look like, we hammered them until they gave up a screenshot.

The new search appears to have much more relevant results, taking into account the popularity of the story as well as the newness for “best match.” Other simple filters include “most dugg” and “newest first.” Compare that to the very old, not very relevant results you get now.

The new search also has a left sidebar with key stats, such as aging, how many stories with a certain number of Diggs, stories broken out by media type (news, video, images), and by topic.

In the debate over whether or not URL shorteners are evil, one service in particular has been singled out: Digg’s new Diggbar. The Diggbar is more than just a URL shortener, but that is one of its main features. Unlike other shorteners, however, which redirect people back to the original link being abbreviated (often for Tweeting purposes), the Diggbar directs traffic back to a Digg.com.

The concern this raises is that if a lot of people start using the Diggbar and its special shortened URLs, Digg will effectively be stealing link juice from the original sites being linked to, which won’t get the proper credit they deserve from search engines. In technical terms, the Diggbar produces a 200 server code instead of a 301 redirect (Danny Sullivan explains the difference here), and on the surface that just does not seem kosher.

But in a post today, Digg VP of engineering John Quinn promises everyone that Digg is not trying to steal any link juice and has taken extra measures to make sure search engines and others credit the original links. He writes:

DiggBar, the new shortURL and toolbar service from Digg, is certainly useful. I expect it to become my default short URL service on Twitter since it is so easy to create a short URL by simply adding Digg.com/ in front of any URL. It will redirect to a short Digg URL like digg.com/d1npNz, which is this story rendered with the DiggBar (click that Digg button!).

The Twitter crowd already loves this based on all the chatter I saw on the site today. This will also expose a lot of new people to Digg since anyone that clicks on the link will see the toolbar wrapper with the view count, Digg comments and other information on the top. And it will also increase Digg’s overall traffic substantially - unlike other short URL services, Digg doesn’t simply redirect to the longer URL. It keeps you on Digg and shows the site being pointed to in an iframe wrapper. You can get to the underlying URL by clicking on the X button on the top right.

But Digg didn’t stop there. They’re also using DiggBar for all stories on Digg as well. So all those home page stories that send massive amounts of traffic around the web are now redirecting right back to Digg, too. That keeps all that traffic in the Digg ecosystem, to the detriment of the sites being linked to.

Digg may have 19 million unique visitors and nearly 85 million pageviews (Comscore worldwide, Feb ‘09), but one thing it’s never had is a decent search engine. There’s no rhyme or reason to the ordering of results. A lot of stuff just seems to disappear. And the filters are overly complicated.

That’ll change in the near future, the company noted in a blog post today. And since we’re so curious about what the new Digg search will look like, we hammered them until they gave up a screenshot.

The new search appears to have much more relevant results, taking into account the popularity of the story as well as the newness for “best match.” Other simple filters include “most dugg” and “newest first.” Compare that to the very old, not very relevant results you get now.

The new search also has a left sidebar with key stats, such as aging, how many stories with a certain number of Diggs, stories broken out by media type (news, video, images), and by topic.

Last month we wrote about the new Digg toolbar product called DiggBar. It launched today. If you want to try it out immediately, just add “digg.com/” before any URL at all (the image above is using the techcrunch domain).

The toolbar is made for the new Twitter generation, which has a desperate need for very short URLs to fit in the tiny space allotted per message. Digg is going to be a popular player in this space because of the easy way to create the URLs (just pop digg.com in front of anything), as well as the stats that they provide in the toolbar wrapper: number of views, comments on the story, and related stories.

StumbleUpon has their own product called su.pr coming shortly as well.

Here’s the official post from Digg on the product, and a demo video: