One of the least appreciated, but smartest, moves Yahoo has made in the past year is to launch Yahoo BOSS, its open search APIs which lets developers create their own custom search engine using Yahoo’s algorithms. We use it to power search across the TechCrunch network. And we are not alone. It has become immensely popular.
By last May, Yahoo BOSS was serving up 30 million search queries a day. You can see the rapid growth in search volume in the chart above, which comes from a Technology Review profile of Vik Singh, the 24-year-old engineer who was one of the main champions behind Yahoo BOSS. (He is one of Technology Review’s 2009 Innovators Under 35)
At 30 million queries a day, that comes out to about 900 million queries a month, which would make Yahoo BOSS the fourth largest search engine in the U.S. with about a 6 percent share. That is just below the 9 percent share (and 1.2 billion queries a month) comScore estimates for Bing.
But that is based on data from April. The chart is missing the last three months because Yahoo won’t update the numbers (I asked). Depending on the growth rate of Yahoo BOSS search queries since then, BOSS should now either be at par with Bing or even slightly larger. Between February, 2009 and April, 2009, search queries grew 50 percent (from 20 million a day to 30 million). Here are three different growth scenarios for the period May through July and the corresponding search volume numbers they would imply:
- Growth Scenarios
- 50 percent growth = 45 million queries a day = 1.35 billion queries a month
- 40 percent growth = 42 million queries a day = 1.26 billion queries a month
- 30 percent growth = 39 million queries a day = 1.17 billion queries a month
Remember, Bing was at an estimated 1.21 billion queries a month for the month of July. So BOSS might very well be as big as Bing. In fact, a couple months before the Bing-Yahoo deal, I kept hearing from people connected to Yahoo that BOSS on its own was bigger than Microsoft search. Now Microsoft owns BOSS as part of its deal to take over Yahoo’s search operations.
On the day the deal was announced, Microsoft SVP Yusuf Mehdi told me that he wants to keep BOSS alive because “there is a lot of goodness there.” I’ll say. It adds about another six points to Bing’s overall search volume market share. (Click the market share table below to enlarge).
Update: After some reflection (and sleep), I think it is less likely that BOSS is bigger than Bing. First, as has been noted in comments, the comScore numbers are only for the U.S., while the BOSS numbers are worldwide. So best case scenario (assuming the U.S. accounts for only half of Bing searches, which is the industry norm), BOSS is really only half the size of Bing proper. That is still quite significant.
But there is something else to consider as well. Bing also has an API, which back in March served 3 billion queries. Microsoft tells me that number has increased by more than 50 percent, so Bing’s own unmetered APIs are responsible for many more searches than BOSS, and are also not counted in the comScore data. All of this underlines the larger point here, which is that opening up search to outside developers is a powerful way to gain search query share.
Big difference here, which is always an issue with Techcrunch is you’re using Yahoo’s internal numbers and comparing them to comScore’s, which every engine (and company, including you guys) would say is pretty faulty reasoning.
Not saying BOSS isn’t huge, it is, and is doing *fantastically well*… and not even saying it’s not as big as Bing (which I’m a fan of), just saying you’re drawing conclusions that the data doesn’t support since we don’t know the correlation between internal queries and comScore recorded queries.
This also begs an even more fundamental question though: if all these queries are happening *via* Yahoo, why aren’t they being counted *somewhere* by comScore?
True, that is why I say “Might” in the headline. The data is suggestive and imperfect, but it’s also the best we’ve got.
To the best of my (and Yahoo’s) knowledge, comScore doesn’t count API calls to YAhoo BOSS
Update: Just heard back from comScore: “Yahoo! Boss is NOT included in qSearch data.”
Comscore numbers are domestic US, while BOSS numbers are worldwide. Am i missing something?
No, I missed that. Thanks. I’ve updated the post.
Really, if the data is reliable and if Yahoo Boss is included in qsearch data of comScore, sure Yahoo Boss would be bigger than Bing.
i do agreeeeeeeeeee
Queries could be more. Because, some of the BOSS engines are using caching, and do not sending all the queries to Yahoo all the time.
They could do so much more if they just had some sort of revenue share option. There’s no real way to integrate adsense into BOSS.
I’ve got adsense integrated into BOSS. just do an internal redirect so the keyterms show up in the URL. btw, it’s Google’s problem, not yahoo’s. Google won’t let you feed them keywords for relevance, and Google doesn’t offer us access to the sponsored links as best I can tell.
agreed – comscore search figures comes from a panel of consumers. the data is weighted up to the gross population.
Yahoo do not have a panel. their data is cookie-based. it counts machines not people.
as the 2 methods of data collection are completely different, comparing the 2 sets of figures in this way, without adjustment, is a schoolboy error. the conclusions cited mean nothing.
Don’t claim “BOSS is huge” @30MM queries per day without looking closely at the Google Apis.
At Google IO this past May, Vic Gundotra shared with the crowd that Google is currently serving ~4b API requests per day, and more than half of these are directly attributed to the Google AJAX Search API.
http://twitter....atus/1938211811
Based on this volume, Google’s Search API is ~20-30x larger than BOSS.
Funny that the “Climbing the Charts” source is…..Yahoo.
Erick, you have a short memory. Don’t you remember when Carol said Yahoo was never a search company?
Erick, you have a short memory. Don’t your remember when Arrington told you to trash Yahoo till they capitulate to Microsoft.
Wait, you already helped that happen. Go ahead, Yahoo’s ok now.
Actually, I argued the merits of open search long before Yahoo announced BOSS. I am very consistent on the subject:
http://www.tech...ut-open-google/
I remember .. when she said it was not a search company .. that was ridiculous … it started off with the search business … I thinks she should read the Yahoo history …..
Nokia, the famous and long established wood pulp and rubber products manufacturer, should never have got in to telecommunications. If only they read the Nokia history and stuck to what they were originally founded to do.
Things change and move on. Deal with it.
I’m not aware of Google having anything similar to BOSS; so sites use custom search instead (using url: in the query, or whatever).
Thus, are the figures for Google’s queries/month inclusive of these searches?
Obviously Yahoo won’t be equal to Google even if you sum it with BOSS but I guess it narrows the gap?
Google Custom Search Engine: http://www.goog...e.com/coop/cse/
There is also the Google AJAX Search API:
http://code.goo...pis/ajaxsearch/
And lastly the Google SOAP Search API (which is deprecated but still used by developpers who have got Keys prior to deprecation):
http://code.goo...pis/soapsearch/
It is sad, Google does not allow you to crawl their site, does not allow more than 1000 requests via API, while they get their data from the web, books, AP. This is not give and take..
Laura,
Where did you get the 1000 requests from? i’m pretty sure there is no limit to requests with the Google Ajax Api.
There is, however, a limit of 64 search results.
If I’m not mistaken, beyond the internal/external difference, the BOSS numbers are world-wide, while the Bing numbers are US-only, no? This seems like a significant effect, not everyone is in the US, remember.
Dear Microsoft,
BOSS is one of best API’s on the internet for developers, and a significant competitive advantage you have over Google. Please don’t throw it out. Thanks.
Boss business model?
relevant ads with revenue share. they’ve been promising it for over a year.
maybe one of these days I’ll understand what all the talk about search is? I get that its important, to find what you’re looking for, but its also important to know what you are looking for.
If Yahoo wants to remain relevant, long term…
1) Ditch microsoft
2) Ditch paid inclusion (what idiot thinks it makes sense? Microsoft won’t keep the program alive, either…Qiu Lu, I’d bet money _hates_ it…MS removed it as soon as they could, even while they used YST results).
3) Add revenue share (a decent adsense competitor to BOSS & the Yahoo network)
4) *lower* your ridiculous 10 cent per click cost…
5) Start to see marketshare expand, again.
To start with, Carol isn’t prioritizing or doing any of these things…*fire* her, and then move on with the action plan to make Yahoo a great company…OR watch as she kicks Y! where it hurts and they see their market share, position and scale crumble till there is next to nothing left (remember Iwon.com? Excite? Lycos? Yahoo’s next if they continue the way they are going…)
One of the things that seems to be getting lost in this discussion, is the reason BOSS is so great, namely, Yahoo’s license is extremely open and flexible, much more so than Bing or Google. People mistakenly think its the inconvenience of switching to another API format that is the issue for start ups, or the loss of particular functionality. The former is a few days work, the latter is minimal as most functionality is in alternative APIs. The real problem lies in Bing or Google’s API licenses. For example, Bing’s license is oriented around getting users to make new UIs for their search; this totally eliminates myriad back end uses.
Not exactly the same as BOSS, but Google offers Custom Search Engine at http://www.google.com/cse. Google is free for the foreseeable future, while the BOSS website says they will be pricing the service soon.
Google custom engine is free but not flexible. As Deep Dhillon mentioned above, it is about licensing. BOSS API licensing is pretty flexible. You need not arrange the results in the same order as in the output of the API. Google Custom Engine does not provide that flexibility.
yeah, google is just trying to keep *their* market growing with free search. you can’t make any money with google custom search, whereas BOSS is (Build your Own Search Service). You can build your own search engine on BOSS and monetize it. You can’t do that with Google’s API.
I just tried out google custom search. Yuck, free version has 3 ads in main column, that too at start. My charge on goog always been against main column ads with “sponsored link” written in far corner, that novice users click thinking as results.
The open question is monetization. Give something a comparable product away for free when your competitors charge for it and you are certainly going to grab some share.
Yet, lets be clear without a monetization solution this is unsustainable. Until BOSS can be monetized it was pricing experiment (does free beat paid?). In talking to the guys who are really running BOSS day-in and day-out they are correctly trying to transition this into a business. With the scale of AdCenter+Panama (30% market share) they just may have the monetization capability to make this work. I hope they do.
Nice Information.Good News for all users.
Erick – check out Sphinx Search (http://www.sphinxsearch.com) It’s an amazing open source search engine powering such sites as craigslist.org … Sphinx website claims craigslist alone is making 50 million queries a day. Add to that all the other Sphinx users (many of which are not publicly announcing it).
Jani, Sphinx is just a search engine, same as Lucene. The scale of the search engine is not as much as the web search engine. Sphinx, Lucene, Solr are good for small set of data (say 10s of millions of rows), but not good @ web search scale (say 10s of billions of documents)
Yogish – Sphinx can handle billions of rows with no problem. Of course it is just the backend engine, you need to seed it with data, but it is not true to say it only works for small data sets.
does anyone know of an open index of the web, like BOSS? the data is the valuable part, not the engine.
When I use BOSS I do not have to worry about “spidering” the web…with Sphinx and Lucene I do…
BOSS allows you to build your own search engine with NO YAHOO branding, and without the overhead of “spidering” or even maintaining the index…as far as I am aware there is no comparable offering on web.
Michael,
There is the Google Ajax Api and the Bing Api. But it is true that neither is as “open” to use as BOSS.
Someone please expain WHY BOSS number ARE NOT included in the monthly Comscore numbers for Yahoo… I still haven’t read a single explanation on this…
Vetinary
Yahoo! BOSS might be the best web service in existence today.
So what is the flexibility of the Bing API like in comparison to BOSS? I haven’t checked it out yet. Does it let you re-order results? I do remember reading that it was going to be free for now but eventually there would be some charge to use it based on no. of queries.
Steven,
The problem with the Bing Api is that its TOU doesn’t allow monetization, except through their own ads. But they don’t easily accept you into the revenue-sharing program…
One thing everybody forgets is that its a search API – that is most of the queries can be automated by computer programs / bots / various tools for SEO / datamining etc. This can inflate the figures by a huge amount – I know there might be policies in a way to prevent that but from experience I know its not difficult to overcome these.
The question whether Yahoo Boss remains as a part of MS-Y! deal is still open. What happens to the developers who’ve built their apps on Boss and MS decides to discontinue it. Will Bing API be the only alternative left?
Well, the question whether MS-Y will ever happen is also still open. There are some heavy anti-trust issues here. As a side note, maybe openly stating that they would leave BOSS alive would help them strenghten their claim that competition would not suffer…
Indeed, Janusz is right. Last time I was presented by YahooBoss some of the projects that were relying on it, I would say that about half of them were services wehere the queries were processed in th background as a job (think stuff like matching queries results with domain name expiring to assess value…).
Nonetheless, always got pretty impressed by YahooBOSS, as much from a tech perspective as from the approach they had to it as team: Very open minded, always willing to help and to see how they could contribute to our projects with YahooBoss.
LIKE A BOSS
Like a BOSS!
too many of this things will leave the users confused, however the new development is welcome
The people using Bing and Yahoo are probably almost all computer illiterates who don’t know how to change their homepages. I doubt anyone can overcome Google’s quality, even with people working on BOSS.
You obviously live under a rock!
You still use a homepage? Homepages are for computer illiterates.
This is interesting, if a bit unclear. Can someone detail a project that uses BOSS as underpinning? What were they using before BOSS came along, or was the project not possible pre-BOSS?
The note about Yahoo’s provided figures for BOSS probably being worldwide and the comScore figures being US only are important.
I’ve also asked Yahoo in the past, at least twice, for a breakdown of who exactly is generating what searches on BOSS. IE — is OneRiot the big BOSS customer? Or someone else?
Got nada. Until we get decent figures about what a BOSS “query” actually is and where it’s at, I’m not that excited. And when we get those, be interesting to see how they measure up against Google custom search queries, which comScore probably doesn’t count, either.
I love how little attention is paid to what kind of queries these are. Apples and Oranges, people.
…but it still is beyond me why the company that made VISTA goes out and try to market themselves as making decisions for people…decision engine my ass….
Here’s a good example of the BOSS api in use.
http://www.linguos.com
Linguos handles all the multilingual algorithms and uses BOSS to perform the actual search.
Linguos enables transcription based search of non-English, non-roman languages, enabling access to the non-English web.
Private labeling allows results to be blended into the site much better than Google or Bing APIs.