I suggest others to try this tool, it is truly amazing.

Charlie

> if its not illegal then it obvious(sic) isn’t that bad

Read my point above re: a “keep off the grass” sign. This is analogous to a request for automated clients, or even *certain* automated clients, not to trawl sections of a site. It’s probably not illegal to ignore the keep off the grass sign. It’s certainly not reasonable or ethical to do so. On a scale with murder? No. Ok to do it then? No. For example, as an IT professional I’m a member of certain professional bodies. As a professional and a member, they set forward certain codes of ethics to follow. I do my level best to follow them because it’s the right thing to do, not because I’m worried about going to jail if I don’t.

> The internet is an innovation in itself and bandwith and broadband
> speeds are increasing so what is the point complaining about bandwith.

This is overly simplistic. Bandwidth costs money. The webservers we are talking about are generally not connected to the home broadband connections that you’re referring to in any case, the issue of the end user’s bandwidth usage is another kettle of fish entirely.

> In the next few months technology will improve and MyLiveSearch has
> seem to have found(sic) a new and innovating approach.

There’s nothing innovative about MLS that I can see. In fact it seems to be very similar in operation to the email-scrapers that spammers use to collect addresses that appear on the web in order to pump out viagra spam. These spammers have the same trouble understanding complaints about wasting bandwidth.

> And you claim your(sic) concerned about the internet at large but who
> are you to be the judge?

Hey, don’t shoot the messenger. The arguments I’ve put forward stand by themselves. I could equally well inquire who you are to decide that wasting bandwidth and server load, and ignoring longstanding web conventions is not important.

In any case I’m going to attempt to make this my last commentary here, because it’s all starting to go around in circles. That said, I’ll say it one last time. If the sort of behaviour MLS is undertaking is permitted without the online populace at large taking a stance, not only will “the way we search” change, so will the way we access information *at all*. The internet is only as open as it is based on the goodwill behaviour of all involved in its earliest days. The behaviour of MLS is probably not malicious, but it is abusive and ignorant. If this behaviour becomes the norm, more and more websites will be forced to *require* explicit login to allow access to material, to avoid the server load created by such clients.

This is ignoring the fact that the search results obtained by MLS don’t seem to be qualitatively any better than those obtained by traditional and rigourous indexing and analysis algorithms. Brains are merely replaced by brute force. This is also ignoring the fact that at the moment the MLS plugin appears to be pretty buggy – I mean it’s possible that they’ll hire some qualified software developers and fix that part yet, but for the moment it seems to crash at least firefox pretty regularly.

In all honesty, this thing is likely to suffer a slow, quiet death from simple obscurity, instead of the quick, brutal demise it deserves at the hands of those qualified to comment on its technical merits.

But hey, if you’d like to pretend that the world is out to get you, that there’s a big conspiracy to hold back “threatening technology” instead of tackling any of the points raised head-on in debate, who am I to argue?

This kind of attention as mentioned before would mean only one thing. MyLiveSearch is onto something big.

All of us at work are now using it by default as are most of my friends and will all agree it’s a really cool way to search.

The more we talk about this technology the better it will get, bring it on I say.

Charlie

> It seems that MyLiveSearch does not place anywhere close to the same
> stress that a conventional search engine would place when crawling a
> website. Conventional search engines download images, flash files, pdf
> files which place a huge stress on sites

This is just laughable! Google can index the content of PDF files, and from what you are saying MyLiveSearch cannot. I guess PDF’s are, in MLS terminology, part of the “invisible internet” for an MLS search. On the other hand, for *every single search* MLS generates huge load in webservers by hitting thousands upon thousands of dynamically generated pages all at the same time.

> I have done some close checking with the MyLiveSearch tool and on
> average it only downloads about 3 meg per search not bad for what you
> get, where most search engines today would download 100’s of meg to
> be able to locate pages on websites from which they would have to index.

These “hundreds of megabytes” that Google might download are amortized against potentially millions of searches. MLS on the other hand hits this (if you can be taken at face value) 3Mb for *every single search*. A million MLS searches is then millions of megabytes of data, over a smaller dataset. Not just millions of megabytes of data, but millions of megabytes of the *same* data, over and over and over. Generated at huge cost to server administrators in terms of both server load and bandwidth.

> The Guys of MyLiveSearch have a very interesting approach that only
> needs to look at the raw text of pages or simply just perform normal
> submit searches which would place very little stress on webservers.

Google similarly only analyzes the textual content for text searches. And once again, submitting forms at random is impolite to say the least, and offensive and abusive in many cases. It’s certainly not innovative. I’ve gone into this several times above, so unless you’re actually going to address some of those points I’d rather not have to again.

> Hmm. It seems that the MyLiveSearch Technology is so different and I
> would say very threatening to other search engines.

“Threatening” ‘eh? That’s the same wording this Rob Gabriel used above. Once again, I allege you have an agenda here.

> No wonder it is getting huge attention , Google has 1000s of computers
> by contrast MyLiveSearch does not require all the infrastructure cost but
> simply the technology on your own computer

Yes, Google has a large infrastructure dedicated to analyzing huge volumes of content. On the other hand, MyLiveSearch tries to freeload off client machines without the benefit of a central datastore to avoid duplicative work. I agree, MLS sounds like a sham, doesn’t it?

> Yes the MyLivesearch has hugely different approach which I believe in
> time will get momentum and become a real player. Some people on this
> forum would clearly know the threat the MyLiveSearch has on
> conventional search engines and judging by their response it is very clear
> why they feel like this.

Why do you accuse others of ulterior motives? My concern is for the health of the internet at large, and the unnecessary load that will be generated on my webservers if the MyLiveSearch behaviour ever gets mistaken for being acceptable.

Charlie

Perhaps MLS does *return* more results. From an unindexed blob of data. Retrieved by hammering the living **** out of numerous webservers, not to mention your own PC. Every single time.

Google on the other hand returns the 1000 results judged to be most relevant to the query, based on algorithms designed by competent computer scientists and software engineers with proven backgrounds in both academic and commercials fields. How many *should* be returned? How far down the pile do you go before you say “results past here are too irrelevant to worry about”?

Get back to me after you’ve reviewed those 1000 results one at a time. Give that a try and get your facts straight before looking silly.

Speaking of facts – “invisible pages” ‘eh? No other search engine on the planet *wants* to do this. There’s nothing going on there that Google, or Yahoo, or Microsoft’s live search couldn’t do. There’s nothing innovative, just following links and submitting simple forms. Existing services don’t do it because:
a) they’d get bogged down needlessly in endless cycles of dynamic pages, which MLS seems to do quite happily
b) it’s rude and inconsiderate to do it, especially in light of the server load generated, and doubly so when it’s done by completely ignoring the norms of spidering in terms of Robots exclusion and identifying your user agent appropriately, as discussed above

MyLiveSearch is an *abusive client*. No other search engine on the planet can do this (and expect to get away with it!).

Speaking of propaganda, the smell of astroturf around here is getting a little thick.

Thank you for your humorous post. Are you a representative of myLiveSearch?

I play no game on this blog at all. The results above (Re post #65) were accurate.

Firstly with your regards to your above posts read the following.
hmm,, Try putting a ” ” around them or try going past more then 1000 results in Goolge or any other search engine., Most Conventional search engines will not allow you to search for more then 1000 results,
Try going past the 1000 reusult,

Google replies.
Sorry, Google does not serve more than 1000 results for any query. (You asked for results starting from 1010.)

MyLiveSearch can bring you well 1000 results in seconds , I would have to say that MyLiveSeach returns more results. Fact..
Give that a try and get your facts clear before looking silly..
Not to mention how cool searching is with the invisible pages. Try searching Wikipedia, Ebay and millions of other sites in Real Time . No other search engine on the Planet can do this.

I think your little game is over. Stop making up stuff, you only make yourself look like a fool.

Charlie

How many hits (I’m on Linux & not WinBlows for the plug-in)???

I just did searches on your string of “Fusion HDTV Remote”:

google: 1,340,000
yahoo: 537,000
live (msn): 38,333

Note: higher results does not mean better quality content.

>> But I don’t think it’s unreasonable to expect someone who clearly wants to play a role in the Internet community to learn a little about the conventions that have developed over the past 20+ years

Very good point. These boys claim to have been working on this “concept” for about 8 years, so surely they must know about “the standards”. One may assume that they may be “ignoring them” so they can get more search results.

>> to Hacker in Melb, MLS is not doing anything wrong

Unfortunately aj, you are wrong.

Check for you self.

It’s early days, but I don’t see the point unless it improves on the quality of the major search engines’ results.

Yeah, there have been a few cases of litigation that I recall. I believe these have been more aimed at services which allow viewing of the cached material though – things like Google’s cache pages, or archive.org’s Wayback machine.

As far as the exclusion protocol, there’s a lot of information at:
http://www.robo...c/norobots.html

The conventions AFAIK were developed less for legal/copyright purposes than as a way for the site administrator to indicate areas that were not suitable for a ’spider’ to crawl, because of bandwidth issues, server load, or even just the wishes of the admin, even if the latter became an issue at a later stage. One would hope that most of the industry respects the protocol because it’s an explicit request by the administrator not to have an automated system interact with parts of their site rather than via fear of copyright litigation. A ‘keep off the grass’ sign may or may not be legally binding, most reasonable people will respect it regardless.

> “intertubes” eh??? First time I’ve heard that phrase
http://en.wikip...wiki/Intertubes

> Having said that, more & more webmasters will use server side scripts
> to generate “random verification images”

They will. But is it reasonable to place that burden on administrative staff the world over, when respectful behaviour from the relatively few crawler-operators would make it unnecessary? Again, the CAPTCHA-style images originated in response to abusive agents – spambots and the like – and I’m forced to reiterate that MyLiveSearch seems closer to the spambot end of the continuum than to the Google end.

That said, I suspect that the approach is more misguided than malicious. But I don’t think it’s unreasonable to expect someone who clearly wants to play a role in the Internet community to learn a little about the conventions that have developed over the past 20+ years before unleashing themselves upon the ranks of tireless system administrators the world over (and yeah, that includes me!)

Thks for the input. I seem to remember a few years back (maybe in the 90’s) of reports of pending/filed litigation against the search engines for indexing &/or caching web pages subject to copyright. This is why the industry in general nowadays respects the standards in regards to “noindex,nocache,nofollow” (also the “robots.txt” as you mentioned).

>> it appears that the plugin submits forms at random, presumably hoping
>> that at least one of them is a search form. It doesn’t seem to care if it’s a
>> GET or a POST form. They actually allude to this on their website,
>> something about “indexing the hidden, dynamic intertubes”.
“intertubes” eh??? First time I’ve heard that phrase I can understand the theory behind submitting to what may be expected as a websites internal search engine, but in practice this will definitely generate extra server processor load and bandwidth. Having said that, more & more webmasters will use server side scripts to generate “random verification images” before a search phrase will be accepted (eg: what’s currently done on some publicly accessed blog’s before accepting a post).

That’s right, totally ignored meta tags as well. IANAL, so I won’t comment on the legal issues mentioned, but it certainly strikes me as disingenuous and abusive.

The other one I noticed but forgot to mention, it appears that the plugin submits forms at random, presumably hoping that at least one of them is a search form. It doesn’t seem to care if it’s a GET or a POST form. They actually allude to this on their website, something about “indexing the hidden, dynamic intertubes”.

All in all, MyLiveSearch seems to share at least as much in common with some of the nastier webscrapers around as it does with a search engine. I hope the internet community at large stands up to this sort of unacceptable behaviour.

>> no attempt to fetch robots.txt, or follow the Robots Exclusion Protocol

Therefore it _may_ breach copyright by crawling those sites. Also I’d assume that it doesn’t comply with the “nofollow” attribute etc???

>> the MLS plugin appears to use an MSIE7 user agent string to hide its identity, wilful deception

Hmmm… this may _technically_ be “passing off” (i.e. the legal term)???

To the development team (if it really was them writing above ):
Look, I’m not looking to bag your technology, I do understand that we need better search results, but I don’t think that having @ home pc’s (you don’t think @ work pc’s will allow plugins do you???) spidering is really gonna do it (btw: I can easily do that hacking complex js therefore being truly cross platform (no plugin)).

What’s needed is a better way of filtering the search results from the current top 3 engines (google, yahoo, msn (live)), i.e. using some type of _intelligence_ to give the user a more richer set of results. Nothing wrong with having the client side do a little bit of the work – but you can’t do it by spidering (I can seriously see admins changing their firewall rules to monitor this). If done correctly, you could create a central database of truly relevant results which in itself is worth quite a bit. Hmmm… I think I just gave myself an idea…

PS. Its well known that plugins can cause bad memory leaks in WinBlows.
PPS. What happened to the days of “alpha” before “beta” before “gold”???