Today, Kapow Technologies has launched a new developer community, OpenKapow, based around their Kapow web-crawling bot. OpenKapow lets anyone use Kapow’s visual IDE (Kapow RoboSuite) to more easily program and share bots that make RSS feeds, REST services, and web clips, which can serve as the backbone for all kinds of mashups. The IDE weighs in at 110MB, so sit tight for a long download. It’s not as sleek as Dapper, with its virtual browser and nontechnical interface, but serves as a good introduction to Kapow’s successful enterprise-level services.
The bot operates much like any other home-grown screen-scrapping bot you would quickly program to grab bits and pieces of pages across the web, but is more flexible and optimized better than bots that rely solely on grabbing html code based on matching text patterns (regular expressions). Instead, Kapow bots follow the DOM structure of a site when grabbing and looping through data.
The IDE’s interface is comprised of three main areas: logical structure of the bot (for loops and all), properties inspector, and an embeded browser you use to direct the bot’s interaction with a web page. Examples of completed bot programs can be found at the community forum page, where all completed bots must be published so that they can be run by Kapow’s servers. Each program can also be downloaded and modified by any user. Look at this NFL sports feed created by one member’s program. With a bit more programming, you can create more interesting mashups based on Kapow data.
Kapow Technologies was founded in Denmark in 1998, with their bot suite originally used to collect the data for the largest marketplace in Europe, Kapow.net. In 2001, Kapow decided to refocus solely on their software. OpenKapow marks a new effort to expose RoboSuite to a wider audience, particularly the mashup crowd. As with Dapper, though, we have yet to see how any copyright issues develop.
Wow, this looks cool I can’t wait to play with it.
Btw, what happened to Marshall Kirkpatrick ? Haven’t seen any posts from him for a while ..
I wonder how long it will take before a black hat SEOer uses this bot for scrapping more content for all those MFA sites
Tim: He left.
Oops, my code in the above post was borked. Try here instead!
Interesting a social network for developer.
A giant leap for automated click fraud !
Site looks great for a beta. I am looking forward to trying it out.
…& it is more like a web scrapper?
rs: It won’t work for clickfraud purposes since it doesn’t parse Javascript. It follows html links, just like Googlebot and others.
I wonder if it’s capable of obeying robots.txt ?
Not quite true Markt, javascript is executed as well by the openkapow robots. And since an openkapow robot isnt like Googelbot it does not take robot.txt into consideration. Another way of thinking of openkapow robots is that it is a way to build an API towards sites that doesnt have an API (using openkapow REST robots) or building an RSS feed from a site that doesnt produce RSS (using openkapow RSS robots).
I was reading your post and believe a product like this should be measured in two ways 1) ease of use and 2) power of functionality. Dapper definitely seems easier to get started on, but openkapow is light-years more powerful. One example of a very complex “mashup” powered by kapow is http://www.momondo.com, that in realtime searches hundreds of travel and airline sites for best fares (similar to kayak.com). Momondo is all powered by Kapow robots, and a product like Dapper could not even get close to their needs.
Looks really cool, but whats status in usage after approx. 2 months.
i think there is n tspace for it