Five Ways to Mix, Rip, and Mash Your Data

Call them pipes, teqlos, dapps, modules, mashups or whatever else but fact is that recently we have seen a good number of new services that allow developers and users to build mini-apps and mashups that mix and re-mix data. Here we run through 5 applications that allow you to mix, rip and mash your data, looking at the data input, output, REST support, suggested use, and required skill level:

mashfeatcomp.png

Yahoo Pipes

pipes200.pngYahoo! Pipes is a GUI web app that lets you create new data feeds by remixing syndication feeds (RSS, Atom, RDF). Pipes takes in feeds from around the web, letting you sort, join, and analyze the feeds items before outputting them in RSS or JSON. It also has a good query builder module that lets you grab feeds based on URL parameters. Yahoo! has also created a community around the service, letting users publish and remix other people’s pipes. The resulting data from the pipes can even be used for other mashups, as Teqlo has done.

Ideal for:Pipes is best suited for mashups between well formed feed data with Yahoo! services such as Search, Local, Flickr, or even Google Base, since the modules are already included. Programming experience is limited to an understanding of procedural programming control structures (loops, logical tests) and aided by the visual interface.

Examples: Apartments near something (Craigslist and Yahoo! Local). eBay Price watch (eBay RSS API).

Teqlo

teqlologo.pngTeqlo is a new widget-based mashup application. You build mashups by dropping specialized widgets onto the canvas and specifying interactions between them. For instance, you can map the results of an eBay search by dropping an eBay search widget and Google map widget on the canvas. Then you connect the two widgets by specifying an interaction such as when an item is selected in the eBay widget, add a marker on the Google map. The application is then accessed by a webpage with the active AJAX widgets. Other widgets include Google Calendar, Gadgets, Spreadsheets, LinkedIn search, DabbleDB search, YouTube viewer, contact lists, and to do lists.

The service is currently in beta, so they have a limited number of modules and have not turned on publishing to the web yet.

Ideal for: Teqlo is a high level masher best suited for non-programmers. Users create interactions between widgets by specifying an action in one widget causing a reaction in another. However, Teqlo’s high level approach means most of its power lies with its developers ability to craft useful widgets and interactions.

Examples: Examples are not public yet, but an example Teqlo is covered in their blog.

Proto

protologo.pngProto is a Windows based mashup application meant to join your desktop apps with the web. You need the Windows application to both create and use the mashups. It’s component based, joining your desktop and web apps by pulling data from your desktop applications, such as Outlook, and feeding it into online web components, such as Yahoo! maps. Proto has the Visual Basic for Applications development environment (VBA IDE) and Adobe Flash baked in, so you can create your own modules to pull and display data from your applications. Proto also has a light database it uses to broken and manipulate data between the application and online component.

Ideal for: Proto takes some familiarity with database concepts and hopefully VBA experience so that you can program your own modules. Their 5 minute intro is indicative of the experience level you need to really use the program. Since Proto allows you to share your mashups, non-programmers can also use Proto for their library of pre-existing mashups.

Examples: The intro video provides a good example of the program, but downloading the viewer is needed to view modules like the restaurant viewer or more enterprise minded Salesforce reporter.

Dapper

dapperDapper is a web based application for generating XML for website content. You create “Dapps” (web services) by using Dapper’s virtual browser to grab content from web pages. Dapper is trained by feeding it several example urls that hold examples of content you’re interested in. Dapper looks at the similarities between the pages to take a guess at the important content to pull from the page. After Dapper has analyzed the page, you can narrow down the fields on the page you want to track. For instance, the titles of stories on Digg. Dapper can then output the content you select from the page in various formats (XML, JSON, HTML, and YAML) and incorporate that data to trigger alerts or even map locations found in the feed. Each Dapper application, “Dapp”, is published to the community for anyone to use.

Ideal for: Dapper takes minimal programming experience and is useful for making well structured feeds for pages that don’t have them already. Their demo movie is a good place to start.

Examples: Fidget is a tool that lets you find videos of your favorite bands based on searches carried out by Dapper.

OpenKapow

openkapowOpenKapow is the industrial strength version of Dapper. It’s a desktop app that programs RSS feeds, REST apps, and web clips through a browser interface. You can use OpenKapow to make a web robot to pull from a web page like Dapper, but can direct that bot to navigate web pages (including form submission), carry out loops, branches, recover from errors, and accept user input at any point in the process. OpenKapow has a community where developers can share their robots to be used and remixed by other users.

Ideal for: OpenKapow is ideal for serious web scrapping. It takes basic knowledge of procedural programming and web markup to use.

Examples: Here is a robot that logs into Gmail and outputs your email in XML. Here’s another one that searches deep into TechCrunch’s posts for a keyword.