Swivel Aims To Become The Internet Archive For Data

December 5th, 2006 at 1:18 am

This will also be the the Internet Archive for Data (rather than just YouTube)

it will be a valuable resources for future generations -seeking snapshots of our cultural trends

December 5th, 2006 at 1:36 am

I think the idea is brilliant. If the execution is as good as I’m hoping, this could really lead to some phenomenal analysis. Stock forecasting, climate change modeling, even diet analysis…

Brilliant.

December 5th, 2006 at 2:11 am

Mike, thanks for the great article. Our investors say, ‘YouTube for data.’ we’re just obsessed with making data exploration fun and insightful. But mostly fun.

Execution, as Andrew Ritchie said, is what keeps us up late at night.

We’re hoping folks who are data freaks help us steer Swivel into something valuable.

-b

December 5th, 2006 at 3:57 am

this is great, I think the way youtube implemented its embeding has opened a lot of peoples eyes.

December 5th, 2006 at 4:28 am

The moment I read this post the idea looked brilliant to me too as it has for the most of you. But as somebody also rightly pointed out above execution and its application is a important thing.

Would like to hear some thoughts on the following possible ‘breakers’ that I see,

1. Most data is typically tied to the ‘environment’ in which it is collected but uploads on this archive will be devoid of this environment. Without this critical piece of the puzzle how valid would the patterns that emerge be - how useful would they be ?

2. Data, whatever it might will have a zillion ways of representing it - both conceptually (graphs, tables, figures, plain text) and in computer formats. In my belief trying to extract any form of patterns must of course nullify the difference in formats but not spread across conceptual data forms.

3. Assuming that this idea really takes of in a big way and people might start basing their ‘decisions and choices’ on this huge internet archive and pattern generator. Is this necessarily a good thing to happen - am doubtful. Yes the collective intelligence of lots of people is probably behind my decision but my creativity is definitely lost !

December 5th, 2006 at 4:53 am

“Academic types are going to go nuts over this” and “Big companies, small companies, thinktanks and non-classified government organizations are going to be similarly dazzled”, are very big statements to make.

As a college students I know that none of my teachers will accept Wikipedia as a valid academic source and I don’t think they would allow this either.

It would seem that this data could easily be skewed by someone or some group with enough time and resources and some end in mind that promotes their own message.

December 5th, 2006 at 5:00 am

This has potential for abuse by ideologues on the Left and the Right. From the Climate Change alarmists to the theory of Evolution deniers, there will be a very strong incentive to load this thing up with bogus data engineered to “prove” whatever pet ideologically-driven point the person wants to ram down the public throat.

Or maybe it will naturally balance out for the most part, in the way Wikipedia seems to be doing for the most part (politically sensitve topics aside).

December 5th, 2006 at 5:49 am

Mike, thank you for the excellent article. We’re getting real close now to taking the wraps off, and seeing all the great comments gets our cylinders firing. We do have a bunch of things to worry about, as Steve points out, but we also know that the folks like you who are passionate about data will help us get it right and bubble the good stuff to the top and weed out junk. We’ll see soon enough. Now back to working the final kinks out!

December 5th, 2006 at 6:21 am

Looks like one day we are in for a comprehensive database for earth and its various data’s and correlations. A complete fit for being called a collaborative web 2.0 Application.

Bravo
http://www.tekno-world.blogspot.com

December 5th, 2006 at 6:26 am

This is great. Sure there will be issues in execution, ensuring data quality, data privacy issues, and more. But this will certainly help in making everyine more dara aware (read Data IQ!).

Yes, if it comes out as advertised, it’s going to be bigger than YouTube and certainly more valuable in improving productivity…

December 5th, 2006 at 6:45 am

Dr. Phil,

I think you’re being too restrictive on who this would appeal to. The emphasis is on the sharing and community aspect of it - not just academics and SMBs. Just imagine what happens when people people on MySpace or their personal blogs want to post embedded data on their page. I could see politically active people doing this a lot with (as stated above) from both idealogies, or sports people wanting to keep track of their own stats (like in high schools), hobbyists tracking things like bird sitings - more people use graphed data than just companies and academics.

As far as abuse goes, that comes with the territory on public and community-driven sites. The comparison to youtube is very accurate - you can opinionate movies just like you opinionate data.

I’m very interested in the slicing and dicing of data - the cross sections could range from interesting to hilarious.

December 5th, 2006 at 6:52 am

For me (a PhD student working in bioinformatics) this sounds potentially very interesting. It would be useful to share data for collaborative research, specially if they make it easy to access the data via APIs. I could even release alongside a publication the means to fully reproduce the calculations via this site. Others could quickly build on the publication with access to the data and analysis. I know this can all be done already locally but this would make it much easier.

December 5th, 2006 at 7:01 am

Hi Hashim,

We kind of agree with you. Spreadsheets are great for editing data. In fact, we mention on our Web site how inspired we were listening to a NerdTV podcast with Dan Bricklin. He boiled down his idea for the spreadsheet as a _word processor_ for data.’ The problem we felt was that after we created a spreadsheet and wanted to switch to sharing and reading mode it wasn’t much fun. Yet you look at the rows and columns and realize there are only so many ways one can pivot and fold this data into graphs. Why isn’t there a Web site that jams out all the combos and then let’s me cruise the data. In one sense that what Swivel is, the read mode of a spreadsheet. Swivel is _Web site_ for data.

December 5th, 2006 at 7:22 am

Like Michael Crichton says: “Science by consensus is not science”. Same goes for data analysis. While this like it might be fun and interesting, I would be wary of anyone drawing any concrete conclusions from any data analysis done by a community of virtual amateurs. Hell we can’t even come up with definitive conclusions on climate even though, “lots of scientists agree”. Same will go for any conclusions attempted to be drawn from this…uh….tool? What sort of things do we expect to get? “We compared the home run ratio of Barry Bonds to the voting record of Barack Osama, and as you can see, clearly there were no WMD in Iraq.

December 5th, 2006 at 7:37 am

Hi Dave,

We’re reading it. Half asleep on our keyboards at this point, but still chasing bugs and checking out TechCrunch. It would be great to get your feedback on this. Here is out current thinking.

We want to make it easy to organize user-driven reviews into two buckets. One is subjective. On the site we represent this with thumbs up or down. The other is objective (or tries to be) and we _will_ represent this with meters.

Subjective stuff is applied to, say, a graph of baseball stats showing the Yankees with more World Series wins than any other team, authored by a guy named Patrick from South Boston. The Title: Yankees Suck! Now, millions of pinstripe fans need to give this graph a big thumbs down. While all the folks from Boston…and Detroit, Cleveland, Oakland…give it a thumbs up. That’s spin and needs to be expressed because…it’s fun.

‘Objective’ stuff will be a bit more involved and will include measures of data accuracy, thoroughness, and attribution. That same Yankees graph with all the subjective thumbs down is also very accurate and would have a high objecive measure.

When we open the doors later this week the subjective stuff will be there, but for the objective parts we’re going to huddle with our founding community members in a few weeks (please join us, Dave) to figure out a way to do the accuracy stuff that engenders trust among data browsers and grows credibility data uploaders.

December 5th, 2006 at 7:51 am

To Hashim:

You are quite possibly correct.

In which case, it would be a perfect sense to have a company that rapidly develops the graphing and explores the best functionality interface and then gets bought by Google, Salesforce or anybody else with massive data sets.

And in a meanwhile, it will allow for quirky consumer uses that being as part of (say) Salesforce from the beginning it would not allow for.

December 5th, 2006 at 8:01 am

The concept is really good. Only I wished I could see it in action.

December 5th, 2006 at 8:30 am

Thank you, Joel:

“I would be stunned if this has nearly the popular appeal of youtube. Uploading videos is something simple and fun…any one with basic video editing skills and a camera can upload videos, and anyone can watch. Data analysis doesn’t seem to have quite the same adoption. I think a lot of people will be able to come up with funny/interesting data sets and correlations, but I don’t know if it will have quite the viral appeal.”

I agree. Are you available to help me set expectations with our investors.

December 5th, 2006 at 8:32 am

Jonah, bring on the Quesadilla. Long live Tufte.

December 5th, 2006 at 8:39 am

JJ Saenz , thanks for the well wishes.

December 5th, 2006 at 8:52 am

To Swivel Team,

This is David from the EditGrid team. I believe there’s something we can explore together. We’ve an online spreadsheet which allows users to collaboratively edit their data.

December 5th, 2006 at 9:18 am

Matthew Leitz,

Great thoughts. Your warning is a good one. It’d be great to get your take after you take a look around.

December 5th, 2006 at 9:23 am

Brilliant! Death to SPSS!

December 5th, 2006 at 9:33 am

most of today’s useful datasets are useful because they are huge. even if swivel would assent to paying the storage costs for multiterabyte datasets, how do you think i should import it?

what data formats do they accept? the data out there is dark matter partly because of the uncountable formats it is stored in.

December 5th, 2006 at 9:40 am

Actually, this seems to be the Myspace for Data. It goes further beyond Myspace and provides good tools and content for the user. The ideal Web 2.0 site.
http://mediavidea.blogspot.com.....thing.html

December 5th, 2006 at 9:57 am

this is groundbreaking technology. I believe all investment banks will stop using excel and proprietary systems and upload all their data to swivel. google will buy them for at least $2 billion. I have a phd in statistics.

December 5th, 2006 at 10:16 am

Swivel sounds like great fun for the average user to play around. I’d play around with it myself.

But there’s no reason why serious data users or data providers would EVER bother with swivel. Major data providers like the U.S. Bureau of Labor Statistics, Census Bureau and the BEA aren’t going to suddenly start placing all their datasets on Swivel. They already have their own websites which allow for the control of and downloading of datasets, many of which are very large. Why pay someone like Swivel to store their data? Furthermore, why would academics or other rearchers who have access to all kinds of on-site disk space bother with swivel either — especially when serious data may involve confidentiality agreements?

SPSS and SAS aren’t dead by a longshot. Most datasets involve considerable work (spatial rearrangement and theoretical considerations) to pound them into some kind of usable shape. Data is NEVER arrayed as needed. There are always issues with how to aggregate the data, scaling, units being used, missing data, miscoded data and a million other things. Swivel isn’t going to do that. Shoot, you have tinker with data to even make a useful spreadsheet — never mind trying to make some kind of useful statistical output.

To think that a person needs to nothing about statistics yet at the press of a keyboard key will have access to all kinds of MEANINGFUL stats is pie in the sky. If only it were that easy. Everyone could skip taking those pesky probability and stats courses in college.

So folks can all have fun with some simple correlations (me too). You can always produce a number, even if it that number no reasonable interpretation. And yes, correlation is definitely not causality.

There will never be a substitute for thinking (aside from artifical intelligence considerations).

December 5th, 2006 at 10:27 am

Wow, another cool idea that some people will be able to make some dangerous and sometimes incorrect observations easier. I wouldn’t be surprised to see the next “Freakanomics”, “Tipping Point”, or “Longest Tail” type book be written based on all information used from information and correlations found within this site and service. Bravo!

December 5th, 2006 at 11:04 am

I want to conratulate the Swivel people just down the street (we’re in Sacramento) on thier launch for a few reasons:

1. Congrats on cross pollinating a web 2.0 look with data sets and making it relavent to the current enviornment.

2. Congrats on your functionality, thats no mean feat, making the data sets robust, appealing and relavent, and then allowing for analysis in a way that allows comparability of disparate data points…..very cool

3. Contrats on getting close to launch, its hard, we’re 1 month away also

4. Congrats on a good showing, that proves how cool, relavent, powerful and important, community generated data sets can be.

I congratulate you on this because I know how hard it is to bring something like this to market, we’re doing something similar in the video games space….

There is a huge opprotunity for business in aggregating data sets, globally aggregated data sets in and of themselves, can become excellent DSS from which one can extrapolate relavent market and trending data as well as predictively model consumer preferances……wow especially where one has comparative data sets…nice job guys.

Allen

December 5th, 2006 at 11:25 am

Finally a novel and very interesting idea. Good luck Swivel!

December 5th, 2006 at 12:24 pm

December 5th, 2006 at 12:37 pm

There it is. #62. The Flying Spaghetti Monster graph. at last.

December 5th, 2006 at 2:02 pm

Well done guys! Looking forward to it!

- You’re old BEA buddy, Simon

December 5th, 2006 at 2:30 pm

I would encourage people to check out Data360. We launched a collaborative data tracking website 2 months ago. http://www.data360.org Best regards, Tom

December 5th, 2006 at 2:51 pm

i prefer to think of it as the youtube of all that is good and plentiful.

offhand, isn’t the internet as a whole one big irreputable source? i personally thrive on it. i quote wikipedia even if it’s wrong and am happy in my ignorance. if i’m really into a subject, i can dig deeper and find out the truth on my own: no data is definitive on any subject no matter the source or binding and everything is a dull, unexciting grey. (props to my main man, global warming)

and vanity still applies to swivel i think, just in a different, nerdier way that isn’t as common in most people. i give respek to milton friedman as much as paris hilton, but definitely in different ways.

December 5th, 2006 at 3:55 pm

BV. All your comments are spot on and balanced. We’ll see how this shakes this out.

-b

December 5th, 2006 at 4:24 pm

Marshall and Reed, our site is up and running. Go ahead and check out http://www.data360.org Best regards, Tom

December 5th, 2006 at 4:37 pm

I see a more mundane utility for people doing work in field biology. It’s hard because everyone works on different things, with data at different granularities, stored in different formats. I’ve got sets of data from 10km transects looking at vascular plants, and 100m transects looking at insects, and 9m square plots looking at pollinators. All in the same area, all done by different biologists.

What happens today is that the data disappears on peoples’ computers. A paper gets published, but the data isn’t shared or shareable. Providing a platform for getting datasets into a single shareable and accessible format is actually a big thing for field biologists. Add some tagging and searching, and people can beging to pull out more meaningful information at a system (ecologically speaking) level than they currently do.

I’ve been collecting and storing and trying to figure out how to deal with disparate biological field data for a while now. Just getting the basics in place has been challenging enough. I hope this service grows and improves.

December 5th, 2006 at 5:36 pm

This is a great idea and hopefully will yield some great insights. I’m glad to see that they have a monetization plan.