Amazon: Grid Storage Web Service Launches

Trackbacks/Pings (Trackback URL)

1. Nik Cubrilovic
   

   March 14th, 2006 at 12:15 am

   Mike you can’t compare this to what your #1 request was, or the other services that have been profiled on Techcrunch because what Amazon have released is nothing more than a hard drive with an API at a good price - you can use it to send it your data but only if you are willing to craft your own REST requests

   This competes with Openomy (which gives developers 1GB for free), and is a developer service not a consumer service. Apples to Oranges etc.

2. kris
   

   March 14th, 2006 at 12:17 am

   Great news! This makes it possible for the little guys to compete with the big guys again without as much up front funding to purchase all the hardware.

3. Michael Arrington
   

   March 14th, 2006 at 12:18 am

   Ok, but finally the pricing is moving towards what it needs to be…

4. Nik Cubrilovic
   

   March 14th, 2006 at 12:25 am

   Ye Mike and that is what is going to destroy 90% of the companies you profiled, is that they didn’t even think about server-side storage and what is involved (costs, scaling etc.)

   Hey a service like this that also has client interfaces for all platforms that are easy to use would be really cool

5. Andy
   

   March 14th, 2006 at 12:27 am

   No business is going to want the core of their infrastructure for data-heavy applications to be reliant on a 3rd party interface like this.

   So to say that flickr, box.net etc. would be best off using S3, is just a wild statement right now.

   Supposing they turn it off?
   Supposing they jack the prices up?
   Supposing their service sucks.

   When part of your core mission is storing lots of data reliably (flickr, box.net, etc.) there is no way on earth you’re gonna outsource it.

   The pricing, I would guess, reflects that this is not a consumer facing service.

6. Michael Arrington
   

   March 14th, 2006 at 12:32 am

   Andy, sure, but there are services out there that just can’t be built (by the people who want to build them) without something like this.

   And I love the downward pressure on pricing.

7. Andy
   

   March 14th, 2006 at 12:38 am

   Agreed, it will certainly be interesting to see who uses this.

   On a similar subject, has anything been built which uses the Amazon/Alexa search engine repository thing yet?

   http://websearch.alexa.com

8. Michael Arrington
   

   March 14th, 2006 at 12:38 am

   nothing that I’ve seen so far. alexaholic?

9. Joe Drumgoole
   

   March 14th, 2006 at 12:50 am

   Aren’t users going to blanch at the unpredictability of the costs? See my pingback above for concerns. If they can address this then I think as Nik says its sayanora to some of the current players. I see Ibackup and Xdrive really hurting because their business models are built on a unsustainable price structure.

10. Rajat Paharia
    

    March 14th, 2006 at 2:30 am

    I looked through the API. From what I saw (and didn’t see) I think it would be difficult to create any service on S3 that had multiple user accounts. There is no way to track storage or bandwidth usage on a granular level (or even on a high level) from the API. So you would have no way of monitoring or limiting an individual’s usage.

11. Aaron Levie
    

    March 14th, 2006 at 3:08 am

    I’m not sure what kind of splash this will really have, except now a web developer can host things on Amazon cheaper than they can on 1and1.com. And yes, to Andy’s comment, building your business off of a web services API is just asking for trouble. Nothing wrong with this offering, just not really a box (or google) killer in its current state.

12. Gaurav
    

    March 14th, 2006 at 3:22 am

    I think Amazon would have to really convince some of the up-and-coming (or even better some already established) services to get on to S3.

    The way I look at it right now, the post and the comments together present the picture well:

    - Product with game-changing qualities; - but core component (infrastructure) needs to be shifted externally;
    - definite downward pressure in pricing though no provider will guarantee that in the agreement;
    - might be too messy to build it this way but some new services just needed something like this.

    Good. This space, as Nik’s and Mike’s discussions through the comments show :), just got more interesting.

13. Daniel Feygin
    

    March 14th, 2006 at 4:23 am

    I do not think the pricing is competitive at all — any site that begins to take off is going to find that its hardware investment would produce a financial return rather quickly while also giving a tremendous performance boost. Nor is the API particularly suitable to business applications, which are way better off on salesforce.com. It is further ironic that salesforce leapfrogged Amazon (with AppExchange) in its role as the third-party developers’ preferred go-to-market platform, which is where the real value is.

14. Vlad
    

    March 14th, 2006 at 6:03 am

    I don’t get it. It may be cheap to store a file there, but remember - you need to retreive too (in many cases).

    So you pay every time you access that data.

    It doesn’t look like they offer web hosting, so you cannot even use it for images.

    So you pay for S3 storage, data transfers and YOUR internet connection too. Add latency and I don’t see any reason how this could be used.

    For example box.net if they would use this service. You upload files to box.net let’s say 1GB. They transfer it to S3 and store it there. But then you access file and they download it again and send it to you. So they need at least 3-4 more bandwidth. Is this correct? More often you retreive file, more box.net pays.

15. Aaron Levie
    

    March 14th, 2006 at 6:25 am

    Vlad,

    Yes, you bring up an interesting point about the bandwidth. Files need to pass through our servers before they are stored or distributed back to the user. This would make our bandwidth costs double (for each direction). We would be paying for Amazon’s cost, and our own. So if you wanted to store and retrieve 1GB in a month, the net effect is .40 (Amazon’s BW) +.15 (Amazon’s storage) +our BW and server costs.

    Now consider sharing, where the average users needs around 10GB transfer/mo for photos/clips/files, and we end up paying $2.20+.15+our BW and server costs.

    This solution really does not scale as well as it would seem, at least not for an online storage provider.

16. Colin Faulkingham
    

    March 14th, 2006 at 6:30 am

    I think the real killer app here is BitTorrent. Which, S3 has built in support for.

    Now that’s cool!

    See,
    http://noisemore.wordpress.com.....t-support/

17. Nik Cubrilovic
    

    March 14th, 2006 at 6:37 am

    Vlad and Aaron, the way we (Omnidrive) take this approach with out API is that the client machine (as in the integrators client) is passed back to the Omnidrive servers to download content

18. Roger Strickland
    

    March 14th, 2006 at 6:40 am

    This is a very interesting development and somthing I might start using right away. I’ve got pricing under .10/GB for transfer right now using my own “poor man’s CDN”. But it is hard to scale my stuff.

    I’d love to hear comments about what you think this does to Akamai. My quess is that AKAM will soon be under extreme pressure to lower prices as their high-margin niche becomes a commodity.

19. Hoss
    

    March 14th, 2006 at 7:10 am

    This service could be tremendous for companies working in the online backup space - they need large storage but typically not that much in bandwidth - why pay $10/month for 5GB for xdrive when you can pay $0.75 to Amazon?

    I expect you’ll see dozens of new solutions using Amazon’s new service in the coming months …

20. Ernie Oporto
    

    March 14th, 2006 at 7:21 am

    The first thing I wonder about is corporate use. I could see backing up an encrypted file from individual PCs to this, but it needs to be grouped and billed centrally. $0.15/GB-month plus the $0.20 to transfer it to them is really cheap!

21. Ben
    

    March 14th, 2006 at 8:05 am

    Wow, this is a lot of validation of the space.

    The pricing is remarkable. At 0.15 per GB, you’re looking at 8+ year amortization schedules on equivalent raw storage gear purchased from EMC (half that for EqualLogic, LeftHand, and other iSCSI vendors). At .20 per month per GB, you’re in the high end of hosting market but nowhere near the approximate .003 effective cost on going with a scaled 95% arrangement with a Tier 1 bandwidth provider.

    I’ll reserve judgment until actually testing the service but it seems the “gang” is going to have to get more aggressive…

22. Brad
    

    March 14th, 2006 at 8:54 am

    With this pricing I never need to buy another “backup” hard-drive. And for dumb backups, middle-man companies are exactly what are NOT needed. I expect a simple client application to be up on sourceforge in a few weeks, tops. FTP clients could also add S3 access as a feature.

    Rajat/#17: A company that front-ends this service wouldn’t have to pass user files straight-through to S3, they could mix user id’s and other metadata into their backend streams. The API is not really limiting.

23. pwb
    

    March 14th, 2006 at 9:39 am

    I don’t think Amazon is envisioning this as yet another net drive. If you look at it in conjunction with things like Simple Queue and Mturk, Amazon is laying some nice groundwork for multi-party web services.

24. gt
    

    March 14th, 2006 at 10:31 am

    You may make calls at any time that the Amazon Web Services are available, provided that you either: (i) do not exceed 1 call per second per IP address, or send files greater than 40K; or (ii) do not exceed the limits set forth in the Service Terms for a particular Service. If you build and release an Application, the stated limitations apply to each installed copy of the Application.

25. Roy
    

    March 14th, 2006 at 11:47 am

    @42: Read further down the TOS:

    B2) The limitation of 1 call/per second/per IP address set forth in Section 1.A.2 above is not applicable to your use of Amazon S3. You may not, however, store “objects” (as described in the user documentation) that contain more than 5 Gigabytes of data, or own more than 100 “buckets” (as described in the user documentation) at any one time.

    —-
    I run a semi-small community that allows free image uploading. S3 is perfect for this; although Amazon’s pricing for bandwidth is a bit high ($0.20 GB/month is high compared to the $0.10 GB/month I can get from a lower-qualitied service like Cogent, although it’s high latency), the value is that I no longer have to maintain my own backup services.

    The cost of maintaining backup services (with RAID-0, which still doesn’t protect against human deletion errors) on my own is quite high - I have to invest in the hardware, the software, and the time to make things work together seamlessly.

    Will this be adopted by larger sites? Probably not; they have the resources to throw at creating a scalable, redundant backup solution.

    This is going to primarily appeal to smaller businesses which want to keep their fixed costs low and don’t want to have to deal with hiring a sysadmin to handle an expensive network.

    The only dissappointment so far has been the lack of “sample code” from Amazon - why not release a full codebase in the popular languages (Ruby, PHP, Perl, ASP) when launching the site? I know I would have implemented this tonight if I had code for it already.

26. Colin Faulkingham
    

    March 14th, 2006 at 12:11 pm

    Roy there are tons of code samples at,

    http://developer.amazonwebserv.....egoryID=47

27. Roy
    

    March 14th, 2006 at 12:31 pm

    Colin,

    I meant something a bit more full-featured than the code samples. Amazon’s code samples are generally only paying a lipservice to each language - it’d be nice if they developed full toolkits that would access their API endpoints (as opposed to “samples” - if I can read the specs, I don’t need code samples) so it could literally be dropped into any system.

28. Jay
    

    March 14th, 2006 at 1:30 pm

    I dont think its a good idea to use SOAP to transfer large fiels. We spent the past 4 years developing software to tranasfer large files and SOAP is not one of them.

    Jay, CTO, http://www.objectcube.com

29. Andrew
    

    March 14th, 2006 at 1:42 pm

    Wow. S3 will rock the hizzouse.

30. Colin Faulkingham
    

    March 14th, 2006 at 1:50 pm

    Roy,

    I see what you mean I have been strugling with the examples for the better part of the day.

    Plug and play modules would be the way to go.

31. pwb
    

    March 14th, 2006 at 2:08 pm

    So if I host my JavaScript Ajax files on Amazon, will they be able to call Amazon web services from pages served by my servers? Does this enable cross-host XMLHttpRequest in a safe manner? Could we envision that web service providers would enable developers to host their JavaScript files from the web service providers servers to take advantage of this?

32. Thomas
    

    March 14th, 2006 at 3:16 pm

    Its not game changing. There are many hosts including mine - netfirms.com, that already offer a reliable load balanced & massive disk storage for $15/month that also includes data transfer for 1500GB per month.

33. Vishal
    

    March 14th, 2006 at 4:21 pm

    Yea, I dont understand either. It’s not game changing at all. I can see how the uninformed developers would jump at this when it says $.15 per GB, and $.20 per GB bandwidth. But when you really think about it, that’s really expensive for bandwidth. I’m in the process of launching a web app thats going to need over 1TB in storage, and its just not practical to use S3.

    Storage services nowadays like Serverpronto.com gives you your own dedicated server with 1TB of space and 2TB of bandwidth, for only 200/month and no contract. If you use S3, thats $150 at least just for the space, at total of $550 per month with the bandwidth, and thats with your data being shared on amazons servers. I just see S3 as being one of the more expensive options for web developers.

34. pwb
    

    March 14th, 2006 at 4:50 pm

    It may not be game changing but I think there is a mis-undertanding for the people comparing it to simple hosting.

    It’s more like an object database with access control and a set of APIs.

35. Roy
    

    March 14th, 2006 at 5:38 pm

    @Vishal: S3 should not be compared to simple webhosting. There are lots of issues with simple webhosting that S3 solves.

    If you know you’ll be set within a certain boundary of disk space for your app (like, anywhere Add(’MyBucket’);
    ?>

    Come on!

36. Roy
    

    March 14th, 2006 at 5:40 pm

    Sorry, the comments system ate my comment, a repost:

    @Vishal: S3 should not be compared to simple webhosting. There are lots of issues with simple webhosting that S3 solves.

    If you know you’ll be set within a certain boundary of disk space for your app (like, anywhere <=400GB), you can set-up a RAID-0 on one of those nice 400GB SATA drives and be set. It would way cheaper to do this (as netfirms, serverpronto, etc. call). However, you still have a single point of failure (if your mobo dies), plus what do you do when you hit 401GB in storage?

    S3’s value lies in the sweet spot between using one server and having to set-up a redundant network of multiple servers. I’m assuming that implicit in Amazon’s offering is that I no longer have to:

    (1) worry about downtime
    (2) worry about patching my system
    (3) worry about adding more hardware for scaling issues

    Again, this isn’t going to be for large sites who can afford a separate sysadmin, nor do I think will it before the casual user (who wants to replace webhosting). It’s going to be for developers who need a quick location to drop an unknown amount of data (user-generated content can grow quite large) who do not want the hassle of dealing with hardware issues (and the fixed costs of running such servers).

    Like I said, it sure would be nice (if someone from AWS is reading this) for some more plug-and-play toolkits in PHP, Ruby, and ASP.NET - in the battle between my wallet and laziness, laziness will always prevail.

    Here is what I’d like to do:

    require_once(’AWSS3.php’);
    $S3 = New S3Bucket;
    $S3->Add(’MyBucket’);

    Come on!

37. Terence
    

    March 14th, 2006 at 6:37 pm

    Vishal,

    It also differs from hosting providers from one simple reason. “Consumers” can access this kind of storage paying only for what they use.

    Example: some geek creates a backup windows app that “connects” directly to S3. The app asks you to have an account on S3 only.

    Now, the consumer can “bypass” the backup middlemen altogether and just get an account for S3 (which is really easy).

    Now, the consumer is just paying $0.35 total for each GB stored and transferred (one way).

    This changes the game, especially to the backup middlemen, including me

    Terence, co-founder, xackup.com

38. Terence
    

    March 14th, 2006 at 6:43 pm

    Jay@49,

    Yes, we have experimented with SOAP as well for transferring large files and the overhead just kills you. It is just not built for it.

    We actually created our own protocol, which we are thinking of open sourcing.

    Having said that, maybe the “middlemen” backup guys’ advantage (including mine) is to create a more integrated and “way, way” faster uploading of data, which S3 will definitely prevent with its REST API.

39. Eric
    

    March 14th, 2006 at 6:57 pm

    Roy:

    check out this link:

    http://developer.amazonwebserv.....egoryID=47

    it’s a ruby library that behaves exactly as you describe:

    conn = S3.AWSAuthConnection(awsAccessKeyId, awsSecretAccessKey)
    conn.create_bucket(”MyBucket”)
    conn.put(”MyBucket”, “MyObject”, “this is some data”)

40. Alex Castro
    

    March 15th, 2006 at 12:29 am

    I’m happy to see my former team members from Amazon Web Services launch S3. It has been in the works for quite some time. I know they must be very proud of what they’ve accomplished.

    I’ve recently started my own Web 2.0 company and I am seriously considering using S3 myself.

    Still there are plenty of questions to be answered.

    http://alexcastro.typepad.com/....._serv.html

41. lenkov
    

    March 15th, 2006 at 1:09 am

    The price is good, but do they offer Akami-style locality balancing. If I have clients in Europe or Asia, will they get the “objects” from a local datacenter, or they have go all the way to San Jose to retrieve the data. If you have 50 JS/css/image/swf files on a single page stored on S3, with average roundtrip from Europe/Asia of 150ms it takes ~4 sec (assuming 2 browser threads) just in roundtrips along. We spend months in building our own global hosting infrastructure for SiteKreator (http://sitekreator.com/sitekreator/sitekreator_global_infrastructure.html) at much higher cost than $0.15 per Gb just to avoid these 4 sec. Will S3 address this locality-balancing issue?

42. Andreas
    

    March 15th, 2006 at 1:16 am

    What, are you kidding me? Ok, they may have a great API and they can make sure the data is protected and always reliable, but is it worth for 6+ times the price you should pay?

    I pay $1450 for a 100mbps server (12 megabytes of transfer per second) with 5×300GB HD’s. Now I don’t have the same infrastructure that amazon has, but I much rather spend $1450 for the storage and bandwidth and invest the other $5k I’d be spending with amazon to make sure the data is doing fine.

    This might bring some small developers, but any person with minimum knowledge of the hosting business would never sign up for such prices.

    Just my $.02

43. Roy
    

    March 15th, 2006 at 1:16 am

    lenkov: I don’t think this is an ideal solution for webhosting, which would render your question somewhat moot.

    Not only is there a $0.15/gig month for storage, you’re also paying $0.20/gig month for transfer, which is quite pricey. Generally you can get $0.10/gig - $0.05/gig for transfers (without the local balancing).

    This isn’t going to undercut Akamai’s mission-critical local delivery anytime soon, nor is it going to take over the webhosting industry &