40 Terabytes More Data For Amazon S3

July 13th, 2007 at 1:08 am

I’ll be curious to see how much money Amazon makes on S3.

July 13th, 2007 at 1:59 am

If Amazon S3 is their back-up strategy, and they are moving “all” their back-ups there, aren’t they in fact still “betting their customer’s data on that”?

July 13th, 2007 at 5:34 am

Good coverage. I really believe that the future of Data Storage is heading to this service type model.

WAAAAY back in 2006, I made some predictions about the future of this market, in this piece ” Future of Online Storage, 40 points to think about”

Many of them are already coming true.

What most people haven’t realized is the tacit knowledge in the cloud, and how we can extract that knowledge.

http://www.web-strategist.com/.....ink-about/

July 13th, 2007 at 7:05 am

has anybody made experience with S3 on serving content/streams for Europe?..have been told once latency was high and once that it serves quite well..as I am not the guru of techs anybody out there with experience :-)?

ice

July 13th, 2007 at 7:20 am

We’re going with EC2 and S3 integration to run our Flex 2 web-application, Convos. It’s currently in closed testing and we’ve been really pleased with experience and performance.

This past week in NYC, Amazon’s AWS evangelist presented at the NY Tech Meetup and I was surprised how many people had not heard of their services. He did a demo of an 3rd party software that monitors your EC2 servers and automatically increases / decreases capacity based on demand. Very cool.

PS #4 - PostgreSQL works with AWS.

July 13th, 2007 at 8:43 am

Amazon must be making a killing on S3!

Darin

July 13th, 2007 at 9:38 am

I have been reading about the online backup industry for a while now, mostly from the aggregator website,

BackupReview.info

As this site ranks the top 25 online backup companies on a monthly basis, I have not seen even once Amazon’s S3 ranked in the top 25 in the past 2 years. If S3 is that good and relaible, then why is that it did not make the cut at least once?

It sounds fishy here, and it looks like that the more a company has money to promote its products and services, the more we hear about them, despite the fact that there are tons others with far better products.

July 13th, 2007 at 10:33 am

Andrew @ 10,

Thanks for the reply. I hear you on RAID6 as your primary.

I am curious if Amazon facilitates periodic integrity checks and tests of back-ups (that do not themselves incur significant transfer costs, e.g. MD5 checks on both sides).

S3 is appealing. In outsourcing critical or back-up systems to S3 I would be looking for comfort that the triple nightmare scenario of “RAID6 failure, back-up corrupt” is demonstrably unlikely.

July 13th, 2007 at 4:55 pm

S3 doesn’t seem to make sense financially. Can someone shed some light on how it’s cost effective?

A 1TB server costs about $1000 (or less) to buy. Expect at least a year of service from that one server, probably closer to 5 years, or longer when you replace the drives.

S3 costs $0.15 per GB per month, which is $0.15 x 1000GB x 12 months, which is $1800. For only one year. Two years is $3600. 3 years is $5400. In the meantime, your $1000 server is still humming along at 1/5th the cost. And will still be working for a full five years while S3 will have cost you $9000 in 5 years for the same amount of storage that you paid $1000 for.

The bandwidth costs seem to be about the same, especially if you’re just using this server for a backup.

Operations costs? Hands-on service at a co-lo can be as high as $200 per hour. So, let’s say a drive goes out once a year and needs to be replaced. You’re still much cheaper than S3.

July 13th, 2007 at 7:10 pm

frank @17: You are correct. You can build out storage more cheaply. Numbers are not as bad as you say though. You did not consider the cost of co-location and bandwidth for the server. But most importantly, you did not consider two additional things:

1. you will almost certainly wind up virtualizing that 1TB server if you buy 100 of them. Writing the virtualization layer so that it appears as limitless disk is non trival. Buying a solution (probably a SAN solution, will increase your storage cost a lot).

2. S3 is considerably more reliable than a single 1TB server. It is more like a single 1TB server and another server or two to back it up. Your single terabyte server is not backed up in multiple locations. S3 automatically does that for you.

If you compare apples to apple and built a S3, you would be hard pressed to do it for less. Certainly it is possible but your scale would need to be quite large to do it.

One of the reasons S3 seems expensive for backup is that it is not really backup. It is a reliable store that is also backed up, essentially. Hence, using S3 for backup is a bit of overkill. That is why to really clean up cost wise, you must make S3 be your primary data store. Whether you feel comfortable doing that without an SLA from Amazon is up to you.

July 13th, 2007 at 7:56 pm

Hi Fabian (we have met),
It took us about a week to create the S3 solution. Granted, we were able to leverage some existing code we had that calculated incremental backups, but my guess is that the TCO on the pictage system would still put you under water relative to an S3 solution today, even with a 300TB. Anyway, you should share your magical distributed, limitless storage solution. Can’t make a claim like that and then just walk away.

Andrew

July 14th, 2007 at 8:42 am

Andrew:

Hey man … how are you? Scott mentioned something about you guys a few weeks ago. You guys still using the same technology when we last talk?

The secret sauce is AFS. That is all the magic you need. In my opinion it is not only the time that takes you to create a solution, but the cost of maintenance and mantainability, availability, risk through time, lover external dependencies, ownership and overall performance. I did not say that S3 is not a good solution; I said that there are better off-market solutions that you can build yourself and leverage to increase the valuation of your company in an M&A.

It is the same old argument of build vs. buy. I am more of the build mentality - but only for core technologies that do make a difference at valuation time. For example, I would not build a general ledger, but in the case of Pictage, I decided to build the image serving technology for two reasons:

1) Totally core to the company.
2) Even though there were some good products at the time, they were not cost effective.

At the end of the day, to this day, nobody has an Image Server as good, flexible, versatile as Pictage (and I built that) and if you do not believe me, look at Pictage’s Album Designer. The imaging back end is all done real time by the Image Server.

I am no longer at Pictage … good riddance … people there get treated like dogs. But the technology I left behind remains really powerful, including the storage solution.

July 16th, 2007 at 11:00 pm

Frank,

Your 1TB server is not a fair comparison. Amazon mirrors your data in multiple data centers so you will need to consider having at least 2 data centers and consider the b/w between them. The problem is that you are comparing cheap backup (as in, i don’t really care if I lose my backup because its just backup) to Amazon’s triple redundant, physically distributed storage.

our calculations are that you can build the storage yourself for less from a hw cost standpoint, even keeping copies in multiple data centers, but if you put even one head on the project fulltime to keep adding storage, fixing broken stuff, and scaling the system, the breakeven point is years out.

Reliability is not just one number. I did not mean availability (where amazon had some issues). I meant that Amazon won’t lose your data.

As for the SLA, we absolutely agree. It is needed. It is not really needed to use Amazon as backup, but it is certainly needed to make them primary storage. Feel free to email me at andrew@phanfare.com if you want more detailed cost numbers.