Scalr: The Auto-Scaling Open-Source Amazon EC2 Effort

Trackbacks/Pings (Trackback URL)

1. Jeff O'Hara
   

   April 3rd, 2008 at 9:56 am

   This is really compelling. I have been trying to wrap my head around scaling of EC2 for a while, and just can’t visualize it, as it doesn’t seem easy. Looking into this is definitely on my todo list.

   -Jeff
   http://edmodo.com

2. Jeremy Wright
   

   April 3rd, 2008 at 9:57 am

   Whoa. This is almost as important a move as AWS itself being launched…

3. Gaurav
   

   April 3rd, 2008 at 10:05 am

   This will be the missing piece of the puzzle if it works well.

4. Nick Messick
   

   April 3rd, 2008 at 10:05 am

   I’ve looked at RightScale. While they are awesome they cost $500/month plus whatever you have to pay Amazon. For a real company, that’s not a big cost. But, for someone that wants to built a web application app as a hobby (such as myself), Scalr looks promising.

5. Gil Megidish
   

   April 3rd, 2008 at 10:06 am

   Kudos for the developers! I would like to see startups use this, rather than taking up investments and hooking up with for-pay services.

   - Gil
   http://www.megidish.net

6. Henry Work
   

   April 3rd, 2008 at 10:25 am

   Nick: Yeah, you gotta love the open-source disruption.

   I’d be interested to hear from other developers who work with EC2 as to their scalability. What I hear is that it’s still a very manual process: you see demand is up one week, so you increase it for the next.

7. ec2box
   

   April 3rd, 2008 at 10:30 am

   Static IP is a nice improvement and gives new ec2 users a familiar starting point.

8. Tom Leasure
   

   April 3rd, 2008 at 11:01 am

   This looks pretty awesome, can’t wait to try it out.

9. whoopie
   

   April 3rd, 2008 at 11:07 am

   somewhere in here is a great idea for real virtualization

   these tools are changing the rules on scaling…now rule zero is - don’t do it if you can get someone else to do it for you

   normally i harsh on everything here, this though is solid stuff

10. Raj
    

    April 3rd, 2008 at 11:12 am

    This is amazing stuff. If it works as said, this could change the way companies deploy their web applications.

    I checked out the documentation and it lacks a real “Get Started” guide. Maybe someone familiar with EC2 should write it up so that newbees can work with this easily.

    Cheers,
    Raj.

11. naffis
    

    April 3rd, 2008 at 11:25 am

    Whoever uses Scalr will no doubt have to be familiar with EC2 and have a strong sys admin background. Probably not the best thing for newbies.

    We may write a getting started guide but we’re hoping the development community will take this project to a whole new level with all the necessary instructions.

12. Simplebucket, Real SImple Photo Hosting
    

    April 3rd, 2008 at 11:27 am

    Wow.. This is like a dream come true

13. james
    

    April 3rd, 2008 at 11:44 am

    that’s very coooool. The we looked into doing this ourselves and could not justify the cost and it became cheaper to keep our server setup where they were.

14. Matt
    

    April 3rd, 2008 at 12:14 pm

    One of the big problems with AWS (EC2+S3) is the data loss between the time of an instance failure and your last database backup.

    I think their new SimpleDB may solve the data gaps, and I would assume Amazon will eventually build tools in-house that eliminate the need for Scalr.

15. boi
    

    April 3rd, 2008 at 12:21 pm

    the best aws management i’ve seen so far is morphexchange.com $39 to start. founded by a serial entrepreneur who sold one of his companies to IBM. I hope to see scalr in their services.

16. Jim
    

    April 3rd, 2008 at 1:20 pm

    This looks great. I’ll use it.

17. rick
    

    April 3rd, 2008 at 1:27 pm

    well don’t forget about Nirvanix, they’re all about scale scale scale—
    and they’re not shy about how they stack up against s3
    for those interested http://www.nirvanix.com/comparison.aspx

18. dhingana
    

    April 3rd, 2008 at 2:18 pm

    very compelling…and a great move…good luck scalr team!

19. Dave Nielsen
    

    April 3rd, 2008 at 2:25 pm

    I organized a demo of 3tera + Nirvanix which essentially lets companies run their own Computing Cloud - with EC2+S3+Scalr-like functionality. Just so you know … there are options!

20. Richard C0r5a13
    

    April 3rd, 2008 at 2:40 pm

    to get redundancy on each node you would need a minimum of 6 servers - 2 load balancers with round robin DNS, 2 database servers, and 2 application servers. The cost savings really come from scaling up and down based on demand and having just the right amount of capacity at all times.

    given the cost of these instances that would run about 2.00 per hour for all 6 ( thats 2 min instances for app servers and 2 for load balancing and 2 mega instances for the DB )

    plus bandwidth .. it would come out to about 30-50 a day for heavy load .. that seems like a bargan for the level of redundancy you get..

21. Fabian Schonholz
    

    April 3rd, 2008 at 3:25 pm

    This looks very cool. Rather then going into details, and if you are interested, I wrote a blog post about this very topic.

    http://www.fabianschonholz.com.....-solution/

22. Scott
    

    April 3rd, 2008 at 4:03 pm

    The MediaPlug transcoding utility mentioned in the article helps power VisualCV.com (http://www.visualcv.com).

    When people add portfolio items (video, images, docs, etc.) to their VisualCVs, MediaPlug is used to transcode, virus-check, and store the items on Amazon S3.

23. Jono
    

    April 3rd, 2008 at 6:43 pm

    @naffis: I agree, We looked at hosting our upcoming web application on EC2 and S3 awhile back. Unfortunately at that point in time it was more trouble than it was worth + the latency would have been too high with most of our users expected to reside in Australia.

    What Amazon really need to do is take care of the scaling on the back end and just present their customers with a simple interface with the customer feeling like they are dealing with 1 server.

    The would of course have some sections (like cpanel) where the user would be able to see how many instances they have running, where they are, how much S3 space/bandwidth they are using and where this is stored/mirrored. They could even let the customers make changes here (e.g. say where to store the data , at what load to open a new instance), but at the same time not force them to, letting users just use their service as they would a shared/dedicated hosting service.

24. Deniz
    

    April 3rd, 2008 at 7:07 pm

    I wonder how the dynamic will play out:

    I expect that there will be an ecosystem around Amazon Services, open source or not, but there is always the danger (or opportunity) that Amazon will pick the winners who are doing things right and either buy them or provide same/similar functionality.

    It might be old news for many, but I remember how the Alexa thing played out last year (the year before last year?) so I hope Amazon takes better care of things this time around.

25. Anonymous RightScale Customer
    

    April 3rd, 2008 at 8:37 pm

    If this project were using a professional-grade distribution, instead of Ubuntu, I’d have shitcanned the Rightscale contract sitting on my desk waiting to be signed.

    As it is, there’s too much to do to port a system like this over to something like CentOS or Fedora. Why is it that the Freetards never understand that the choice of OS makes a difference?

    First it was Gentoo, now Ubuntu — don’t you guys realize that *no one* supports apps running on these platforms? Try getting a professional services/support contract with someone like MySQL … and don’t forget to tell them you’re running on a gay OS like Ubuntu. See how far you get.

    Great idea, Scalr folks. Use a real distro at the core if you really want to rock the boat.

26. naffis
    

    April 3rd, 2008 at 10:17 pm

    @Jono and Deniz: I imagine Amazon will continue proving the basic infrastructure we need and leave the next layer to other companies. There are a lot of companies building tools and platforms to expose the power or EC2 and other platforms and it’s a win win for Amazon to keep it this way. It give customers choice in choosing their providers and tools and it creates that ecosystem you were talking about. eBay is a good example of this.

    @Anonymous RightScale Customer : Scalr is not tied to any particular distro, you can use whatever you want including CentOS or Fedora. The prebuilt images are Ubuntu but you can take the scripts and setup and put them on whatever you like. And if you took a look at the actual code you would see that there isn’t too much to do to port it over. There are some simple scripts which can probably be copied over directly without changing anything, create some new images and you’re good to go.

    If you’re at the scale where a MySQL support contract is necessary or for that matter being on something like CentOS or Fedora is an absolute necessity then this isn’t for you right now anyway.

27. Thorsten - CTO RightScale
    

    April 3rd, 2008 at 10:33 pm

    Congrats to the scalr team for putting some nice stuff out there! Dunno why they think they need to “disrupt the established players”. There are so many people that need help getting onto EC2 that we can’t possibly serve them all at RightScale. I believe that more options and some good discussions about how to best leverage the cloud will help everyone and float all boats. At this point we provide far more for the fees we charge than a pile of code. There’s the dashboard, the monitoring, alerting, automation based on alerts to do autoscaling an other things, server templates for dozens of popular packages, plus real support. We’d love nothing more than to lower the fees we charge, but we also need to make sure we can actually make our customers successful in the cloud.

28. Rex Chung
    

    April 8th, 2008 at 6:57 am

    Interesting to see that we’re making almost the same thing using Amazon EC2 and S3. We’re also a ruby on rails company building a video transcoding web service. Congrats to Scalr for releasing it and writing up the documentations. We can probably learn something from it.