Saturday, February 14, 2015

AWS Spot EC2 Instances for Elastic Beanstalk saves around 70-90% of instance costs over reserved or on-demand

    Amazon AWS is an excellent service - that Amazon runs very well.  There are opportunities to save between 25 and 60% - but there is and even better way to reduce costs for non-production development servers.
    On-demand instance cost for AWS resources can be reduced by 25% using no-upfront reservations or by 56% by purchasing the instance for 3 years.  However the use of the spot instance market allows a reduction from 80 to 90% - but not in beanstalk.
    There is however a way to reserve spot instances for elastic beanstalk through the .ebextension mechanism like we do for other types of EC2 instance customization.  There are the usual problems of not being compatible with auto-scaling but this price reduction is excellent for development instances.
    Enter the following .config with a spot price that is just above the lowest price across all regions for the past 3 months - so you run the lowest risk of loosing the instance.  For example I reserved the new C4.Large at 0.018 which is above the normal spot of 0.0161 - however this is a lot better than the on-demand price of 0.160 - or 10% of the normal price.
Note the spot price which is a reduction of 90% on the AWS EC2 on-demand price.  We will put this into the .ebextension directory in a config file


Add the following to your pom.xml if you are using "ebextensions" instead of the target ".ebextensions"

  <build>
      <plugins>
      <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <configuration>
                    <webResources>
                        <resource>
                            <directory>src/main/ebextensions</directory>
                            <targetPath>.ebextensions</targetPath>
                            <filtering>true</filtering>                         

                        </resource>

After you build your WAR and deploy it, you will initially see some scaling errors until the spot request is fulfilled.


As soon as your instance is up and the war completes deployment - the healthcheck will report a good instance.

    If you are running a set of development servers like 10 M3.xlarge instances at a on-demand price of $0.28/hour - this comes out to $67/day or $25,000/year.  Switching to using spot instances at the current stable price in us-east-1b of $0.0321 - but lets use $0.034 - would come out to $8/day or $3000/year - a cost savings of $22,000 at 88% off.  Actual savings would be more in line with $8,000 or 72% if you normally use 3yr reserved instances.

    Note that you can loose your instance at any time if the lowest spot price jumps.
    There are ways to get environment variables into the config files without resorting to commands or sh scripts - so we can change the spot price when we want to up the instance to a C4.8xlarge for example.

/michael


4 comments:

Anonymous said...

This is a very very good tip. Thanks a lot!

Now, one question: You said about putting the bid price at an environment property/variable at beanstalk, so there's no need to change the .config file at ebextensions everytime we want to increase our bid. Do you have an example of a .config that do that? I'm trying to get the environment property value using Fn:GetAtt or Ref, but I'm not able to do so.

Thanks again!

Unknown said...

I would also be interested in seeing this. Daniel did you have any luck?

Anonymous said...

Unfortunately no. But, given that the bid price you specify is actually the maximum price you're able to pay, I just put a high value (actually, the current on demand price) at my .ebextensions and it was enough for my use case.

Elad Nava said...

Another option is to find the underlying Launch Configuration of your Elastic Beanstalk environment, copy it, specify the EC2 instance type and spot price within the launch configuration, and finally, modify the underlying Auto Scaling Group of your EB to use the new launch configuration.

That way, you don't have to mess with the .ebextensions and you get the nice AWS console when configuring the Spot Fleet.

Post by Elad Nava

Total Pageviews

Followers