Amazon Web Services Developer Flight Checklist

Working in the clouds on Amazon AWS benefits from some best practices.  I am capturing details into a pre-flight checklist before you fly between the clouds as I come across them in my daily development.

Amazon Web Service Best Practices

Elastic Beanstalk

1) Add JVM tracking like New Relic

2) Add Log tracking like SumoLogic or Splunk

3) Add GEOIP tracking like MaxMind geoIP

When deploying the mod_geoip conf file to elastic beanstalk remember that we are running behind an elastic load balancer proxy.  This proxy will add its IP address to the %{X-Forwarded-For} variable.  geoIP will not be able to pick out the actual client IP until you add the following parameter to the conf file to workaround this nonstandard NCSA log format.
see http://dev.maxmind.com/geoip/legacy/mod_geoip2/#Proxy-Related_Directives

4) Install an SSL certificate on the ELB (Elastic Load Balancer)
Follow http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/ssl-server-cert.html but gunzip, tar xvf and cd to the install directory first.

EC2 instances need the Java AWS SDK

obrienlabs-mbp15:_deployment michaelobrien$ sudo scp ~/.aws/credentials ec2-user@54.5.6.7:/home/ec2-user

Password:

credentials                                                                                                                            100%  221     0.2KB/s   00:00  

Pick your JVM
They do seem to quietly upgrade the JDK after official published versions - they have since went for 1.6.0_24 to 32 and 1.7.0_51 to 72
OpenJDK versions lag SUN by about a week, but amazon seems to favour Java 8
The 3 ami versions for 6, 7 and 8 look to be behind more as we drop JDK versions - with Java 8 up to date and 6 way behind.

OpenJDK versions
Beanstalk Tomcat 8 Java 8 =  1.8.0_25 (latest 25) 2014.09 v1.0.0
Beanstalk Tomcat 7 Java 7 = 1.7.0_65 (latest 72) 2014.09 v1.0.9
Beanstalk Tomcat 7 Java 6 = 1.6.0_32 (latest 85) 2014.09 v1.0.9

Their official page on supported platforms
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.platforms.html#concepts.platforms.java
On all your windows machines install cygwin and enable the ssh server.
http://www.howtogeek.com/howto/41560/how-to-get-ssh-command-line-access-to-windows-7-using-cygwin/

AWS Technical Details

Amazon Service Disruptions:

Some AWS services encounter issues from time to time at the service, zone or region.  If you happen to be in the console you see the occasional message at the top - however the RSS feed gives a bit more detail on the issue.  Again if you build in standard zone and region redundancy then failover kicks in and no service disruption affects your instances.

Install the client - passed by my teamlead
https://chrome.google.com/webstore/detail/rss-feed-reader/pnjaodmkngahhkoihejjehlcdlnohgmp?hl=en

VPC restrictions

us-east-1a is restricted by amazon for new subnets (I read this somewhere – and also read that is was full) – and for t2 instances

us-east-1c came back as restricted during a CloudFormation json deployment run – but I read somewhere that this may be only for customers that have accounts predating when the “default” VPC (only 1 public subnet at 172.31.0.0/16) – customers like my personal  that signed up after Aug 2014 get are in the default VPC.

Elastic Beanstalk

Configuring a local OSX dev environment for Beanstalk

see

https://news.ycombinator.com/item?id=7172060

Get an SSL Certificate signed by Entrust

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/configuring-https.html

Entrust will reply to your request within 10 min.

obrienlabs-mbp15:nutridat_domain_cert michaelobrien$ openssl genrsa 2048 > privatekey.pem

Generating RSA private key, 2048 bit long modulus

..+++

.............+++

e is 65537 (0x10001)

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [AU]:CA

State or Province Name (full name) [Some-State]:Ontario

Locality Name (eg, city) []:Ottawa

Organization Name (eg, company) [Internet Widgits Pty Ltd]:.

Organizational Unit Name (eg, section) []:.

Common Name (e.g. server FQDN or YOUR name) []:nutridat.org

Email Address []:michael@o

Please enter the following 'extra' attributes

to be sent with your certificate request

A challenge password []:

An optional company name []:.

Elastic Beanstalk on AWS runs on a customized EC2 AMI as Tomcat 6/7/8 on top of Apache HTTPD.
Configure your maven project to compile against Tomcat 7 or 8

  
    
      junit
      junit
      3.8.1
      test
    
            
        
            jstl
            jstl
            1.2
            
               
        
            javax.el
            el-api
            2.2
            provided
        
                
        
            org.apache.tomcat
            tomcat-el-api
            ${version.tomcat}
            provided
        
        
            org.apache.tomcat
            tomcat-servlet-api
            ${version.tomcat}
            provided
        
  

No automatic maven dependencies in eclipse/sts? add the following

Get the AWS CLI and EC2 CLI

http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html#setting_up_ec2_command_linux

curl "http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip" -o "ec2-api-tools.zip"

obrienlabs-mbp15:aws michaelobrien$ ec2-describe-instances
 

http://docs.aws.amazon.com/cli/latest/userguide/installing.html

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"



obrienlabs-mbp15:aws michaelobrien$ aws --version

aws-cli/1.7.0 Python/2.7.6 Darwin/14.0.0




Install Apache HTTP Server

http://httpd.apache.org/docs/trunk/install.html



Install Apache Tomcat Web Container

When registering a domain with a Canadian address - you will need a space in the postal code to get past the verification check on AWS Route53
EC2

Get metadata using the link local address

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

Increase the upload size for files greater than 5Mb in server.xml by changing the maxSwallowSize

    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               maxSwallowSize="52428800"
               redirectPort="8443" />

Install monitoring services like SumoLogic, New Relic and Boundary

curl -fsS -d '{"token":"api.6f44444444-3333"}' -H 'Content-Type: application/json' https://meter.boundary.com/setup_meter > setup_meter.sh && chmod +x setup_meter.sh && ./setup_meter.sh


Issues

20150408: beanstalk environment configuration update does not work - it fails to update the AMI

before ami: ami-986327f0
after update from 2014.09 v1.2.0 to 2015.03 v1.3.0
after ami: ami-986327f0
2015-04-08 11:45:33 UTC-0400 ERROR Update environment operation is complete, but with errors. For more information, see troubleshooting documentation. 


Since the default ami for 2015.03 v1.3.0 is
ami-0c6f5f64

This should be the ami during the upgrade
Fix: replace the ami with the new ami-0c6f5f64 version after the AWS upgrade
2015-04-08 15:13:12 UTC-0400 INFO Environment health has transitioned from RED to GREEN 

AWS Technical Links
Beanstalk CloudFormation parameters - http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html

Elastic Container Service - http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html

http://aws.amazon.com/blogs/aws/ec2-container-service-in-action/?sc_ichannel=ha&sc_ipage=homepage&sc_icountry=en&sc_isegment=c&sc_iplace=hero1&sc_icampaigntype=product_launch&sc_icampaign=ha_en_ECS_Launch&sc_icategory=none&sc_idetail=ha_en_281_1&sc_icontent=ha_281&

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted.html
Send Mail - http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-using-smtp-java.html

SES Sending Limits -

https://console.aws.amazon.com/ses/home?region=us-east-1#dashboard:

Installing AWS CLI - http://docs.aws.amazon.com/cli/latest/userguide/installing.html

configure secure access

http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_VirtualMFA.html

Get all the hosts up on your private subnet

for ip in $(seq 1 254); do ping -c 1 192.168.0.$ip>/dev/null; [ $? -eq 0 ] && echo "192.168.0.$ip UP" || : ; done


Notes:
20150329: RDS Performance: My t2-micro RDS instances are taking 13 sec for all queries (count, select, filter) where they used to take less than 1.  Rebooting - no change, rebuilding the RDS instance decreases the time to 9 sec (indexing?)
When I switched up to a t2-small with double the ram - speed increased to 6 sec.  Therefore I have hit some sweet spot with my 3 million biometric records.
After a couple warm up queries (directly in MySQL Workbench) to kick in burst mode for "select count(1) from biometric.gps_record"
t2-micro = 11.9 sec
t2-small = 2.5 to 1.1 sec
t2-medium = 1.6 to 0.9 sec 
I took a T2.medium (2 threads & 4G) and modified it back to a T2.micro (1 thread @ 1G) and performance went from 1.1 sec to 10 sec as expected - as the query is memory intensive and peaks at 1700Mb from a baseline of 450Mb - therefore a T2.small (1 thread @ 2G) will sufficed until I run up 6 million records