Friday, December 19, 2014

Amazon Web Services Developer Flight Checklist

Working in the clouds on Amazon AWS benefits from some best practices.  I am capturing details into a pre-flight checklist before you fly between the clouds as I come across them in my daily development.

Amazon Web Service Best Practices

Elastic Beanstalk

1) Add JVM tracking like New Relic
2) Add Log tracking like SumoLogic or Splunk
3) Add GEOIP tracking like MaxMind geoIP
When deploying the mod_geoip conf file to elastic beanstalk remember that we are running behind an elastic load balancer proxy.  This proxy will add its IP address to the %{X-Forwarded-For} variable.  geoIP will not be able to pick out the actual client IP until you add the following parameter to the conf file to workaround this nonstandard NCSA log format.
4) Install an SSL certificate on the ELB (Elastic Load Balancer)
Follow but gunzip, tar xvf and cd to the install directory first.

EC2 instances need the Java AWS SDK
obrienlabs-mbp15:_deployment michaelobrien$ sudo scp ~/.aws/credentials ec2-user@

credentials                                                                                                                            100%  221     0.2KB/s   00:00  

Pick your JVM
They do seem to quietly upgrade the JDK after official published versions - they have since went for 1.6.0_24 to 32 and 1.7.0_51 to 72
OpenJDK versions lag SUN by about a week, but amazon seems to favour Java 8
The 3 ami versions for 6, 7 and 8 look to be behind more as we drop JDK versions - with Java 8 up to date and 6 way behind.

OpenJDK versions
Beanstalk Tomcat 8 Java 8 =  1.8.0_25 (latest 25) 2014.09 v1.0.0
Beanstalk Tomcat 7 Java 7 = 1.7.0_65 (latest 72) 2014.09 v1.0.9
Beanstalk Tomcat 7 Java 6 = 1.6.0_32 (latest 85) 2014.09 v1.0.9

Their official page on supported platforms
On all your windows machines install cygwin and enable the ssh server.

AWS Technical Details

Amazon Service Disruptions:

Some AWS services encounter issues from time to time at the service, zone or region.  If you happen to be in the console you see the occasional message at the top - however the RSS feed gives a bit more detail on the issue.  Again if you build in standard zone and region redundancy then failover kicks in and no service disruption affects your instances.

Install the client - passed by my teamlead

VPC restrictions

us-east-1a is restricted by amazon for new subnets (I read this somewhere – and also read that is was full) – and for t2 instances

us-east-1c came back as restricted during a CloudFormation json deployment run – but I read somewhere that this may be only for customers that have accounts predating when the “default” VPC (only 1 public subnet at – customers like my personal  that signed up after Aug 2014 get are in the default VPC.

Elastic Beanstalk

Configuring a local OSX dev environment for Beanstalk


Get an SSL Certificate signed by Entrust

Entrust will reply to your request within 10 min.

obrienlabs-mbp15:nutridat_domain_cert michaelobrien$ openssl genrsa 2048 > privatekey.pem
Generating RSA private key, 2048 bit long modulus
e is 65537 (0x10001)

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Ontario
Locality Name (eg, city) []:Ottawa
Organization Name (eg, company) [Internet Widgits Pty Ltd]:.
Organizational Unit Name (eg, section) []:.
Common Name (e.g. server FQDN or YOUR name) []
Email Address []:michael@o

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:

An optional company name []:.

Elastic Beanstalk on AWS runs on a customized EC2 AMI as Tomcat 6/7/8 on top of Apache HTTPD.
Configure your maven project to compile against Tomcat 7 or 8


No automatic maven dependencies in eclipse/sts? add the following
Get the AWS CLI and EC2 CLI
curl "" -o ""
obrienlabs-mbp15:aws michaelobrien$ ec2-describe-instances
curl "" -o ""
obrienlabs-mbp15:aws michaelobrien$ aws --version
aws-cli/1.7.0 Python/2.7.6 Darwin/14.0.0
Install Apache HTTP Server
Install Apache Tomcat Web Container
When registering a domain with a Canadian address - you will need a space in the postal code to get past the verification check on AWS Route53
EC2 Get metadata using the link local address
Increase the upload size for files greater than 5Mb in server.xml by changing the maxSwallowSize

    <Connector port="8080" protocol="HTTP/1.1"
               redirectPort="8443" />
Install monitoring services like SumoLogic, New Relic and Boundary

curl -fsS -d '{"token":"api.6f44444444-3333"}' -H 'Content-Type: application/json' > && chmod +x && ./

20150408: beanstalk environment configuration update does not work - it fails to update the AMI
before ami: ami-986327f0
after update from 2014.09 v1.2.0 to 2015.03 v1.3.0
after ami: ami-986327f0
2015-04-08 11:45:33 UTC-0400 ERROR Update environment operation is complete, but with errors. For more information, see troubleshooting documentation. 

Since the default ami for 2015.03 v1.3.0 is

This should be the ami during the upgrade
Fix: replace the ami with the new ami-0c6f5f64 version after the AWS upgrade
2015-04-08 15:13:12 UTC-0400 INFO Environment health has transitioned from RED to GREEN 

AWS Technical Links

Beanstalk CloudFormation parameters -
Elastic Container Service -
Send Mail - SES Sending Limits - Installing AWS CLI -

configure secure access

Get all the hosts up on your private subnet

for ip in $(seq 1 254); do ping -c 1 192.168.0.$ip>/dev/null; [ $? -eq 0 ] && echo "192.168.0.$ip UP" || : ; done

20150329: RDS Performance: My t2-micro RDS instances are taking 13 sec for all queries (count, select, filter) where they used to take less than 1.  Rebooting - no change, rebuilding the RDS instance decreases the time to 9 sec (indexing?)
When I switched up to a t2-small with double the ram - speed increased to 6 sec.  Therefore I have hit some sweet spot with my 3 million biometric records.
After a couple warm up queries (directly in MySQL Workbench) to kick in burst mode for "select count(1) from biometric.gps_record"
t2-micro = 11.9 sec
t2-small = 2.5 to 1.1 sec
t2-medium = 1.6 to 0.9 sec 
I took a T2.medium (2 threads & 4G) and modified it back to a T2.micro (1 thread @ 1G) and performance went from 1.1 sec to 10 sec as expected - as the query is memory intensive and peaks at 1700Mb from a baseline of 450Mb - therefore a T2.small (1 thread @ 2G) will sufficed until I run up 6 million records

Total Pageviews