Friday, December 19, 2014

Amazon Web Services Developer Flight Checklist


Working in the clouds on Amazon AWS benefits from some best practices.  I am capturing details into a pre-flight checklist before you fly between the clouds as I come across them in my daily development.

Amazon Web Service Best Practices

Elastic Beanstalk

1) Add JVM tracking like New Relic
2) Add Log tracking like SumoLogic or Splunk
3) Add GEOIP tracking like MaxMind geoIP
When deploying the mod_geoip conf file to elastic beanstalk remember that we are running behind an elastic load balancer proxy.  This proxy will add its IP address to the %{X-Forwarded-For} variable.  geoIP will not be able to pick out the actual client IP until you add the following parameter to the conf file to workaround this nonstandard NCSA log format.
see http://dev.maxmind.com/geoip/legacy/mod_geoip2/#Proxy-Related_Directives
4) Install an SSL certificate on the ELB (Elastic Load Balancer)
Follow http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/ssl-server-cert.html but gunzip, tar xvf and cd to the install directory first.

EC2 instances need the Java AWS SDK
obrienlabs-mbp15:_deployment michaelobrien$ sudo scp ~/.aws/credentials ec2-user@54.5.6.7:/home/ec2-user
Password:

credentials                                                                                                                            100%  221     0.2KB/s   00:00  

Pick your JVM
They do seem to quietly upgrade the JDK after official published versions - they have since went for 1.6.0_24 to 32 and 1.7.0_51 to 72
OpenJDK versions lag SUN by about a week, but amazon seems to favour Java 8
The 3 ami versions for 6, 7 and 8 look to be behind more as we drop JDK versions - with Java 8 up to date and 6 way behind.

OpenJDK versions
Beanstalk Tomcat 8 Java 8 =  1.8.0_25 (latest 25) 2014.09 v1.0.0
Beanstalk Tomcat 7 Java 7 = 1.7.0_65 (latest 72) 2014.09 v1.0.9
Beanstalk Tomcat 7 Java 6 = 1.6.0_32 (latest 85) 2014.09 v1.0.9

Their official page on supported platforms
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.platforms.html#concepts.platforms.java
On all your windows machines install cygwin and enable the ssh server.
http://www.howtogeek.com/howto/41560/how-to-get-ssh-command-line-access-to-windows-7-using-cygwin/

AWS Technical Details

Amazon Service Disruptions:

Some AWS services encounter issues from time to time at the service, zone or region.  If you happen to be in the console you see the occasional message at the top - however the RSS feed gives a bit more detail on the issue.  Again if you build in standard zone and region redundancy then failover kicks in and no service disruption affects your instances.

Install the client - passed by my teamlead
https://chrome.google.com/webstore/detail/rss-feed-reader/pnjaodmkngahhkoihejjehlcdlnohgmp?hl=en

VPC restrictions

us-east-1a is restricted by amazon for new subnets (I read this somewhere – and also read that is was full) – and for t2 instances

us-east-1c came back as restricted during a CloudFormation json deployment run – but I read somewhere that this may be only for customers that have accounts predating when the “default” VPC (only 1 public subnet at 172.31.0.0/16) – customers like my personal  that signed up after Aug 2014 get are in the default VPC.

Elastic Beanstalk

Configuring a local OSX dev environment for Beanstalk

see

Get an SSL Certificate signed by Entrust

Entrust will reply to your request within 10 min.

obrienlabs-mbp15:nutridat_domain_cert michaelobrien$ openssl genrsa 2048 > privatekey.pem
Generating RSA private key, 2048 bit long modulus
..+++
.............+++
e is 65537 (0x10001)

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Ontario
Locality Name (eg, city) []:Ottawa
Organization Name (eg, company) [Internet Widgits Pty Ltd]:.
Organizational Unit Name (eg, section) []:.
Common Name (e.g. server FQDN or YOUR name) []:nutridat.org
Email Address []:michael@o

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:

An optional company name []:.


Elastic Beanstalk on AWS runs on a customized EC2 AMI as Tomcat 6/7/8 on top of Apache HTTPD.
Configure your maven project to compile against Tomcat 7 or 8

  
    
      junit
      junit
      3.8.1
      test
    
            
        
            jstl
            jstl
            1.2
            
               
        
            javax.el
            el-api
            2.2
            provided
        
                
        
            org.apache.tomcat
            tomcat-el-api
            ${version.tomcat}
            provided
        
        
            org.apache.tomcat
            tomcat-servlet-api
            ${version.tomcat}
            provided
        
  

No automatic maven dependencies in eclipse/sts? add the following
 
  
   
Get the AWS CLI and EC2 CLI
http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html#setting_up_ec2_command_linux
curl "http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip" -o "ec2-api-tools.zip"
obrienlabs-mbp15:aws michaelobrien$ ec2-describe-instances
 

http://docs.aws.amazon.com/cli/latest/userguide/installing.html
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
obrienlabs-mbp15:aws michaelobrien$ aws --version
aws-cli/1.7.0 Python/2.7.6 Darwin/14.0.0
Install Apache HTTP Server
Install Apache Tomcat Web Container
When registering a domain with a Canadian address - you will need a space in the postal code to get past the verification check on AWS Route53
EC2 Get metadata using the link local address http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
Increase the upload size for files greater than 5Mb in server.xml by changing the maxSwallowSize

    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               maxSwallowSize="52428800"
               redirectPort="8443" />
Install monitoring services like SumoLogic, New Relic and Boundary

curl -fsS -d '{"token":"api.6f44444444-3333"}' -H 'Content-Type: application/json' https://meter.boundary.com/setup_meter > setup_meter.sh && chmod +x setup_meter.sh && ./setup_meter.sh

Issues
20150408: beanstalk environment configuration update does not work - it fails to update the AMI
before ami: ami-986327f0
after update from 2014.09 v1.2.0 to 2015.03 v1.3.0
after ami: ami-986327f0
2015-04-08 11:45:33 UTC-0400 ERROR Update environment operation is complete, but with errors. For more information, see troubleshooting documentation. 


Since the default ami for 2015.03 v1.3.0 is
ami-0c6f5f64

This should be the ami during the upgrade
Fix: replace the ami with the new ami-0c6f5f64 version after the AWS upgrade
2015-04-08 15:13:12 UTC-0400 INFO Environment health has transitioned from RED to GREEN 

AWS Technical Links

Beanstalk CloudFormation parameters - http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html
Elastic Container Service - http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html
http://aws.amazon.com/blogs/aws/ec2-container-service-in-action/?sc_ichannel=ha&sc_ipage=homepage&sc_icountry=en&sc_isegment=c&sc_iplace=hero1&sc_icampaigntype=product_launch&sc_icampaign=ha_en_ECS_Launch&sc_icategory=none&sc_idetail=ha_en_281_1&sc_icontent=ha_281&
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted.html
Send Mail - http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-using-smtp-java.html SES Sending Limits - https://console.aws.amazon.com/ses/home?region=us-east-1#dashboard: Installing AWS CLI - http://docs.aws.amazon.com/cli/latest/userguide/installing.html

configure secure access
http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_VirtualMFA.html

Get all the hosts up on your private subnet

for ip in $(seq 1 254); do ping -c 1 192.168.0.$ip>/dev/null; [ $? -eq 0 ] && echo "192.168.0.$ip UP" || : ; done

Notes:
20150329: RDS Performance: My t2-micro RDS instances are taking 13 sec for all queries (count, select, filter) where they used to take less than 1.  Rebooting - no change, rebuilding the RDS instance decreases the time to 9 sec (indexing?)
When I switched up to a t2-small with double the ram - speed increased to 6 sec.  Therefore I have hit some sweet spot with my 3 million biometric records.
After a couple warm up queries (directly in MySQL Workbench) to kick in burst mode for "select count(1) from biometric.gps_record"
t2-micro = 11.9 sec
t2-small = 2.5 to 1.1 sec
t2-medium = 1.6 to 0.9 sec 
I took a T2.medium (2 threads & 4G) and modified it back to a T2.micro (1 thread @ 1G) and performance went from 1.1 sec to 10 sec as expected - as the query is memory intensive and peaks at 1700Mb from a baseline of 450Mb - therefore a T2.small (1 thread @ 2G) will sufficed until I run up 6 million records




Total Pageviews

Followers