Working in the clouds on Amazon AWS benefits from some best practices. I am capturing details into a pre-flight checklist before you fly between the clouds as I come across them in my daily development.
Amazon Web Service Best Practices
Elastic Beanstalk
1) Add JVM tracking like New Relic
2) Add Log tracking like SumoLogic or Splunk
3) Add GEOIP tracking like MaxMind geoIP
When deploying the mod_geoip conf file to elastic beanstalk remember that we are running behind an elastic load balancer proxy. This proxy will add its IP address to the %{X-Forwarded-For} variable. geoIP will not be able to pick out the actual client IP until you add the following parameter to the conf file to workaround this nonstandard NCSA log format.
see http://dev.maxmind.com/geoip/legacy/mod_geoip2/#Proxy-Related_Directives
see http://dev.maxmind.com/geoip/legacy/mod_geoip2/#Proxy-Related_Directives
4) Install an SSL certificate on the ELB (Elastic Load Balancer)
Follow http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/ssl-server-cert.html but gunzip, tar xvf and cd to the install directory first.
EC2 instances need the Java AWS SDK
Follow http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/ssl-server-cert.html but gunzip, tar xvf and cd to the install directory first.
EC2 instances need the Java AWS SDK
obrienlabs-mbp15:_deployment michaelobrien$ sudo scp ~/.aws/credentials ec2-user@54.5.6.7:/home/ec2-user
Password:
credentials 100% 221 0.2KB/s 00:00
They do seem to quietly upgrade the JDK after official published versions - they have since went for 1.6.0_24 to 32 and 1.7.0_51 to 72
OpenJDK versions lag SUN by about a week, but amazon seems to favour Java 8
The 3 ami versions for 6, 7 and 8 look to be behind more as we drop JDK versions - with Java 8 up to date and 6 way behind.
OpenJDK versions
Beanstalk Tomcat 8 Java 8 = 1.8.0_25 (latest 25) 2014.09 v1.0.0
Beanstalk Tomcat 7 Java 7 = 1.7.0_65 (latest 72) 2014.09 v1.0.9
Beanstalk Tomcat 7 Java 6 = 1.6.0_32 (latest 85) 2014.09 v1.0.9
Their official page on supported platforms
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts.platforms.html#concepts.platforms.java
On all your windows machines install cygwin and enable the ssh server.
http://www.howtogeek.com/howto/41560/how-to-get-ssh-command-line-access-to-windows-7-using-cygwin/
AWS Technical Details
Amazon Service Disruptions:
Some AWS services encounter issues from time to time at the service, zone or region. If you happen to be in the console you see the occasional message at the top - however the RSS feed gives a bit more detail on the issue. Again if you build in standard zone and region redundancy then failover kicks in and no service disruption affects your instances.Install the client - passed by my teamlead
https://chrome.google.com/webstore/detail/rss-feed-reader/pnjaodmkngahhkoihejjehlcdlnohgmp?hl=en
VPC restrictions
us-east-1a is restricted by amazon for new subnets (I read this
somewhere – and also read that is was full) – and for t2 instances
us-east-1c came back as restricted during a CloudFormation json deployment run – but I read
somewhere that this may be only for customers that have accounts predating when
the “default” VPC (only 1 public subnet at 172.31.0.0/16) – customers like my personal that signed up after Aug 2014 get are in the default VPC.
Elastic Beanstalk
Configuring a local OSX dev environment for Beanstalk
see
Get an SSL Certificate signed by Entrust
Entrust will reply to your request within 10 min.
obrienlabs-mbp15:nutridat_domain_cert michaelobrien$ openssl genrsa 2048 > privatekey.pem
Generating RSA private key, 2048 bit long modulus
..+++
.............+++
e is 65537 (0x10001)
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Ontario
Locality Name (eg, city) []:Ottawa
Organization Name (eg, company) [Internet Widgits Pty Ltd]:.
Organizational Unit Name (eg, section) []:.
Common Name (e.g. server FQDN or YOUR name) []:nutridat.org
Email Address []:michael@o
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:.
Elastic Beanstalk on AWS runs on a customized EC2 AMI as Tomcat 6/7/8 on top of Apache HTTPD.
Configure your maven project to compile against Tomcat 7 or 8
No automatic maven dependencies in eclipse/sts? add the following
Configure your maven project to compile against Tomcat 7 or 8
junit junit 3.8.1 test jstl jstl 1.2 javax.el el-api 2.2 provided org.apache.tomcat tomcat-el-api ${version.tomcat} provided org.apache.tomcat tomcat-servlet-api ${version.tomcat} provided
No automatic maven dependencies in eclipse/sts? add the following
Get the AWS CLI and EC2 CLI
http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/set-up-ec2-cli-linux.html#setting_up_ec2_command_linux
curl "http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip" -o "ec2-api-tools.zip"obrienlabs-mbp15:aws michaelobrien$ ec2-describe-instances
http://docs.aws.amazon.com/cli/latest/userguide/installing.html
Increase the upload size for files greater than 5Mb in server.xml by changing the maxSwallowSize
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
maxSwallowSize="52428800"
redirectPort="8443" />
Install monitoring services like SumoLogic, New Relic and Boundary
curl -fsS -d '{"token":"api.6f44444444-3333"}' -H 'Content-Type: application/json' https://meter.boundary.com/setup_meter > setup_meter.sh && chmod +x setup_meter.sh && ./setup_meter.shIssues
20150408: beanstalk environment configuration update does not work - it fails to update the AMI
before ami: ami-986327f0 after update from 2014.09 v1.2.0 to 2015.03 v1.3.0 after ami: ami-986327f0 2015-04-08 11:45:33 UTC-0400 ERROR Update environment operation is complete, but with errors. For more information, see troubleshooting documentation. Since the default ami for 2015.03 v1.3.0 is ami-0c6f5f64 This should be the ami during the upgrade Fix: replace the ami with the new ami-0c6f5f64 version after the AWS upgrade 2015-04-08 15:13:12 UTC-0400 INFO Environment health has transitioned from RED to GREENAWS Technical Links
Beanstalk CloudFormation parameters - http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.htmlElastic Container Service - http://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.htmlSend Mail - http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-using-smtp-java.html SES Sending Limits - https://console.aws.amazon.com/ses/home?region=us-east-1#dashboard: Installing AWS CLI - http://docs.aws.amazon.com/cli/latest/userguide/installing.html
http://aws.amazon.com/blogs/aws/ec2-container-service-in-action/?sc_ichannel=ha&sc_ipage=homepage&sc_icountry=en&sc_isegment=c&sc_iplace=hero1&sc_icampaigntype=product_launch&sc_icampaign=ha_en_ECS_Launch&sc_icategory=none&sc_idetail=ha_en_281_1&sc_icontent=ha_281&
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted.html
configure secure access
http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_VirtualMFA.html
Get all the hosts up on your private subnet
for ip in $(seq 1 254); do ping -c 1 192.168.0.$ip>/dev/null; [ $? -eq 0 ] && echo "192.168.0.$ip UP" || : ; doneNotes:20150329: RDS Performance: My t2-micro RDS instances are taking 13 sec for all queries (count, select, filter) where they used to take less than 1. Rebooting - no change, rebuilding the RDS instance decreases the time to 9 sec (indexing?)When I switched up to a t2-small with double the ram - speed increased to 6 sec. Therefore I have hit some sweet spot with my 3 million biometric records. After a couple warm up queries (directly in MySQL Workbench) to kick in burst mode for "select count(1) from biometric.gps_record" t2-micro = 11.9 sec t2-small = 2.5 to 1.1 sec t2-medium = 1.6 to 0.9 secI took a T2.medium (2 threads & 4G) and modified it back to a T2.micro (1 thread @ 1G) and performance went from 1.1 sec to 10 sec as expected - as the query is memory intensive and peaks at 1700Mb from a baseline of 450Mb - therefore a T2.small (1 thread @ 2G) will sufficed until I run up 6 million records
