Monday, March 25, 2019

FPGA development for parallel processing


Had a discussion with a trusted mentor of mine - spanning hardware and software home based work today - specifically about restarting FPGA work I looked at in 2010.

Move to OPENCL for FPGA based
$249
voucher for 1 year of 
60d-eval/$1k

Deprecate all non-FPGA work below

Sunday, March 10, 2019

Java 8 Lambda Parallel Streams :: Use thread safe collections

Changing code to use parallelStream from stream is not as simple as it looks - there are several design principles that need to be accounted for.

The parallel stream code in the lambda libraries wrap the ForkJoin framework from Java 7.  There are several issues that will affect performance and accuracy of your code.

Using non-threadsafe collections - this will show up as randomly varying results and modifications of Lists and Maps if they use out of the box non-threadsafe implementations like ArrayList and HashMap.  Stick to CopyOnWriteArrayList and ConcurrentHashMap.



For example


Using a small dataset for parallel operations may be slower than sequential code because of the parallel framework overhead.  Try to use data sets that are at lease 10k in size

For example - filtering on 10 Million biometric records.



Friday, March 8, 2019

Testing for Radon 222 radioactive gas levels above 100 Bq/m3 in the home as a result of the U235-Ra226-Rn222 decay chain


I have been meaning to test the radon(222) levels for a while - it looks like we have elevated gas levels in the lower airspace in the basement.  I just started preliminary testing before placing 6+ month laboratory grade sensors.  The results from the Airthings sensor even after 3 days is interesting.  The plan is to move the meter around daily to get a 3d view of the house before winter ends.

Have an ERV running.

The issue is unventilated confined lower spaces - we are looking for a number lower than 100 Bq/m(3) - but any level is bad.  A Bq is defined as 1 alpha particle (helium nucleus) per second per cubic meter.

Results:

20190307:0823 - bedroom 1st floor 1m above floor (12h)
79
20190307:2240 - basement by work area - 50cm above floor
67
20190308:0858 - basement back by furnace by crack in floor - right on the floor
666
20190308:1243 - basement back by furnace by crack in floor - on 30cm box
732



highest link

Canada specific

we need to be below 200 Bq/m(3) but even 50 is still dangerous

getting these (2) from Canada - to send in for a 90+ day check
and likely this one for continuous monitoring

causes:
Since we life on bedrock - we have a couple sources.
The cracks in the basement are over crushed stone
The water around the basement
The soil around the house - origin - and the rocks like the stepping stones.
Ventilation is critical - we should be good with floor vents on the 1st floor - it is the basement ventilation that may be an issue - our unfinished basement.


some 5 min science.
Because I always hear about how old the canadian shield is here - good for earthquake stability and no shifting of the ground - but bad for having Uranium and Gold - 
we are on 4.5 billion year old bedrock
map of ottawa - see purple, blue

Radon comes from Uranium - which is mostly made in merging neutron stars when parts of the neutron star pieces up to 1km wide (essentially a large atomic nucleus) are release from the main 10km wide star.  These pieces because they are under a lot less gravitational force form normal atomic matter - they condense into less dense isotopes like U235 and even gold/silver.  A smaller portion of elements higher than iron are made during the bounce off the core of the collapse of a supernova in the last minute of a large > 20 solar mass start stops burning silicon.  

Radon https://www.radon.com/radon_facts/ turns into lead slowly - https://www.researchgate.net/figure/The-Basic-Radon-222-Rn-Decay-Chain-The-isotopes-and-their-atomic-masses-are-shown_fig1_51026112 is 9 times heavier than air and accumulates at the bottom of anything - it comes not necessarily out of the basement cracks - more from the sides of the basement from the dirt
Cause of radium in the soil
Radon (originally from decayed Uranium) is released during the recoil of an alpha particle decay (essentially a helium nucleus) from Radium 226 - the alpha particle itself is destructive radiation for cells - but it is the ejected radon atom as it moves from radium 226 through radon 222 (ejecting a helium atom) that causes issues for us - until radon 222 decays with a half life of 4 days into Polonium (very toxic). Polonium decays within 3 min into ordinary lead.  Even though Radon 222 has a small half life of 4 days - it is under constant replacement because it is a gas and being ejected at a rate of 10-50 percent from the pores inside soil when the ejected radon is near the surface and the vector of the alpha particle emission is in the right angle - https://certmapper.cr.usgs.gov/data/PubArchives/radon/georadon/3.html




Links
https://davidsuzuki.org/wp-content/uploads/2017/09/revisiting-canada-radon-guideline.pdf 

Friday, January 25, 2019

Experiment 1901 Sending Packets around the Earth

    I know this sounds weird but not to a science geek.  I would like to make sure my REST calls propagate around the globe - not just to a server on the other side of the earth like from us-east-1 to the Singapore region but through us-west-2 through singapore - to europe and back to us-east-1.
    Sort of when I fly - in order to maximize the number of takeoffs and landings - I try to get the most number of stops - a direct flight is not the best way to get somewhere when testing latency - as in packets on the cable from Oregon to China are better routed through Hawaii.
    I liked Andrew Blum's "Tubes" book - which include details on the endpoints for sub-marine cables - https://www.amazon.com/Tubes-Journey-Internet-Andrew-Blum/dp/0061994952  - I also ran into a bit of this propagation delay working at Nokia.

    First principles will be install proxies at several regions in Amazon EC2 - https://aws.amazon.com/about-aws/global-infrastructure/

    see https://www.submarinecablemap.com/
This is the path we will take - hopefully no backtracking and ideally it is roughly valid - as we do not actually cross the equator - I will also try through Sau Paulo.
 
through the following Amazon regions

  AWS Regions

Propagation order

Thursday, November 8, 2018

Bluetooth 5 LE Mobile Rest client on iOS 12 with a backend JAX-RS JPA 2 Java cloud service on AWS

Bluetooth 4 LE Mobile Rest client on iOS 12/3 with a backend JAX-RS JPA 2 Java cloud service on AWS
# Overview
- Full phone metric capture - GPS, Gravity, Accell, Orientation, Magnetic field, pressure, altitude, ambience, heading
- Additional device capture - 2 HRM bluetooth LE metrics - 2A37
- Derived metric capture - Geohash, speed

# JIRAs
Older system : https://obrienlabs.atlassian.net/projects/BIOM/issues/BIOM-2?filter=allopenissues
https://github.com/obrienlabs/biometric/issues

# Features
- Live google maps/HRM browser based user tracking
- KML google earth extract
- CSV extract
- XML extract












Monday, September 18, 2017

Docker and Kubernetes DevOps


NAT settings for OSX Fusion 8.5


sudo vi /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf
[incomingtcp]

# Use these with care - anyone can enter into your VM through these...
# The format and example are as follows:
#<external port number> = <VM's IP address>:<VM's port number>
#8080 = 172.16.3.128:80

8880 = 192.168.241.134:8880

$ sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --stop
Stopped DHCP service on vmnet1
Disabled hostonly virtual adapter on vmnet1
Stopped DHCP service on vmnet8
Stopped NAT service on vmnet8
Disabled hostonly virtual adapter on vmnet8
Stopped all configured services on all networks
$ sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --start
Enabled hostonly virtual adapter on vmnet1
Started DHCP service on vmnet1
Started NAT service on vmnet8
Enabled hostonly virtual adapter on vmnet8
Started DHCP service on vmnet8
Started all configured services on all networks
$ curl 127.0.0.1:8880

{"id":"v1","type":"apiVersion","links":{"accounts":"http:\/\/127.0.0.1:8880\/v1\/accounts","agents":"http:\/\/127.0.0.1:8880\/v1\/agents","apiKeys":"http:\/\/127.0.0.1:8880\/

Running a VNC ubuntu VM inside the namespace

for root just run the following line and skip to next section
curl https://releases.rancher.com/install-docker/1.12.sh | sh
# when running as non-root (ubuntu) run the following and logout/log back in
sudo usermod -aG docker ubuntu

Also we are using this vnc-portal in production - been vetted for 4 months


his docker image is on


docker run -it --rm -p 6080:80 dorowu/ubuntu-desktop-lxde-vnc
access it with


if you ctrl-c then the docker will exit - it is stateless

Sunday, March 19, 2017

Optimizing your Macbook Pro 15 2016 for development - minimize touchbar interference


This review is for the Late 2016 Macbook Pro 15 inch with the upgraded 2.9GHz i7-6920HQ, the 1TB SSD and the ATI Radeon Pro 460 4Gb and the touchbar.

Sometimes I get asked why I just don't use a high end Lenovo (with a Xeon and 64G ram).  The issue is I also need iOS development via XCode.  So for now I need to put up with 16G ram until 32G comes out.

Most hard core users of the macbook pro will benefit from the workarounds below - especially working around the new touchbar - by mostly disabling it.

After working with the laptop for home and office development - I can say that it is a better machine than the previous generation.  The improvements outweigh the discrepancies at a bit higher price.

The HD is really fast - faster than either my raid0 4 drive Samsung 850 pro or my Intel 750.  The screen is brighter, the CPU is 23% faster under full load (12% because of clock speed), the GPU is twice as fast.

The keys are louder and bit clunky, but the biggest issue is the touchbar - it must be disabled!.
As soon as you start using the machine in its default config - your finger will slide off the delete key - where it will start siri (all the time).  The fix for the touchbar is to remove all the icons by default and only enable the function keys for specific applications.  The rest of the time only the escape key shows.  When the mac touchbar is disabled like this then when your fingers - just touch above the number keys - you wont invoke the touchbar.

Summary

0 - The F4 key rarely works - this is a Safari issue though

1 - Globally disable the touchbar except for the escape key for all non-development apps


2 - Enable the function keys for development apps like Eclipse and IntelliJ





Macbook 2016 Improvements

The 2016 i7-6920HZ CPU runs 23% faster than the my late 2013 i7-4960 (12% of this is due to a clock speed jump from 2.6 to 2.9 GHz)
The GPU is faster - although this is hard to measure because I went from an NVidia 750GT to an ATI 460

Macbook 2016 Issues

The keyboard is lower quality than the 2015 model
The GPU is ATI instead of NVidia - which locks us out of must CUDA based libraries like TensorFlow leaving us with OpenCL
The touchbar causes multiple issues - the biggest one is launching siri or changing sound/screen inadvertently by accidentally touching the bar above the number row.

Minimize Macbook Pro 2016 Touchbar effects

The first thing you will notice when using the 2016 macbook will be how the touchbar causes issues north of the delete key when your fingers slide off the northbound part of the key.
The goal is to disable the touchbar for most apps leaving just the escape key.



For development apps that require function keys like the F3/F4 (navigation)and F5/F6/F7/F8 (debug) functions in Eclipse - enable just the function key view.

Here in eclipse we frequently need F3 to load a class and F4 to get the inheritance heirarchy

During a debug session we use F5 to enter, F6 to run, F7 to skip and F8 to resume



CPU and GPU performance

On average GPU speed for the 2016 macbook pro (top model) is 12x the cpu speed (all 4 cores)
Surprisingly the embedded GPU in the CPU runs slower than the embedded Iris GPU in my old 2013 macbook pro.

OpenCL

Single Core CPU = 13
Multicore 4 CPU = 50
Embedded HD 530 GPU = 128
ATI HD 460 GPU = 505

Java

Connectors

I'll get into theUSB-C connectors shortly

Cases

Get the Thule case for your Macbook Pro 15 https://www.thule.com/en-us/ca/computer-phone-cases/laptop-cases/thule-gauntlet-30-macbook-pro-sleeve-15-_-tl_85854234856

Noise

The new keyboard is louder - try to type quieter in an office setting

Thursday, February 2, 2017

Deep Learning Experiments

Goal: 

Ramp up on Deep Learning techniques by building a basic ANN to classify radar images for example

Get some background in deep learning (a specialized subset of machine learning concentrated on the "Greedy Layer-Wise Unsupervised Pretraining procedure" - Hinton, 2006 - University of Toronto) and the "Long Short-Term Memory model - Hochreiter and Schmidhuber 1997".
Design/Theory Preparation
Get the following Nov 2016 MIT book from Amazon by Ian Goodfellow, Yoshua Benglo, Aaron Courville - (one for work, and one for home - as it is usually out of stock).  Review your linear transformation and matrix math to prep.

References:

Deep Learning, 2016, Ian Goodfellow, Yoshua Benglo, Aaron Courville

http://127.0.0.1:20959/doc/manual/R-intro.html#Related-software-and-documentation
https://www.microsoft.com/cognitive-services/en-us/sign-up
https://deeplearning4j.org/devguide
https://openai.com/blog/
https://github.com/obriensystems/requests-for-research
https://deepfield.com/cloud-genome/


Saturday, December 10, 2016

Distributed in-memory Cache - Apache Ignite


Apache Ignite is highly performant because it has a deep history of distributed deployment with sql, transactional and locking support.  It was recently open sourced to Apache.

Mixed metal and VM's
[19:05:22] Ignite node started OK (id=7352a302)
[19:05:22] Topology snapshot [ver=1, servers=1, clients=0, CPUs=4, heap=12.0GB]
[19:05:32] Topology snapshot [ver=2, servers=2, clients=0, CPUs=8, heap=24.0GB]
[19:05:42] Topology snapshot [ver=3, servers=3, clients=0, CPUs=12, heap=36.0GB]
[19:05:50] Topology snapshot [ver=4, servers=4, clients=0, CPUs=16, heap=48.0GB]
[19:05:57] Topology snapshot [ver=5, servers=5, clients=0, CPUs=24, heap=60.0GB]
[19:06:07] Topology snapshot [ver=6, servers=6, clients=0, CPUs=30, heap=68.0GB]

[19:06:16] Topology snapshot [ver=7, servers=7, clients=0, CPUs=36, heap=76.0GB]

Bare Metal NUCs


VMware RHEL 7.3 Virtual Machines (bridged mode) across two physical hosts




GridGain Enterprise Edition 7.5 corresponds to Apache Ignite 1.5 in February 2016

Release 1.8.0 8 Dec 2016

Amazon Image
2015-06-25


  • Download Apache Ignite as ZIP archive from https://ignite.apache.org/
  • Unzip ZIP archive into the installation folder in your system
  • Set IGNITE_HOME environment variable to point to the installation folder and make sure there is no trailing / in the path (this step is optional)


Maven/Gradle dependencies

<dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-core</artifactId>
    <version>${ignite.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-spring</artifactId>
    <version>${ignite.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-indexing</artifactId>
    <version>${ignite.version}</version>

</dependency>

Setup


Open ports on all machines
47100 = local
48100 = shared memory

   90  wget http://apache.mirror.iweb.ca//ignite/1.8.0/apache-ignite-fabric-1.8.0-bin.zip
   91  unzip apache-ignite-fabric-1.8.0-bin.zip 

export IGNITE_HOME=/root/ignite-180


 bin/ignite.sh examples/config/example-ignite.xml

Thursday, December 1, 2016

Running High Memory Java Applications on Amazon EC2 2048Gb 128 vCPU Memory Optimized x1.32xlarge instances

Amazon has a huge instance that we can use to run massive memory models.
Here I am creating hypercube (see CM-1 from the 90's).  On a normal 64G system like a Haswell 5820K board fully populated with DDR4 we top out at 22 dimensions.


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852    3801596 2010941256         64      13396     238792
-/+ buffers/cache:    3549408 2011193444
Swap:            0          0          0



[ec2-user@ip-172-31-57-105 ~]$ java -version
java version "1.7.0_121"
OpenJDK Runtime Environment (amzn-2.6.8.1.69.amzn1-x86_64 u121-b00)
OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)
[ec2-user@ip-172-31-57-105 ~]$ sudo yum install java-1.8.0
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/latest                                                                                                                                | 2.1 kB     00:00     
amzn-updates/latest                                                                                                                             | 2.3 kB     00:00     
Resolving Dependencies
--> Running transaction check
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1 will be installed
--> Processing Dependency: java-1.8.0-openjdk-headless = 1:1.8.0.111-1.b15.25.amzn1 for package: 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64
--> Running transaction check
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1 will be installed
amzn-main/latest/filelists_db                                                                                                                   | 5.1 MB     00:00     
amzn-updates/latest/filelists_db                                                                                                                | 1.2 MB     00:00     
--> Processing Dependency: lksctp-tools for package: 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64
--> Running transaction check
---> Package lksctp-tools.x86_64 0:1.0.10-7.7.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================================================
 Package                                          Arch                        Version                                          Repository                         Size
=======================================================================================================================================================================
Installing:
 java-1.8.0-openjdk                               x86_64                      1:1.8.0.111-1.b15.25.amzn1                       amzn-updates                      227 k
Installing for dependencies:
 java-1.8.0-openjdk-headless                      x86_64                      1:1.8.0.111-1.b15.25.amzn1                       amzn-updates                       39 M
 lksctp-tools                                     x86_64                      1.0.10-7.7.amzn1                                 amzn-main                          89 k

Transaction Summary
=======================================================================================================================================================================
Install  1 Package (+2 Dependent packages)

Total download size: 39 M
Installed size: 102 M
Is this ok [y/d/N]: y
Downloading packages:
(1/3): java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64.rpm                                                                                   | 227 kB     00:00     
(2/3): java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64.rpm                                                                          |  39 MB     00:00     
(3/3): lksctp-tools-1.0.10-7.7.amzn1.x86_64.rpm                                                                                                 |  89 kB     00:00     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                   48 MB/s |  39 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : lksctp-tools-1.0.10-7.7.amzn1.x86_64                                                                                                                1/3 
  Installing : 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                       2/3 
  Installing : 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                                3/3 
  Verifying  : lksctp-tools-1.0.10-7.7.amzn1.x86_64                                                                                                                1/3 
  Verifying  : 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                                2/3 
  Verifying  : 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                       3/3 

Installed:
  java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1                                                                                                                 

Dependency Installed:
  java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1                                 lksctp-tools.x86_64 0:1.0.10-7.7.amzn1                                

Complete!

[ec2-user@ip-172-31-57-105 ~]$ sudo alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
   2           /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java

Enter to keep the current selection[+], or type selection number: 2
[ec2-user@ip-172-31-57-105 ~]$ java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)


[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 22 1
22 dimensional hypercube: 1480643986605
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
47345 time: Total Ram: 8450998272 free: 2292978440

[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852   20609684 1994133168         68      14064     406412
-/+ buffers/cache:   20189208 1994553644
Swap:            0          0          0

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 22 2 -server 
22 dimensional hypercube: 1480644388217
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
43511 time: Total Ram: 8432648192 free: 2267825616

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 23 1 
23 dimensional hypercube: 1480644473000
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.
140562 time: Total Ram: 18184929280 free: 5885736464

top - 02:09:40 up  2:05,  2 users,  load average: 39.26, 21.54, 10.94
Tasks: 730 total,   1 running, 729 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 19851880k used, 1994890972k free,    14168k buffers
Swap:        0k total,        0k used,        0k free,   406416k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
  7999 ec2-user  20   0 39.9g  15g  14m S 8058.5  0.8  96:23.57 java  

Tasks: 729 total,   2 running, 727 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.8%us,  0.4%sy,  0.0%ni, 36.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 32436576k used, 1982306276k free,    14304k buffers
Swap:        0k total,        0k used,        0k free,   406416k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
  8226 ec2-user  20   0 39.8g  27g  15m S 8098.1  1.4 255:48.30 java  

[vagrant@nsp-latest workspace]$ free
              total        used        free      shared  buff/cache   available
Mem:        7495248      263396     5030584       10720     2201268     6923444
Swap:       1048572      171480      877092

[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852   33580832 1981162020         68      14372     406416
-/+ buffers/cache:   33160044 1981582808

Swap:            0          0          0


running with extra memory


[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 24 1 
24 dimensional hypercube: 1480649540504
_created 201326592 bidirectional links between 16777216 routers in the 24 dimensional HyperCube.
101381 time: Total Ram: 526849671168 free: 471873878864



[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 23 1 
23 dimensional hypercube: 1480649699358
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.

50312 time: Total Ram: 526849671168 free: 496612985560



Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 52853856k used, 1961888996k free,    18704k buffers
Swap:        0k total,        0k used,        0k free,   436572k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 43697 ec2-user  20   0  538g  46g  14m S 99.9  2.4   1:26.89 java    


[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 25 1 
25 dimensional hypercube: 1480649786242
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.

289783 time: Total Ram: 526849671168 free: 397656555744


[ec2-user@ip-172-31-57-105 ~]$ nohup java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1 &
[1] 44231
[ec2-user@ip-172-31-57-105 ~]$ ps -ef | grep java

ec2-user  44231  43947 91 03:55 pts/0    00:00:05 java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1

[ec2-user@ip-172-31-57-105 ~]$ tail -f nohup.out

2 dimensional hypercube: 1480650669281
_created 4 bidirectional links between 4 routers in the 2 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
3 dimensional hypercube: 1480650669283
_created 12 bidirectional links between 8 routers in the 3 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
4 dimensional hypercube: 1480650669283
_created 32 bidirectional links between 16 routers in the 4 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
5 dimensional hypercube: 1480650669283
_created 80 bidirectional links between 32 routers in the 5 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
6 dimensional hypercube: 1480650669284
_created 192 bidirectional links between 64 routers in the 6 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
7 dimensional hypercube: 1480650669284
_created 448 bidirectional links between 128 routers in the 7 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
8 dimensional hypercube: 1480650669286
_created 1024 bidirectional links between 256 routers in the 8 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
9 dimensional hypercube: 1480650669287
_created 2304 bidirectional links between 512 routers in the 9 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
10 dimensional hypercube: 1480650669288
_created 5120 bidirectional links between 1024 routers in the 10 dimensional HyperCube.
3 time: Total Ram: 1843972800512 free: 1826792931288
11 dimensional hypercube: 1480650669291
_created 11264 bidirectional links between 2048 routers in the 11 dimensional HyperCube.
5 time: Total Ram: 1843972800512 free: 1826792931288
12 dimensional hypercube: 1480650669296
_created 24576 bidirectional links between 4096 routers in the 12 dimensional HyperCube.
10 time: Total Ram: 1843972800512 free: 1826792931288
13 dimensional hypercube: 1480650669306
_created 53248 bidirectional links between 8192 routers in the 13 dimensional HyperCube.
21 time: Total Ram: 1843972800512 free: 1826792931288
14 dimensional hypercube: 1480650669327
_created 114688 bidirectional links between 16384 routers in the 14 dimensional HyperCube.
38 time: Total Ram: 1843972800512 free: 1826792931288
15 dimensional hypercube: 1480650669365
_created 245760 bidirectional links between 32768 routers in the 15 dimensional HyperCube.
82 time: Total Ram: 1843972800512 free: 1826792931288
16 dimensional hypercube: 1480650669447
_created 524288 bidirectional links between 65536 routers in the 16 dimensional HyperCube.
174 time: Total Ram: 1843972800512 free: 1826792931288
17 dimensional hypercube: 1480650669621
_created 1114112 bidirectional links between 131072 routers in the 17 dimensional HyperCube.
384 time: Total Ram: 1843972800512 free: 1826792931288
18 dimensional hypercube: 1480650670005
_created 2359296 bidirectional links between 262144 routers in the 18 dimensional HyperCube.
923 time: Total Ram: 1843972800512 free: 1826792931288
19 dimensional hypercube: 1480650670928
_created 4980736 bidirectional links between 524288 routers in the 19 dimensional HyperCube.
1970 time: Total Ram: 1843972800512 free: 1826792931288
20 dimensional hypercube: 1480650672898
_created 10485760 bidirectional links between 1048576 routers in the 20 dimensional HyperCube.
4677 time: Total Ram: 1843972800512 free: 1826792931288
21 dimensional hypercube: 1480650677576
_created 22020096 bidirectional links between 2097152 routers in the 21 dimensional HyperCube.
9739 time: Total Ram: 1843972800512 free: 1818202996656
22 dimensional hypercube: 1480650687314
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
21156 time: Total Ram: 1843972800512 free: 1809613062048
23 dimensional hypercube: 1480650708470
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.
44953 time: Total Ram: 1843972800512 free: 1783843258176


Tasks: 743 total,   1 running, 742 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 55841328k used, 1958901524k free,    19244k buffers
Swap:        0k total,        0k used,        0k free,   436796k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                

 44386 ec2-user  20   0 1865g  49g  14m S 100.2  2.6   1:19.67 java

24 dimensional hypercube: 1480651245506
_created 201326592 bidirectional links between 16777216 routers in the 24 dimensional HyperCube.
101306 time: Total Ram: 1843972800512 free: 1732303649872
25 dimensional hypercube: 1480651346812
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.
264376 time: Total Ram: 1843972800512 free: 1603454628832



Tasks: 742 total,   1 running, 741 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 278517036k used, 1736225816k free,    19408k buffers
Swap:        0k total,        0k used,        0k free,   436796k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
 44386 ec2-user  20   0 1865g 261g  15m S 100.2 13.6   8:45.96 java  


26 dimensional hypercube: 1480651611188
_created 872415232 bidirectional links between 67108864 routers in the 26 dimensional HyperCube.

656891 time: Total Ram: 1843972800512 free: 1647312996576


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852  661273556 1353469296         72      19672     436804

-/+ buffers/cache:  660817080 1353925772

 44386 ec2-user  20   0 1865g 625g  15m S 99.9 32.6  82:37.39 java   


Cpu(s):  0.8%us,  0.0%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1088382060k used, 926360792k free,    19984k buffers
Swap:        0k total,        0k used,        0k free,   436804k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 44386 ec2-user  20   0 1870g 1.0t  15m S 100.2 53.7 313:16.41 java   



25 dimensional hypercube: 1480651346812
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.
264376 time: Total Ram: 1843972800512 free: 1603454628832
26 dimensional hypercube: 1480651611188
_created 872415232 bidirectional links between 67108864 routers in the 26 dimensional HyperCube.
656891 time: Total Ram: 1843972800512 free: 1647312996576
27 dimensional hypercube: 1480652268079
_created 1811939328 bidirectional links between 134217728 routers in the 27 dimensional HyperCube.
1433987 time: Total Ram: 1843972800512 free: 1253174190840
28 dimensional hypercube: 1480653702066
_created 3758096384 bidirectional links between 268435456 routers in the 28 dimensional HyperCube.
4437315 time: Total Ram: 1843972800512 free: 1001914613768
29 dimensional hypercube: 1480658139381
[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852 1992699228   22043624         64      28708     437000
-/+ buffers/cache: 1992233520   22509332
Swap:            0          0          0
[ec2-user@ip-172-31-57-105 ~]$ ps -ef | grep java

ec2-user  44386      1 99 03:59 ?        18-01:29:52 java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1

Tasks: 739 total,   1 running, 738 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1992699740k used, 22043112k free,    28736k buffers
Swap:        0k total,        0k used,        0k free,   437000k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
 44386 ec2-user  20   0 1898g 1.8t  15m S 8059.8 98.5  26052:50 java  


Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1734049420k used, 280693432k free,    32136k buffers
Swap:        0k total,        0k used,        0k free,   437292k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 45667 ec2-user  20   0 1888g 1.6t  14m S 8045.2 85.7   5836:37 java  


Tasks: 744 total,   1 running, 743 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  0.0%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1971207064k used, 43535788k free,    33936k buffers
Swap:        0k total,        0k used,        0k free,   437308k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 45667 ec2-user  20   0 1886g 1.8t  14m S 100.0 97.4  13311:12 java   


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached

Mem:    2014742852 1971206692   43536160         64      33944     437308

29 dimensional hypercube: 1480681783260
_created 7784628224 bidirectional links between 536870912 routers in the 29 dimensional HyperCube.

16316219 time: Total Ram: 1843972800512 free: 147852676280

10 min 26d 30g
25 min 27d 75g
75 min 28d 185g
225 min 29d 460g
800 min 30d 1300g

Total Pageviews

Followers