Saturday, December 10, 2016

Distributed in-memory Cache - Apache Ignite


Apache Ignite is highly performant because it has a deep history of distributed deployment with sql, transactional and locking support.  It was recently open sourced to Apache.

Mixed metal and VM's
[19:05:22] Ignite node started OK (id=7352a302)
[19:05:22] Topology snapshot [ver=1, servers=1, clients=0, CPUs=4, heap=12.0GB]
[19:05:32] Topology snapshot [ver=2, servers=2, clients=0, CPUs=8, heap=24.0GB]
[19:05:42] Topology snapshot [ver=3, servers=3, clients=0, CPUs=12, heap=36.0GB]
[19:05:50] Topology snapshot [ver=4, servers=4, clients=0, CPUs=16, heap=48.0GB]
[19:05:57] Topology snapshot [ver=5, servers=5, clients=0, CPUs=24, heap=60.0GB]
[19:06:07] Topology snapshot [ver=6, servers=6, clients=0, CPUs=30, heap=68.0GB]

[19:06:16] Topology snapshot [ver=7, servers=7, clients=0, CPUs=36, heap=76.0GB]

Bare Metal NUCs


VMware RHEL 7.3 Virtual Machines (bridged mode) across two physical hosts




GridGain Enterprise Edition 7.5 corresponds to Apache Ignite 1.5 in February 2016

Release 1.8.0 8 Dec 2016

Amazon Image
2015-06-25


  • Download Apache Ignite as ZIP archive from https://ignite.apache.org/
  • Unzip ZIP archive into the installation folder in your system
  • Set IGNITE_HOME environment variable to point to the installation folder and make sure there is no trailing / in the path (this step is optional)


Maven/Gradle dependencies

<dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-core</artifactId>
    <version>${ignite.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-spring</artifactId>
    <version>${ignite.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.ignite</groupId>
    <artifactId>ignite-indexing</artifactId>
    <version>${ignite.version}</version>

</dependency>

Setup


Open ports on all machines
47100 = local
48100 = shared memory

   90  wget http://apache.mirror.iweb.ca//ignite/1.8.0/apache-ignite-fabric-1.8.0-bin.zip
   91  unzip apache-ignite-fabric-1.8.0-bin.zip 

export IGNITE_HOME=/root/ignite-180


 bin/ignite.sh examples/config/example-ignite.xml

Thursday, December 1, 2016

Running High Memory Java Applications on Amazon EC2 2048Gb 128 vCPU Memory Optimized x1.32xlarge instances

Amazon has a huge instance that we can use to run massive memory models.
Here I am creating hypercube (see CM-1 from the 90's).  On a normal 64G system like a Haswell 5820K board fully populated with DDR4 we top out at 22 dimensions.


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852    3801596 2010941256         64      13396     238792
-/+ buffers/cache:    3549408 2011193444
Swap:            0          0          0



[ec2-user@ip-172-31-57-105 ~]$ java -version
java version "1.7.0_121"
OpenJDK Runtime Environment (amzn-2.6.8.1.69.amzn1-x86_64 u121-b00)
OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)
[ec2-user@ip-172-31-57-105 ~]$ sudo yum install java-1.8.0
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/latest                                                                                                                                | 2.1 kB     00:00     
amzn-updates/latest                                                                                                                             | 2.3 kB     00:00     
Resolving Dependencies
--> Running transaction check
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1 will be installed
--> Processing Dependency: java-1.8.0-openjdk-headless = 1:1.8.0.111-1.b15.25.amzn1 for package: 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64
--> Running transaction check
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1 will be installed
amzn-main/latest/filelists_db                                                                                                                   | 5.1 MB     00:00     
amzn-updates/latest/filelists_db                                                                                                                | 1.2 MB     00:00     
--> Processing Dependency: lksctp-tools for package: 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64
--> Running transaction check
---> Package lksctp-tools.x86_64 0:1.0.10-7.7.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================================================
 Package                                          Arch                        Version                                          Repository                         Size
=======================================================================================================================================================================
Installing:
 java-1.8.0-openjdk                               x86_64                      1:1.8.0.111-1.b15.25.amzn1                       amzn-updates                      227 k
Installing for dependencies:
 java-1.8.0-openjdk-headless                      x86_64                      1:1.8.0.111-1.b15.25.amzn1                       amzn-updates                       39 M
 lksctp-tools                                     x86_64                      1.0.10-7.7.amzn1                                 amzn-main                          89 k

Transaction Summary
=======================================================================================================================================================================
Install  1 Package (+2 Dependent packages)

Total download size: 39 M
Installed size: 102 M
Is this ok [y/d/N]: y
Downloading packages:
(1/3): java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64.rpm                                                                                   | 227 kB     00:00     
(2/3): java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64.rpm                                                                          |  39 MB     00:00     
(3/3): lksctp-tools-1.0.10-7.7.amzn1.x86_64.rpm                                                                                                 |  89 kB     00:00     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                   48 MB/s |  39 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : lksctp-tools-1.0.10-7.7.amzn1.x86_64                                                                                                                1/3 
  Installing : 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                       2/3 
  Installing : 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                                3/3 
  Verifying  : lksctp-tools-1.0.10-7.7.amzn1.x86_64                                                                                                                1/3 
  Verifying  : 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                                2/3 
  Verifying  : 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                       3/3 

Installed:
  java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1                                                                                                                 

Dependency Installed:
  java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1                                 lksctp-tools.x86_64 0:1.0.10-7.7.amzn1                                

Complete!

[ec2-user@ip-172-31-57-105 ~]$ sudo alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
   2           /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java

Enter to keep the current selection[+], or type selection number: 2
[ec2-user@ip-172-31-57-105 ~]$ java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-b15)
OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)


[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 22 1
22 dimensional hypercube: 1480643986605
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
47345 time: Total Ram: 8450998272 free: 2292978440

[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852   20609684 1994133168         68      14064     406412
-/+ buffers/cache:   20189208 1994553644
Swap:            0          0          0

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 22 2 -server 
22 dimensional hypercube: 1480644388217
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
43511 time: Total Ram: 8432648192 free: 2267825616

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 23 1 
23 dimensional hypercube: 1480644473000
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.
140562 time: Total Ram: 18184929280 free: 5885736464

top - 02:09:40 up  2:05,  2 users,  load average: 39.26, 21.54, 10.94
Tasks: 730 total,   1 running, 729 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 19851880k used, 1994890972k free,    14168k buffers
Swap:        0k total,        0k used,        0k free,   406416k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
  7999 ec2-user  20   0 39.9g  15g  14m S 8058.5  0.8  96:23.57 java  

Tasks: 729 total,   2 running, 727 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.8%us,  0.4%sy,  0.0%ni, 36.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 32436576k used, 1982306276k free,    14304k buffers
Swap:        0k total,        0k used,        0k free,   406416k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
  8226 ec2-user  20   0 39.8g  27g  15m S 8098.1  1.4 255:48.30 java  

[vagrant@nsp-latest workspace]$ free
              total        used        free      shared  buff/cache   available
Mem:        7495248      263396     5030584       10720     2201268     6923444
Swap:       1048572      171480      877092

[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852   33580832 1981162020         68      14372     406416
-/+ buffers/cache:   33160044 1981582808

Swap:            0          0          0


running with extra memory


[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 24 1 
24 dimensional hypercube: 1480649540504
_created 201326592 bidirectional links between 16777216 routers in the 24 dimensional HyperCube.
101381 time: Total Ram: 526849671168 free: 471873878864



[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 23 1 
23 dimensional hypercube: 1480649699358
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.

50312 time: Total Ram: 526849671168 free: 496612985560



Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 52853856k used, 1961888996k free,    18704k buffers
Swap:        0k total,        0k used,        0k free,   436572k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 43697 ec2-user  20   0  538g  46g  14m S 99.9  2.4   1:26.89 java    


[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 25 1 
25 dimensional hypercube: 1480649786242
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.

289783 time: Total Ram: 526849671168 free: 397656555744


[ec2-user@ip-172-31-57-105 ~]$ nohup java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1 &
[1] 44231
[ec2-user@ip-172-31-57-105 ~]$ ps -ef | grep java

ec2-user  44231  43947 91 03:55 pts/0    00:00:05 java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1

[ec2-user@ip-172-31-57-105 ~]$ tail -f nohup.out

2 dimensional hypercube: 1480650669281
_created 4 bidirectional links between 4 routers in the 2 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
3 dimensional hypercube: 1480650669283
_created 12 bidirectional links between 8 routers in the 3 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
4 dimensional hypercube: 1480650669283
_created 32 bidirectional links between 16 routers in the 4 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
5 dimensional hypercube: 1480650669283
_created 80 bidirectional links between 32 routers in the 5 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
6 dimensional hypercube: 1480650669284
_created 192 bidirectional links between 64 routers in the 6 dimensional HyperCube.
0 time: Total Ram: 1843972800512 free: 1826792931288
7 dimensional hypercube: 1480650669284
_created 448 bidirectional links between 128 routers in the 7 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
8 dimensional hypercube: 1480650669286
_created 1024 bidirectional links between 256 routers in the 8 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
9 dimensional hypercube: 1480650669287
_created 2304 bidirectional links between 512 routers in the 9 dimensional HyperCube.
1 time: Total Ram: 1843972800512 free: 1826792931288
10 dimensional hypercube: 1480650669288
_created 5120 bidirectional links between 1024 routers in the 10 dimensional HyperCube.
3 time: Total Ram: 1843972800512 free: 1826792931288
11 dimensional hypercube: 1480650669291
_created 11264 bidirectional links between 2048 routers in the 11 dimensional HyperCube.
5 time: Total Ram: 1843972800512 free: 1826792931288
12 dimensional hypercube: 1480650669296
_created 24576 bidirectional links between 4096 routers in the 12 dimensional HyperCube.
10 time: Total Ram: 1843972800512 free: 1826792931288
13 dimensional hypercube: 1480650669306
_created 53248 bidirectional links between 8192 routers in the 13 dimensional HyperCube.
21 time: Total Ram: 1843972800512 free: 1826792931288
14 dimensional hypercube: 1480650669327
_created 114688 bidirectional links between 16384 routers in the 14 dimensional HyperCube.
38 time: Total Ram: 1843972800512 free: 1826792931288
15 dimensional hypercube: 1480650669365
_created 245760 bidirectional links between 32768 routers in the 15 dimensional HyperCube.
82 time: Total Ram: 1843972800512 free: 1826792931288
16 dimensional hypercube: 1480650669447
_created 524288 bidirectional links between 65536 routers in the 16 dimensional HyperCube.
174 time: Total Ram: 1843972800512 free: 1826792931288
17 dimensional hypercube: 1480650669621
_created 1114112 bidirectional links between 131072 routers in the 17 dimensional HyperCube.
384 time: Total Ram: 1843972800512 free: 1826792931288
18 dimensional hypercube: 1480650670005
_created 2359296 bidirectional links between 262144 routers in the 18 dimensional HyperCube.
923 time: Total Ram: 1843972800512 free: 1826792931288
19 dimensional hypercube: 1480650670928
_created 4980736 bidirectional links between 524288 routers in the 19 dimensional HyperCube.
1970 time: Total Ram: 1843972800512 free: 1826792931288
20 dimensional hypercube: 1480650672898
_created 10485760 bidirectional links between 1048576 routers in the 20 dimensional HyperCube.
4677 time: Total Ram: 1843972800512 free: 1826792931288
21 dimensional hypercube: 1480650677576
_created 22020096 bidirectional links between 2097152 routers in the 21 dimensional HyperCube.
9739 time: Total Ram: 1843972800512 free: 1818202996656
22 dimensional hypercube: 1480650687314
_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.
21156 time: Total Ram: 1843972800512 free: 1809613062048
23 dimensional hypercube: 1480650708470
_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.
44953 time: Total Ram: 1843972800512 free: 1783843258176


Tasks: 743 total,   1 running, 742 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 55841328k used, 1958901524k free,    19244k buffers
Swap:        0k total,        0k used,        0k free,   436796k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                

 44386 ec2-user  20   0 1865g  49g  14m S 100.2  2.6   1:19.67 java

24 dimensional hypercube: 1480651245506
_created 201326592 bidirectional links between 16777216 routers in the 24 dimensional HyperCube.
101306 time: Total Ram: 1843972800512 free: 1732303649872
25 dimensional hypercube: 1480651346812
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.
264376 time: Total Ram: 1843972800512 free: 1603454628832



Tasks: 742 total,   1 running, 741 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 278517036k used, 1736225816k free,    19408k buffers
Swap:        0k total,        0k used,        0k free,   436796k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
 44386 ec2-user  20   0 1865g 261g  15m S 100.2 13.6   8:45.96 java  


26 dimensional hypercube: 1480651611188
_created 872415232 bidirectional links between 67108864 routers in the 26 dimensional HyperCube.

656891 time: Total Ram: 1843972800512 free: 1647312996576


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852  661273556 1353469296         72      19672     436804

-/+ buffers/cache:  660817080 1353925772

 44386 ec2-user  20   0 1865g 625g  15m S 99.9 32.6  82:37.39 java   


Cpu(s):  0.8%us,  0.0%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1088382060k used, 926360792k free,    19984k buffers
Swap:        0k total,        0k used,        0k free,   436804k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 44386 ec2-user  20   0 1870g 1.0t  15m S 100.2 53.7 313:16.41 java   



25 dimensional hypercube: 1480651346812
_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.
264376 time: Total Ram: 1843972800512 free: 1603454628832
26 dimensional hypercube: 1480651611188
_created 872415232 bidirectional links between 67108864 routers in the 26 dimensional HyperCube.
656891 time: Total Ram: 1843972800512 free: 1647312996576
27 dimensional hypercube: 1480652268079
_created 1811939328 bidirectional links between 134217728 routers in the 27 dimensional HyperCube.
1433987 time: Total Ram: 1843972800512 free: 1253174190840
28 dimensional hypercube: 1480653702066
_created 3758096384 bidirectional links between 268435456 routers in the 28 dimensional HyperCube.
4437315 time: Total Ram: 1843972800512 free: 1001914613768
29 dimensional hypercube: 1480658139381
[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached
Mem:    2014742852 1992699228   22043624         64      28708     437000
-/+ buffers/cache: 1992233520   22509332
Swap:            0          0          0
[ec2-user@ip-172-31-57-105 ~]$ ps -ef | grep java

ec2-user  44386      1 99 03:59 ?        18-01:29:52 java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1

Tasks: 739 total,   1 running, 738 sleeping,   0 stopped,   0 zombie
Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1992699740k used, 22043112k free,    28736k buffers
Swap:        0k total,        0k used,        0k free,   437000k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 
 44386 ec2-user  20   0 1898g 1.8t  15m S 8059.8 98.5  26052:50 java  


Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1734049420k used, 280693432k free,    32136k buffers
Swap:        0k total,        0k used,        0k free,   437292k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 45667 ec2-user  20   0 1888g 1.6t  14m S 8045.2 85.7   5836:37 java  


Tasks: 744 total,   1 running, 743 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.8%us,  0.0%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  2014742852k total, 1971207064k used, 43535788k free,    33936k buffers
Swap:        0k total,        0k used,        0k free,   437308k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 45667 ec2-user  20   0 1886g 1.8t  14m S 100.0 97.4  13311:12 java   


[ec2-user@ip-172-31-57-105 ~]$ free
             total       used       free     shared    buffers     cached

Mem:    2014742852 1971206692   43536160         64      33944     437308

29 dimensional hypercube: 1480681783260
_created 7784628224 bidirectional links between 536870912 routers in the 29 dimensional HyperCube.

16316219 time: Total Ram: 1843972800512 free: 147852676280

10 min 26d 30g
25 min 27d 75g
75 min 28d 185g
225 min 29d 460g
800 min 30d 1300g

Total Pageviews

Followers