Distributed in-memory Cache - Apache Ignite

Apache Ignite is highly performant because it has a deep history of distributed deployment with sql, transactional and locking support.  It was recently open sourced to Apache.

Mixed metal and VM's

[19:05:22] Ignite node started OK (id=7352a302)

[19:05:22] Topology snapshot [ver=1, servers=1, clients=0, CPUs=4, heap=12.0GB]

[19:05:32] Topology snapshot [ver=2, servers=2, clients=0, CPUs=8, heap=24.0GB]

[19:05:42] Topology snapshot [ver=3, servers=3, clients=0, CPUs=12, heap=36.0GB]

[19:05:50] Topology snapshot [ver=4, servers=4, clients=0, CPUs=16, heap=48.0GB]

[19:05:57] Topology snapshot [ver=5, servers=5, clients=0, CPUs=24, heap=60.0GB]

[19:06:07] Topology snapshot [ver=6, servers=6, clients=0, CPUs=30, heap=68.0GB]

[19:06:16] Topology snapshot [ver=7, servers=7, clients=0, CPUs=36, heap=76.0GB]

Bare Metal NUCs

VMware RHEL 7.3 Virtual Machines (bridged mode) across two physical hosts

https://dzone.com/articles/deadlock-free-transactions-with-apache-ignite

GridGain Enterprise Edition 7.5 corresponds to Apache Ignite 1.5 in February 2016

https://ignite.apache.org/

Release 1.8.0 8 Dec 2016

Amazon Image

guide

2015-06-25

us-west:ami-51fb0d15, us-east:ami-0f824764, eu-central:ami-0a4d7617

https://apacheignite.readme.io/docs/docker-deployment#amazon-ec2-deployment

https://ignite.apache.org/use-cases/caching/hibernate-l2-cache.html

https://imcsummit.org/2016/videos-and-slides/

• Download Apache Ignite as ZIP archive from https://ignite.apache.org/
• Unzip ZIP archive into the installation folder in your system
• Set IGNITE_HOME environment variable to point to the installation folder and make sure there is no trailing / in the path (this step is optional)

https://dzone.com/articles/running-microservices-on-top-of-in-memory-data-gri

Maven/Gradle dependencies

<dependency>

    <groupId>org.apache.ignite</groupId>

    <artifactId>ignite-core</artifactId>

    <version>${ignite.version}</version>

</dependency>

<dependency>

    <groupId>org.apache.ignite</groupId>

    <artifactId>ignite-spring</artifactId>

    <version>${ignite.version}</version>

</dependency>

<dependency>

    <groupId>org.apache.ignite</groupId>

    <artifactId>ignite-indexing</artifactId>

    <version>${ignite.version}</version>

</dependency>

Setup

Open ports on all machines

47100 = local

48100 = shared memory

   90  wget http://apache.mirror.iweb.ca//ignite/1.8.0/apache-ignite-fabric-1.8.0-bin.zip

   91  unzip apache-ignite-fabric-1.8.0-bin.zip 

export IGNITE_HOME=/root/ignite-180

 bin/ignite.sh examples/config/example-ignite.xml

Running High Memory Java Applications on Amazon EC2 2048Gb 128 vCPU Memory Optimized x1.32xlarge instances

Amazon has a huge instance that we can use to run massive memory models.
Here I am creating hypercube (see CM-1 from the 90's).  On a normal 64G system like a Haswell 5820K board fully populated with DDR4 we top out at 22 dimensions.

[ec2-user@ip-172-31-57-105 ~]$ free

             total       used       free     shared    buffers     cached

Mem:    2014742852    3801596 2010941256         64      13396     238792

-/+ buffers/cache:    3549408 2011193444

Swap:            0          0          0

[ec2-user@ip-172-31-57-105 ~]$ java -version

java version "1.7.0_121"

OpenJDK Runtime Environment (amzn-2.6.8.1.69.amzn1-x86_64 u121-b00)

OpenJDK 64-Bit Server VM (build 24.121-b00, mixed mode)

[ec2-user@ip-172-31-57-105 ~]$ sudo yum install java-1.8.0

Loaded plugins: priorities, update-motd, upgrade-helper

amzn-main/latest                                                                                                                                | 2.1 kB     00:00     

amzn-updates/latest                                                                                                                             | 2.3 kB     00:00     

Resolving Dependencies

--> Running transaction check

---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1 will be installed

--> Processing Dependency: java-1.8.0-openjdk-headless = 1:1.8.0.111-1.b15.25.amzn1 for package: 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64

--> Running transaction check

---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1 will be installed

amzn-main/latest/filelists_db                                                                                                                   | 5.1 MB     00:00     

amzn-updates/latest/filelists_db                                                                                                                | 1.2 MB     00:00     

--> Processing Dependency: lksctp-tools for package: 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64

--> Running transaction check

---> Package lksctp-tools.x86_64 0:1.0.10-7.7.amzn1 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================================================

 Package                                          Arch                        Version                                          Repository                         Size

=======================================================================================================================================================================

Installing:

 java-1.8.0-openjdk                               x86_64                      1:1.8.0.111-1.b15.25.amzn1                       amzn-updates                      227 k

Installing for dependencies:

 java-1.8.0-openjdk-headless                      x86_64                      1:1.8.0.111-1.b15.25.amzn1                       amzn-updates                       39 M

 lksctp-tools                                     x86_64                      1.0.10-7.7.amzn1                                 amzn-main                          89 k

Transaction Summary

=======================================================================================================================================================================

Install  1 Package (+2 Dependent packages)

Total download size: 39 M

Installed size: 102 M

Is this ok [y/d/N]: y

Downloading packages:

(1/3): java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64.rpm                                                                                   | 227 kB     00:00     

(2/3): java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64.rpm                                                                          |  39 MB     00:00     

(3/3): lksctp-tools-1.0.10-7.7.amzn1.x86_64.rpm                                                                                                 |  89 kB     00:00     

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

Total                                                                                                                                   48 MB/s |  39 MB  00:00:00     

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Installing : lksctp-tools-1.0.10-7.7.amzn1.x86_64                                                                                                                1/3 

  Installing : 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                       2/3 

  Installing : 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                                3/3 

  Verifying  : lksctp-tools-1.0.10-7.7.amzn1.x86_64                                                                                                                1/3 

  Verifying  : 1:java-1.8.0-openjdk-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                                2/3 

  Verifying  : 1:java-1.8.0-openjdk-headless-1.8.0.111-1.b15.25.amzn1.x86_64                                                                                       3/3 

Installed:

  java-1.8.0-openjdk.x86_64 1:1.8.0.111-1.b15.25.amzn1                                                                                                                 

Dependency Installed:

  java-1.8.0-openjdk-headless.x86_64 1:1.8.0.111-1.b15.25.amzn1                                 lksctp-tools.x86_64 0:1.0.10-7.7.amzn1                                

Complete!

[ec2-user@ip-172-31-57-105 ~]$ sudo alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command

-----------------------------------------------

*+ 1           /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java

   2           /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java

Enter to keep the current selection[+], or type selection number: 2

[ec2-user@ip-172-31-57-105 ~]$ java -version

openjdk version "1.8.0_111"

OpenJDK Runtime Environment (build 1.8.0_111-b15)

OpenJDK 64-Bit Server VM (build 25.111-b15, mixed mode)

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 22 1

22 dimensional hypercube: 1480643986605

_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.

47345 time: Total Ram: 8450998272 free: 2292978440

[ec2-user@ip-172-31-57-105 ~]$ free

             total       used       free     shared    buffers     cached

Mem:    2014742852   20609684 1994133168         68      14064     406412

-/+ buffers/cache:   20189208 1994553644

Swap:            0          0          0

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 22 2 -server 

22 dimensional hypercube: 1480644388217

_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.

43511 time: Total Ram: 8432648192 free: 2267825616

[ec2-user@ip-172-31-57-105 ~]$ java -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 23 1 

23 dimensional hypercube: 1480644473000

_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.

140562 time: Total Ram: 18184929280 free: 5885736464

top - 02:09:40 up  2:05,  2 users,  load average: 39.26, 21.54, 10.94

Tasks: 730 total,   1 running, 729 sleeping,   0 stopped,   0 zombie

Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  2014742852k total, 19851880k used, 1994890972k free,    14168k buffers

Swap:        0k total,        0k used,        0k free,   406416k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

  7999 ec2-user  20   0 39.9g  15g  14m S 8058.5  0.8  96:23.57 java  

Tasks: 729 total,   2 running, 727 sleeping,   0 stopped,   0 zombie

Cpu(s): 62.8%us,  0.4%sy,  0.0%ni, 36.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  2014742852k total, 32436576k used, 1982306276k free,    14304k buffers

Swap:        0k total,        0k used,        0k free,   406416k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

  8226 ec2-user  20   0 39.8g  27g  15m S 8098.1  1.4 255:48.30 java  

[vagrant@nsp-latest workspace]$ free

              total        used        free      shared  buff/cache   available

Mem:        7495248      263396     5030584       10720     2201268     6923444

Swap:       1048572      171480      877092

[ec2-user@ip-172-31-57-105 ~]$ free

             total       used       free     shared    buffers     cached

Mem:    2014742852   33580832 1981162020         68      14372     406416

-/+ buffers/cache:   33160044 1981582808

Swap:            0          0          0


running with extra memory

[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 24 1 

24 dimensional hypercube: 1480649540504

_created 201326592 bidirectional links between 16777216 routers in the 24 dimensional HyperCube.

101381 time: Total Ram: 526849671168 free: 471873878864

[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 23 1 

23 dimensional hypercube: 1480649699358

_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.

50312 time: Total Ram: 526849671168 free: 496612985560

Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  2014742852k total, 52853856k used, 1961888996k free,    18704k buffers

Swap:        0k total,        0k used,        0k free,   436572k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 43697 ec2-user  20   0  538g  46g  14m S 99.9  2.4   1:26.89 java    

[ec2-user@ip-172-31-57-105 ~]$ java -Xmx512g -Xms512g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 25 1 

25 dimensional hypercube: 1480649786242

_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.

289783 time: Total Ram: 526849671168 free: 397656555744

[ec2-user@ip-172-31-57-105 ~]$ nohup java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1 &

[1] 44231

[ec2-user@ip-172-31-57-105 ~]$ ps -ef | grep java

ec2-user  44231  43947 91 03:55 pts/0    00:00:05 java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1

[ec2-user@ip-172-31-57-105 ~]$ tail -f nohup.out

2 dimensional hypercube: 1480650669281

_created 4 bidirectional links between 4 routers in the 2 dimensional HyperCube.

1 time: Total Ram: 1843972800512 free: 1826792931288

3 dimensional hypercube: 1480650669283

_created 12 bidirectional links between 8 routers in the 3 dimensional HyperCube.

0 time: Total Ram: 1843972800512 free: 1826792931288

4 dimensional hypercube: 1480650669283

_created 32 bidirectional links between 16 routers in the 4 dimensional HyperCube.

0 time: Total Ram: 1843972800512 free: 1826792931288

5 dimensional hypercube: 1480650669283

_created 80 bidirectional links between 32 routers in the 5 dimensional HyperCube.

1 time: Total Ram: 1843972800512 free: 1826792931288

6 dimensional hypercube: 1480650669284

_created 192 bidirectional links between 64 routers in the 6 dimensional HyperCube.

0 time: Total Ram: 1843972800512 free: 1826792931288

7 dimensional hypercube: 1480650669284

_created 448 bidirectional links between 128 routers in the 7 dimensional HyperCube.

1 time: Total Ram: 1843972800512 free: 1826792931288

8 dimensional hypercube: 1480650669286

_created 1024 bidirectional links between 256 routers in the 8 dimensional HyperCube.

1 time: Total Ram: 1843972800512 free: 1826792931288

9 dimensional hypercube: 1480650669287

_created 2304 bidirectional links between 512 routers in the 9 dimensional HyperCube.

1 time: Total Ram: 1843972800512 free: 1826792931288

10 dimensional hypercube: 1480650669288

_created 5120 bidirectional links between 1024 routers in the 10 dimensional HyperCube.

3 time: Total Ram: 1843972800512 free: 1826792931288

11 dimensional hypercube: 1480650669291

_created 11264 bidirectional links between 2048 routers in the 11 dimensional HyperCube.

5 time: Total Ram: 1843972800512 free: 1826792931288

12 dimensional hypercube: 1480650669296

_created 24576 bidirectional links between 4096 routers in the 12 dimensional HyperCube.

10 time: Total Ram: 1843972800512 free: 1826792931288

13 dimensional hypercube: 1480650669306

_created 53248 bidirectional links between 8192 routers in the 13 dimensional HyperCube.

21 time: Total Ram: 1843972800512 free: 1826792931288

14 dimensional hypercube: 1480650669327

_created 114688 bidirectional links between 16384 routers in the 14 dimensional HyperCube.

38 time: Total Ram: 1843972800512 free: 1826792931288

15 dimensional hypercube: 1480650669365

_created 245760 bidirectional links between 32768 routers in the 15 dimensional HyperCube.

82 time: Total Ram: 1843972800512 free: 1826792931288

16 dimensional hypercube: 1480650669447

_created 524288 bidirectional links between 65536 routers in the 16 dimensional HyperCube.

174 time: Total Ram: 1843972800512 free: 1826792931288

17 dimensional hypercube: 1480650669621

_created 1114112 bidirectional links between 131072 routers in the 17 dimensional HyperCube.

384 time: Total Ram: 1843972800512 free: 1826792931288

18 dimensional hypercube: 1480650670005

_created 2359296 bidirectional links between 262144 routers in the 18 dimensional HyperCube.

923 time: Total Ram: 1843972800512 free: 1826792931288

19 dimensional hypercube: 1480650670928

_created 4980736 bidirectional links between 524288 routers in the 19 dimensional HyperCube.

1970 time: Total Ram: 1843972800512 free: 1826792931288

20 dimensional hypercube: 1480650672898

_created 10485760 bidirectional links between 1048576 routers in the 20 dimensional HyperCube.

4677 time: Total Ram: 1843972800512 free: 1826792931288

21 dimensional hypercube: 1480650677576

_created 22020096 bidirectional links between 2097152 routers in the 21 dimensional HyperCube.

9739 time: Total Ram: 1843972800512 free: 1818202996656

22 dimensional hypercube: 1480650687314

_created 46137344 bidirectional links between 4194304 routers in the 22 dimensional HyperCube.

21156 time: Total Ram: 1843972800512 free: 1809613062048

23 dimensional hypercube: 1480650708470

_created 96468992 bidirectional links between 8388608 routers in the 23 dimensional HyperCube.

44953 time: Total Ram: 1843972800512 free: 1783843258176

Tasks: 743 total,   1 running, 742 sleeping,   0 stopped,   0 zombie

Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  2014742852k total, 55841328k used, 1958901524k free,    19244k buffers

Swap:        0k total,        0k used,        0k free,   436796k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                

 44386 ec2-user  20   0 1865g  49g  14m S 100.2  2.6   1:19.67 java

24 dimensional hypercube: 1480651245506

_created 201326592 bidirectional links between 16777216 routers in the 24 dimensional HyperCube.

101306 time: Total Ram: 1843972800512 free: 1732303649872

25 dimensional hypercube: 1480651346812

_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.

264376 time: Total Ram: 1843972800512 free: 1603454628832

Tasks: 742 total,   1 running, 741 sleeping,   0 stopped,   0 zombie

Cpu(s):  0.6%us,  0.2%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  2014742852k total, 278517036k used, 1736225816k free,    19408k buffers

Swap:        0k total,        0k used,        0k free,   436796k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 44386 ec2-user  20   0 1865g 261g  15m S 100.2 13.6   8:45.96 java  

26 dimensional hypercube: 1480651611188

_created 872415232 bidirectional links between 67108864 routers in the 26 dimensional HyperCube.

656891 time: Total Ram: 1843972800512 free: 1647312996576

[ec2-user@ip-172-31-57-105 ~]$ free

             total       used       free     shared    buffers     cached

Mem:    2014742852  661273556 1353469296         72      19672     436804

-/+ buffers/cache:  660817080 1353925772

 44386 ec2-user  20   0 1865g 625g  15m S 99.9 32.6  82:37.39 java   

Cpu(s):  0.8%us,  0.0%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  2014742852k total, 1088382060k used, 926360792k free,    19984k buffers

Swap:        0k total,        0k used,        0k free,   436804k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 44386 ec2-user  20   0 1870g 1.0t  15m S 100.2 53.7 313:16.41 java   

25 dimensional hypercube: 1480651346812

_created 419430400 bidirectional links between 33554432 routers in the 25 dimensional HyperCube.

264376 time: Total Ram: 1843972800512 free: 1603454628832

26 dimensional hypercube: 1480651611188

_created 872415232 bidirectional links between 67108864 routers in the 26 dimensional HyperCube.

656891 time: Total Ram: 1843972800512 free: 1647312996576

27 dimensional hypercube: 1480652268079

_created 1811939328 bidirectional links between 134217728 routers in the 27 dimensional HyperCube.

1433987 time: Total Ram: 1843972800512 free: 1253174190840

28 dimensional hypercube: 1480653702066

_created 3758096384 bidirectional links between 268435456 routers in the 28 dimensional HyperCube.

4437315 time: Total Ram: 1843972800512 free: 1001914613768

29 dimensional hypercube: 1480658139381

[ec2-user@ip-172-31-57-105 ~]$ free

             total       used       free     shared    buffers     cached

Mem:    2014742852 1992699228   22043624         64      28708     437000

-/+ buffers/cache: 1992233520   22509332

Swap:            0          0          0

[ec2-user@ip-172-31-57-105 ~]$ ps -ef | grep java

ec2-user  44386      1 99 03:59 ?        18-01:29:52 java -Xmx1792g -Xms1792g -server -cp graph-0.0.1-SNAPSHOT.jar org.obrienscience.ai.HyperCube 26 1

Tasks: 739 total,   1 running, 738 sleeping,   0 stopped,   0 zombie

Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  2014742852k total, 1992699740k used, 22043112k free,    28736k buffers

Swap:        0k total,        0k used,        0k free,   437000k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 44386 ec2-user  20   0 1898g 1.8t  15m S 8059.8 98.5  26052:50 java  

Cpu(s): 62.6%us,  0.3%sy,  0.0%ni, 37.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  2014742852k total, 1734049420k used, 280693432k free,    32136k buffers

Swap:        0k total,        0k used,        0k free,   437292k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 45667 ec2-user  20   0 1888g 1.6t  14m S 8045.2 85.7   5836:37 java  

Tasks: 744 total,   1 running, 743 sleeping,   0 stopped,   0 zombie

Cpu(s):  0.8%us,  0.0%sy,  0.0%ni, 99.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  2014742852k total, 1971207064k used, 43535788k free,    33936k buffers

Swap:        0k total,        0k used,        0k free,   437308k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                 

 45667 ec2-user  20   0 1886g 1.8t  14m S 100.0 97.4  13311:12 java   

[ec2-user@ip-172-31-57-105 ~]$ free

             total       used       free     shared    buffers     cached

Mem:    2014742852 1971206692   43536160         64      33944     437308

29 dimensional hypercube: 1480681783260

_created 7784628224 bidirectional links between 536870912 routers in the 29 dimensional HyperCube.

16316219 time: Total Ram: 1843972800512 free: 147852676280

10 min 26d 30g

25 min 27d 75g

75 min 28d 185g

225 min 29d 460g

800 min 30d 1300g