Tuesday, November 8, 2016

Neo4j 3.1 Graph Database for Distributed Development

Neo4j has released its 3.1.0 version of their graph database as of Dec 2016.  

Neo4j 3.1 Distributed Server Graph Database

This section details setting up Neo4j 3.1 Server on several metal and VM instances of RHEL 7.3
follow https://neo4j.com/docs/operations-manual/current/installation/linux/tarball/
Download: neo4j-enterprise-3.1.0-unix.tar.gz
copy it over to each box (I don't use a chef/ansible combo yet on these)

Source
https://github.com/obrienlabs/nbi-neo4j-embedded-aws-war

Add a node to the embedded graph db
(bolt enabled) http browser - user:neo4j pass:password



[root@obrien ~]# scp michael@192.168.15.7:/Users/michael/Dropbox/neo4j/neo4j-enterprise-3.1.0-unix.tar.gz .
[root@obrien neo4j-enterprise-3.1.0]# vi ~/.bashrc
export NEO4J_HOME=/root/neo4j-enterprise-3.1.0
I am blindly running as root inside my private subnet so I increase the file handle limits as

vi /etc/security/limits.conf
neo4j soft nofile 40000
neo4j hard nofile 40000
root soft nofile 40000
root hard nofile 40000

[root@obrien bin]# ./neo4j console
Starting Neo4j.
2016-12-18 23:25:26.073+0000 INFO  No SSL certificate found, generating a self-signed certificate..
2016-12-18 23:25:26.499+0000 INFO  Starting...
2016-12-18 23:25:27.227+0000 INFO  Bolt enabled on localhost:7687.
2016-12-18 23:25:27.241+0000 INFO  Initiating metrics...
2016-12-18 23:25:30.054+0000 INFO  Started.
2016-12-18 23:25:30.239+0000 INFO  Mounted REST API at: /db/manage
2016-12-18 23:25:31.331+0000 INFO  Remote interface available at http://localhost:7474/

Neo4j 3.1 Embedded Graph Database


This section details several ways to get an embedded version of Neo4j running in a spring container.

https://neo4j.com/docs/java-reference/current/

Results

Amazon Elastic Beanstalk deployment to t2.tiny 2.3.1

/biometric-nbi/FrontController?action=graph

SLF4J: Found binding in [jar:file:/var/lib/tomcat8/webapps/ROOT/WEB-INF/lib/neo4j-security-enterprise-3.1.0-M13-beta3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
01-Dec-2016 04:15:23.925 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.0.36
01-Dec-2016 04:15:23.928 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            4.4.23-31.54.amzn1.x86_64
01-Dec-2016 04:15:23.928 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.24.amzn1.x86_64/jre
01-Dec-2016 04:15:23.929 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_101-b13
01-Dec-2016 04:15:42.683 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 18422 ms
register: org.obrienlabs.nbi.graph.service.HaMonitor@b37b7a9
processAction: graph
<state>Node[0]+0:0</state>
processAction: graph
<state>Node[1]+1:1</state>

pom.xml

      <neo4j.version>3.1.0-M13-beta3</neo4j.version>
  </properties>
  <dependencies>
      <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-cypher</artifactId>
        <version>${neo4j.version}</version>
    </dependency> 
    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-enterprise</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j.app</groupId>
        <artifactId>neo4j-server</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-ha</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j-management</artifactId>
        <version>${neo4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j.app</groupId>
        <artifactId>neo4j-server-enterprise</artifactId>
        <version>${neo4j.version}</version>

    </dependency>
    <dependency>
        <groupId>org.neo4j.app</groupId>
        <artifactId>neo4j-bolt</artifactId>
        <version>${neo4j.version}</version>

    </dependency> 

spring.xml (embedded)


    <context:annotation-config />
    <context:spring-configured />
    <!-- rest annotations -->
    <mvc:annotation-driven />
    
      <context:component-scan base-package="org.obrienlabs.nbi.graph.service" />

    <!-- in cases where the DAO's are in a separate jar - list them -->
    <bean id="daoFacade" class="org.obrienlabs.nbi.graph.service.ApplicationService"/>
    <bean id="IGraphDatabaseService" class="org.obrienlabs.nbi.graph.service.GraphDatabaseServiceImpl"/>

  
    <util:map id="config">
        <entry key="enable_remote_shell" value="true"/>
        <entry key="ha.server_id" value="1"/>
        <entry key="ha.initial_hosts" value="127.0.0.1:5001"/>
    </util:map>
    
    <bean id="haMonitor" class="com.obrienlabs.graph.service.HaMonitor"/>
    <bean id="graphDbFactory" class="com.obrienlabs.graph.service.ExtendedHighlyAvailableGraphDatabaseFactory">
        <constructor-arg ref="haMonitor" />
    </bean>
    
    <bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newEmbeddedDatabaseBuilder">
        <constructor-arg value="neo4j"/>
    </bean>
    <bean id="graphDbBuilderFinal" factory-bean="graphDbBuilder" factory-method="setConfig">
        <constructor-arg ref="config"/>
    </bean>
    <bean id="graphDatabaseService" factory-bean="graphDbBuilderFinal" factory-method="newGraphDatabase" 
    destroy-method="shutdown" />

servlet

    @Autowired
    @Qualifier("daoFacade")

    private ApplicationServiceLocal service;

spring bean hierarchy


public interface ApplicationServiceLocal {
String getGraph();
}

@Service("daoFacade")
public class ApplicationService implements ApplicationServiceLocal {
@Inject

private IGraphDatabaseService graphDatabaseService;
}
public interface IGraphDatabaseService {
Node createNode();
Node getNodeById( long id);


}
@Named("graphDatabaseService")
public class GraphDatabaseServiceImpl implements IGraphDatabaseService {
@Inject 
GraphDatabaseService databaseService;

@Override
public Node createNode() {
Node node = null;
      try ( org.neo4j.graphdb.Transaction tx = databaseService.beginTx() )  { 
            node =  databaseService.createNode();
            node.setProperty( "message", " node1 " );
            tx.success();
        }
return node;

}


HA listener override

We are currently running the older paxos not the raft based causal cluster HA mode

2016-12-19 00:54:47.703+0000 INFO  [o.n.c.m.p.PaxosClusterMemberAvailability] Listening at:cluster://127.0.0.1:5001

package org.obrienlabs.nbi.graph.service;

import org.neo4j.kernel.ha.HighlyAvailableGraphDatabase;
import org.neo4j.kernel.ha.cluster.HighAvailabilityMemberChangeEvent;
import org.neo4j.kernel.ha.cluster.HighAvailabilityMemberListener;

public class HaMonitor implements HighAvailabilityMemberListener {

    protected HighlyAvailableGraphDatabase db;
    
    public void setDb(HighlyAvailableGraphDatabase aInDb) {
        db = aInDb;
    }
    
@Override
public void masterIsElected(HighAvailabilityMemberChangeEvent event) {
System.out.println("MasterIsElected");
}

@Override
public void masterIsAvailable(HighAvailabilityMemberChangeEvent event) {
System.out.println("masterIsAvailable");
}

@Override
public void slaveIsAvailable(HighAvailabilityMemberChangeEvent event) {
System.out.println("slaveIsAvailabl");
}

@Override
public void instanceStops(HighAvailabilityMemberChangeEvent event) {
System.out.println("instanceStops");
}

@Override
public void instanceDetached(HighAvailabilityMemberChangeEvent event) {
System.out.println("instanceDetached");
}

}

package org.obrienlabs.nbi.graph.service;

import java.io.File;
import java.util.Map;

import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.factory.GraphDatabaseBuilder;
import org.neo4j.graphdb.factory.GraphDatabaseFactoryState;

import org.neo4j.graphdb.factory.GraphDatabaseSettings;
import org.neo4j.graphdb.factory.HighlyAvailableGraphDatabaseFactory;
import org.neo4j.kernel.ha.HighlyAvailableGraphDatabase;
import org.neo4j.kernel.ha.cluster.HighAvailabilityMemberStateMachine;

public class ExtendedHighlyAvailableGraphDatabaseFactory extends HighlyAvailableGraphDatabaseFactory {

    HaMonitor haMonitor;
        GraphDatabaseSettings.BoltConnector bolt = GraphDatabaseSettings.boltConnector( "0" );
    public ExtendedHighlyAvailableGraphDatabaseFactory(HaMonitor aInHaMonitor)  {
        haMonitor = aInHaMonitor;
    }

    @Override
     protected GraphDatabaseBuilder.DatabaseCreator createDatabaseCreator(
                final File storeDir, final GraphDatabaseFactoryState state)  {
            return new GraphDatabaseBuilder.DatabaseCreator() {

                @Override
                public GraphDatabaseService newDatabase( final Map<String, String> config ) {
                    config.put( "unsupported.dbms.ephemeral", "false" );
                    // enable bolt to enable 7474 http browser
                    config.put( bolt.type.name(), "BOLT" );
                    config.put( bolt.enabled.name(), "true" );
                    config.put( bolt.address.name(), "localhost:7687" );
                    
                    HighlyAvailableGraphDatabase haDB = 
                    new HighlyAvailableGraphDatabase( storeDir, config, state.databaseDependencies());
                    haMonitor.setDb(haDB);
                    HighAvailabilityMemberStateMachine memberStateMachine = 
                    haDB.getDependencyResolver().resolveDependency(HighAvailabilityMemberStateMachine.class);
                    if ( memberStateMachine != null ) {
                        memberStateMachine.addHighAvailabilityMemberListener(haMonitor);
                        System.out.println("register: " +  haMonitor);
                    }
                    return haDB;
                }
            };
        }

}


Issue: constructor arg takes a file not a string anymore
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'graphDbBuilder' defined in ServletContext resource [/WEB-INF/spring.xml]: Unsatisfied dependency expressed through constructor argument with index 0 of type [java.io.File]: Could not convert factory method argument value of type [java.lang.String] to required type [java.io.File]: Failed to convert value of type [java.lang.String] to required type [java.io.File]; nested exception is java.lang.IllegalArgumentException: The resource path [/../work/graph.db] has been normalized to [null] which is not valid

in
    <bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newEmbeddedDatabaseBuilder">
    <!-- bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newHighlyAvailableDatabaseBuilder"-->
        <constructor-arg value="neo4j"/>

    </bean>

This works

<constructor-arg value="neo4j"/>
This does not anymore - we need to pass in a ${} variable via spring

<constructor-arg value="../neo4j"/>

https://neo4j.com/docs/java-reference/current/#tutorial-traversal


Getting the Neo4j 7474 http browser working on an Embedded graph.db

The 7474 browser no longer directly works out of the box on Neo4j 3.x as it did in 2.3.  Partially because of the introduction of bolt.
Going through the following workarounds to get it working again

Open a Bolt connector to your embedded instance to get GUI administration and other benefits
section 2.14 of



2016-12-18 03:37:42.112+0000 INFO  [o.n.k.i.DiagnosticsManager] com.sun.management.jmxremote.port = 8086
2016-12-18 03:37:42.490+0000 INFO  [o.n.b.v.r.WorkerFactory] Bolt Server extension loaded.

2016-12-18 03:37:42.490+0000 INFO  [o.n.b.v.r.WorkerFactory] Bolt enabled on localhost:7687.



It turns out with a bit of help from the following post we can still bring up a wrapping NeoServer around our embedded mode Neo4j 3.1.0 server. This code however is not integrated yet with our HA listener overrides on the constructor tree we bootstrap in Spring - so is a POC. 

With this NeoServer wrapper code - we don’t need the secondary workaround where we bring up a 2nd Neo4j server mode instance, bring up its’ 7474 browser and modify the bolt address (default 7687) in the config to point to the exposed (7688) embedded mode graph.db - without bringing either Neo4j server down as long as they point to different graph.db instances - in effect sharing the 7474 browser as a bolt client.

Disclaimer: I believe a proper architecture involves a clustered set of server mode bolt enabled Neo4j servers.
This undocumented/internal API is used as a temporary workaround to get the neo4j browser up on an embedded Neo4j server in 3.1.0 as it used to work under the WrappingNeoServerBootstrapper in 2.3 so we can continue to use in-jvm Dijkstra algos until we are able to rework/benchmark the same or greater traversal speed via a bolt enabled cluster.

FrontController/spring service bean starts graph
```
2016-12-20 01:28:04.186+0000 INFO  Starting...
2016-12-20 01:28:04.533+0000 INFO  Bolt enabled on localhost:7687.
2016-12-20 01:28:04.534+0000 INFO  Initiating metrics...
2016-12-20 01:28:06.091+0000 INFO  Started.
2016-12-20 01:28:07.009+0000 INFO  Remote interface available at http://localhost:7575/
<state>community single [/ec2-user/graph2.db] : community single [/ec2-user/graph2.db/data/databases/graph.db]</state>
```

after opening browser

```
    // http://stackoverflow.com/questions/30074232/replacement-for-deprecated-wrappingneoserverbootstrapper
    // Warning: internal/unsupported API - NeoServer wrappers are usually only run in  server mode - not embedded
        ServerBootstrapper serverBootstrapper = new EnterpriseBootstrapper();//CommunityBootstrapper();
        NeoServer directNeoServer = serverBootstrapper.getServer();
        int i = serverBootstrapper.start(new File("/ec2-user"), // will resolve to /ec2-user/data/databases/graph.db 
        Optional.of(new File("/Users/michaeal/Documents/Neo4j/.neo4j.conf")), //  stub only
            Pair.of("dbms.connector.http.address","0.0.0.0:7575"),
            Pair.of("dbms.connector.http.enabled","true" ),
            Pair.of("dbms.connector.http.type", "HTTP" ),
            Pair.of("dbms.connector.http.tls_level", "DISABLED")
            // BOLT not advised if we run an embedded Neo4j DB inside a NeoServer wrapper
            // BOLT should be enabled if you run a standalone server's browser against this embedded bolt port
            // WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver. Please use your browsers development console to determine the root cause of the failure. Common reasons include the database being unavailable, using the wrong connection URL or temporary network problems. If you have enabled encryption, ensure your browser is configured to trust the certificate Neo4j is configured to use. WebSocket `readyState` is: 3
             ,Pair.of("dbms.connector.bolt.address","0.0.0.0:7688"),
            Pair.of("dbms.connector.bolt.enabled","true" ),
            Pair.of("dbms.connector.bolt.type", "HTTP" ),
            Pair.of("dbms.connector.bolt.tls_level", "DISABLED")           
        );
        NeoServer neoServer = serverBootstrapper.getServer();
        GraphDatabaseService graph = neoServer.getDatabase().getGraph();  
```


Debugging we see that we get the proper Jetty server available usually only to server mode Neo4j not embedded mode.
```
AbstractNeoServer.<clinit>() line: 114
EnterpriseBootstrapper.createNeoServer(Config, GraphDatabaseDependencies, LogProvider) line: 45
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 90
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

CommunityNeoServer.<clinit>() line: 54
EnterpriseBootstrapper.createNeoServer(Config, GraphDatabaseDependencies, LogProvider) line: 45
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 90
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

Config.get(Setting<T>) line: 137
EnterpriseNeoServer(AbstractNeoServer).<init>(Config, Database$Factory, GraphDatabaseFacadeFactory$Dependencies, LogProvider) line: 159
EnterpriseNeoServer(CommunityNeoServer).<init>(Config, Database$Factory, GraphDatabaseFacadeFactory$Dependencies, LogProvider) line: 69
EnterpriseNeoServer.<init>(Config, GraphDatabaseFacadeFactory$Dependencies, LogProvider) line: 105
EnterpriseBootstrapper.createNeoServer(Config, GraphDatabaseDependencies, LogProvider) line: 45
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 90
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

Jetty9WebServer.<init>(LogProvider, Config) line: 135
EnterpriseNeoServer(CommunityNeoServer).createWebServer() line: 89
EnterpriseNeoServer.createWebServer() line: 131
EnterpriseNeoServer(AbstractNeoServer).init() line: 181
EnterpriseNeoServer(AbstractNeoServer).start() line: 196
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 91
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

GraphDatabaseFacadeFactory.initFacade(File, Map<String,String>, Dependencies, GraphDatabaseFacade) line: 141
EnterpriseGraphDatabase.<init>(File, Map<String,String>, Dependencies) line: 36
EnterpriseNeoServer.lambda$static$1(Config, GraphDatabaseFacadeFactory$Dependencies) line: 90
1274309106.newGraphDatabase(Config, GraphDatabaseFacadeFactory$Dependencies) line: not available
LifecycleManagingDatabase.start() line: 89
LifeSupport$LifecycleInstance.start() line: 433
LifeSupport.start() line: 107
EnterpriseNeoServer(AbstractNeoServer).start() line: 199
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 91
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

Jetty9WebServer.loadJAXRSResource(SessionManager, String, JaxRsServletHolderFactory) line: 536
Jetty9WebServer.loadJAXRSClasses(SessionManager, String) line: 522
Jetty9WebServer.loadAllMounts() line: 409
Jetty9WebServer.start() line: 171
EnterpriseNeoServer(AbstractNeoServer).startWebServer() line: 326
EnterpriseNeoServer(AbstractNeoServer).start() line: 218
EnterpriseBootstrapper(ServerBootstrapper).start(File, Optional<File>, Pair<String,String>...) line: 91
FrontController.processGraph3(HttpServletRequest, HttpServletResponse, PrintWriter) line: 161

serverBootstrapper EnterpriseBootstrapper  (id=299)
dependencies GraphDatabaseDependencies  (id=347)
log FormattedLog  (id=573)
server EnterpriseNeoServer  (id=357)
serverAddress "0.0.0.0:7575" (id=575)
shutdownHook ServerBootstrapper$1  (id=577)

neoServer EnterpriseNeoServer  (id=357)
….
webServer Jetty9WebServer  (id=406)

from NeoServer
graph EnterpriseGraphDatabase  (id=417)
contextFactory Neo4jTransactionalContextFactory  (id=1119)
defaultTransactionTimeout 0
indexManager Suppliers$1  (id=1121)
nodeActions StandardNodeActions  (id=1123)
relActions StandardRelationshipActions  (id=1128)
schema SchemaImpl  (id=1132)
spi ClassicCoreSPI  (id=425)

Without NeoServer
graphDb GraphDatabaseFacade  (id=150)
contextFactory Neo4jTransactionalContextFactory  (id=1182)
defaultTransactionTimeout 0
indexManager Suppliers$1  (id=1183)
nodeActions StandardNodeActions  (id=1184)
relActions StandardRelationshipActions  (id=1208)
schema SchemaImpl  (id=1219)
spi ClassicCoreSPI  (id=160)
```


```
2016-12-20 02:15:02.825+0000 INFO  Starting...
2016-12-20 02:15:03.304+0000 INFO  Bolt enabled on 0.0.0.0:7688.
2016-12-20 02:15:03.316+0000 INFO  Initiating metrics...
2016-12-20 02:15:06.232+0000 INFO  Started.
2016-12-20 02:15:06.412+0000 INFO  Mounted REST API at: /db/manage
2016-12-20 02:15:07.103+0000 INFO  Remote interface available at http://localhost:7575/
<state> : enterprise single [/ec2-user/data/databases/graph.db]</state>
```

settings and logs
```
cat /ec2-user/logs/debug.log
2016-12-20 02:15:03.225+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.enabled=true
2016-12-20 02:15:03.225+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.security.auth_enabled=true
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.type=HTTP
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.directories.neo4j_home=/ec2-user
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.directories.import=import
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.tls_level=DISABLED
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.type=HTTP
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.logs.http.enabled=false
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.edition=enterprise
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.address=0.0.0.0:7688
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.address=0.0.0.0:7575
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.enabled=true
2016-12-20 02:15:03.226+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.tls_level=DISABLED
2016-12-20 02:15:03.304+0000 INFO  [o.n.b.v.r.WorkerFactory] Bolt Server extension loaded.
2016-12-20 02:15:03.304+0000 INFO  [o.n.b.v.r.WorkerFactory] Bolt enabled on 0.0.0.0:7688.
2016-12-20 02:15:04.687+0000 INFO  [o.n.k.i.DiagnosticsManager] Kernel version: 3.1.0,16a782b42d76ca37db72958eb2565cf6aa671a29
```

These errors are non-fatal related to JMX and can be fixed
```
2016-12-20 02:17:30.759+0000 ERROR [o.n.b.v.r.ErrorReporter] Client triggered an unexpected error [UnknownError]: Exception invoking method writeBufSize, reference 59659a03-461e-4532-9b98-7afe3a2e5704. Exception invoking method writeBufSize
javax.management.RuntimeOperationsException: Exception invoking method writeBufSize
at org.apache.tomcat.util.modeler.BaseModelMBean.getAttribute(BaseModelMBean.java:196)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at org.neo4j.kernel.builtinprocs.JmxQueryProcedure.toNeo4jValue(JmxQueryProcedure.java:131)
Caused by: java.lang.NullPointerException
at org.apache.tomcat.util.net.SocketProperties.getTxBufSize(SocketProperties.java:300)
```


7575 http and 7688 bolt ports
```
obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7575
tcp46      0      0  *.7575                 *.*                    LISTEN      131072 131072  49013      0
tcp4       0      0  127.0.0.1.7575         127.0.0.1.60685        TIME_WAIT   407296 146988  49013      0
tcp6       0      0  ::1.7575               ::1.60699              TIME_WAIT   407284 146808  49013      0
tcp6       0      0  ::1.7575               ::1.60700              TIME_WAIT   407284 146808  49013      0
obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7688
tcp6       0      0  ::1.7688               ::1.60704              ESTABLISHED 406582 146808  49013      0
tcp6       0      0  ::1.60704              ::1.7688               ESTABLISHED 398196 146808  48165      0
tcp6       0      0  ::1.7688               ::1.60702              ESTABLISHED 406570 146808  49013      0
tcp6       0      0  ::1.60702              ::1.7688               ESTABLISHED 398185 146808  48165      0
tcp6       0      0  ::1.7688               ::1.60701              ESTABLISHED 407255 146808  49013      0
tcp6       0      0  ::1.60701              ::1.7688               ESTABLISHED 407628 146808  48165      0
tcp46      0      0  *.7688                 *.*                    LISTEN      131072 131072  49013      0
obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 8080
tcp4       0      0  127.0.0.1.8080         127.0.0.1.60584        FIN_WAIT_2  408104 146988  49013      0
tcp4     994      0  127.0.0.1.60584        127.0.0.1.8080         CLOSE_WAIT  408128 146988  42992      0
tcp46      0      0  *.8080                 *.*                    LISTEN      131072 131072  49013      0
udp4       0      0  *.*                    *.*                                196724   9216  38080      0
```


This code however is not integrated yet with our HA listener overrides on the constructor tree we bootstrap in Spring - the following scrubbed/distilled mirror POC of our corporate production code will be adjusted in the next post.

```
public class ExtendedHighlyAvailableGraphDatabaseFactory extends HighlyAvailableGraphDatabaseFactory {
    private HaMonitor haMonitor;
    private GraphDatabaseSettings.BoltConnector bolt = GraphDatabaseSettings.boltConnector( "0" );
    @Override
     protected GraphDatabaseBuilder.DatabaseCreator createDatabaseCreator(
                final File storeDir, final GraphDatabaseFactoryState state)  {
            return new GraphDatabaseBuilder.DatabaseCreator() {
                @Override
                public GraphDatabaseService newDatabase( final Map<String, String> config ) {
                    HighlyAvailableGraphDatabase haDB = 
                    new HighlyAvailableGraphDatabase( storeDir, config, state.databaseDependencies());
                    haMonitor.setDb(haDB);
                    HighAvailabilityMemberStateMachine memberStateMachine = 
                    haDB.getDependencyResolver().resolveDependency(HighAvailabilityMemberStateMachine.class);
                    if ( memberStateMachine != null ) {
                        memberStateMachine.addHighAvailabilityMemberListener(haMonitor);
                    }
                    return haDB;
                } };}

```

Neo4j 3.1.0 HA (paxos) Embedded mode HighlyAvailableGraphDatabase wrapped with an EnterpriseBootstrapper NeoServer

see

In this code we get everything (except causal cluster using the raft protocol).

With a few modifications I was able to wrap a HighlyAvailableGraphDatabase with a EnterpriseBootstrapper.

There are some non-fatal exceptions around JMX reporting in debug.log likely related to my 8.0.28 tomcat version but the graph db running embedded in Tomcat is OK.

        2016-12-21 16:20:00.574+0000 INFO  Bolt enabled on 0.0.0.0:7688.
        2016-12-21 16:20:09.554+0000 INFO  Attempting to join cluster of [127.0.0.1:5001]
        2016-12-21 16:20:11.566+0000 INFO  Creating new cluster with name [neo4j.ha]...
        2016-12-21 16:20:11.607+0000 INFO  Instance 1 (this server)  entered the cluster
        2016-12-21 16:20:12.164+0000 INFO  I am 1, moving to master
        2016-12-21 16:20:12.293+0000 INFO  Instance 1 (this server)  was elected as coordinator
        2016-12-21 16:20:12.462+0000 INFO  I am 1, successfully moved to master
        2016-12-21 16:20:12.513+0000 INFO  Instance 1 (this server)  is available as master at ha://127.0.0.1:6001?serverId=1 with StoreId{creationTime=1482199697648, randomId=7800059877674392627, storeVersion=15531981201765894, upgradeTime=1482199697648, upgradeId=1}
        2016-12-21 16:20:14.495+0000 INFO  Database available for write transactions
        2016-12-21 16:20:31.917+0000 INFO  Mounted REST API at: /db/manage
        2016-12-21 16:20:53.264+0000 INFO  Remote interface available at http://localhost:7575/
        register: org.obrienlabs.nbi.graph.service.HaMonitor@1c0f80c9


        public class ExtendedHighlyAvailableGraphDatabaseFactory extends HighlyAvailableGraphDatabaseFactory {
            private HaMonitor haMonitor;    
            public GraphDatabaseService getEnterpriseGraphDatabase(final File _storeDir, final GraphDatabaseFactoryState _state,  
            final Map<String, String> _config) {
                String graphPath = "/ec2-user/graph2.db" ;
            // http://stackoverflow.com/questions/30074232/replacement-for-deprecated-wrappingneoserverbootstrapper
            // Warning: internal/unsupported API - NeoServer wrappers are usually only run in  server mode - not embedded
                EnterpriseBootstrapper serverBootstrapper = new EnterpriseBootstrapper();//CommunityBootstrapper();
                int i = serverBootstrapper.start(new File("/ec2-user"), // will resolve to /ec2-user/data/databases/graph.db 
                Optional.of(new File("/some-path/Neo4j/.neo4j.conf")), //  stub only
    Pair.of("dbms.connector.http.address","0.0.0.0:7575"),
                    Pair.of("dbms.connector.http.enabled","true" ),
                    Pair.of("dbms.connector.http.type", "HTTP" ),
                    Pair.of("dbms.connector.http.tls_level", "DISABLED"),
         Pair.of("dbms.connector.bolt.address","0.0.0.0:7688"),
                    Pair.of("dbms.connector.bolt.enabled","true" ),
                    Pair.of("dbms.connector.bolt.type", "HTTP" ),
                    Pair.of("dbms.connector.bolt.tls_level", "DISABLED") ,
                    Pair.of("ha.initial_hosts", "127.0.0.1:5001"),
                       Pair.of("ha.server_id", "1"),
                    Pair.of("dbms.mode", EnterpriseNeoServer.Mode.HA.toString()));
                NeoServer neoServer = serverBootstrapper.getServer();
                GraphDatabaseService graph = neoServer.getDatabase().getGraph(); 
                return graph;
            }
        
            @Override
            protected GraphDatabaseBuilder.DatabaseCreator createDatabaseCreator(
                        final File storeDir, final GraphDatabaseFactoryState state)  {
                    return new GraphDatabaseBuilder.DatabaseCreator() {
                        @Override
                        public GraphDatabaseService newDatabase( final Map<String, String> config ) {
                            // section 2.14 of http://neo4j.com/docs/java-reference/current/#tutorials-java-embedded       
                            /*HighlyAvailableGraphDatabase haDB = 
                            new HighlyAvailableGraphDatabase( storeDir, config, state.databaseDependencies());*/
                            HighlyAvailableGraphDatabase haDB =(HighlyAvailableGraphDatabase) getEnterpriseGraphDatabase( storeDir, state,  config);
                            haMonitor.setDb(haDB);
                            // Add the HA monitor listener
                            HighAvailabilityMemberStateMachine memberStateMachine = 
                            haDB.getDependencyResolver().resolveDependency(HighAvailabilityMemberStateMachine.class);
                            if ( memberStateMachine != null ) {
                                memberStateMachine.addHighAvailabilityMemberListener(haMonitor);
                            }
                            return haDB;
                        }};}

Transactions go through

    2016-12-21 20:51:07.478+0000 INFO  [o.n.k.i.s.c.CountsTracker] About to rotate counts store at transaction 9 to [/ec2-user/data/databases/graph.db/neostore.counts.db.b], from [/ec2-user/data/databases/graph.db/neostore.counts.db.a].
    2016-12-21 20:51:07.480+0000 INFO  [o.n.k.i.s.c.CountsTracker] Successfully rotated counts store at transaction 9 to [/ec2-user/data/databases/graph.db/neostore.counts.db.b], from [/ec2-user/data/databases/graph.db/neostore.counts.db.a].
    2016-12-21 20:51:07.483+0000 INFO  [o.n.k.i.t.l.c.CheckPointerImpl] Check Pointing triggered by scheduler for time threshold [9]:  Store flush completed

Settings from the mbean in jconsole are

    2016-12-21 20:55:46.503+0000 INFO  [o.n.k.i.DiagnosticsManager] --- REQUESTED diagnostics START ---
    2016-12-21 20:55:46.503+0000 INFO  [o.n.k.i.DiagnosticsManager] Graph Database: enterprise ha StoreId{creationTime=1482199697648, randomId=7800059877674392627, storeVersion=15531981201765894, upgradeTime=1482199697648, upgradeId=1}
    2016-12-21 20:55:46.503+0000 INFO  [o.n.k.i.DiagnosticsManager] Kernel version: 3.1.0,16a782b42d76ca37db72958eb2565cf6aa671a29
    2016-12-21 20:55:46.503+0000 INFO  [o.n.k.i.DiagnosticsManager] Neo4j Kernel properties:
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.enabled=true
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.mode=HA
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.block_size.labels=56
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.security.auth_enabled=true
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.type=HTTP
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.directories.neo4j_home=/ec2-user
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.directories.import=import
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.ephemeral=false
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.tls_level=DISABLED
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.type=HTTP
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] ha.server_id=1
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.block_size.strings=120
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.logs.http.enabled=false
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.block_size.array_properties=120
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] ha.initial_hosts=127.0.0.1:5001
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.allow_format_migration=false
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] unsupported.dbms.edition=enterprise
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.address=0.0.0.0:7688
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.address=0.0.0.0:7575
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.enabled=true
    2016-12-21 20:55:46.504+0000 INFO  [o.n.k.i.DiagnosticsManager] dbms.connector.http.tls_level=DISABLED

7575(jetty server) and 7688 (bolt) ports are open

    obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7575
    tcp46      0      0  *.7575                 *.*                    LISTEN      131072 131072  49013      0
    tcp4       0      0  127.0.0.1.7575         127.0.0.1.60685        TIME_WAIT   407296 146988  49013      0
    tcp6       0      0  ::1.7575               ::1.60699              TIME_WAIT   407284 146808  49013      0
    tcp6       0      0  ::1.7575               ::1.60700              TIME_WAIT   407284 146808  49013      0
    obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7688
    tcp6       0      0  ::1.7688               ::1.60704              ESTABLISHED 406582 146808  49013      0
    tcp6       0      0  ::1.60704              ::1.7688               ESTABLISHED 398196 146808  48165      0
    tcp6       0      0  ::1.7688               ::1.60702              ESTABLISHED 406570 146808  49013      0
    tcp6       0      0  ::1.60702              ::1.7688               ESTABLISHED 398185 146808  48165      0
    tcp6       0      0  ::1.7688               ::1.60701              ESTABLISHED 407255 146808  49013      0
    tcp6       0      0  ::1.60701              ::1.7688               ESTABLISHED 407628 146808  48165      0
    tcp46      0      0  *.7688                 *.*                    LISTEN      131072 131072  49013      0
    obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 8080
    tcp4       0      0  127.0.0.1.8080         127.0.0.1.60584        FIN_WAIT_2  408104 146988  49013      0
    tcp4     994      0  127.0.0.1.60584        127.0.0.1.8080         CLOSE_WAIT  408128 146988  42992      0
    tcp46      0      0  *.8080                 *.*                    LISTEN      131072 131072  49013      0
    udp4       0      0  *.*                    *.*                                196724   9216  38080      0

Disclaimer: I believe a proper architecture involves a clustered set of server mode bolt enabled Neo4j servers using causal clustering under the raft protocol.
This undocumented/internal API is used as a temporary workaround to get the neo4j browser up on an embedded Neo4j server in 3.1.0 as it used to work under the WrappingNeoServerBootstrapper in 2.3.  Use of the embedded server for faster traversals was the use case - performance must be reevaluated to see if a bolt based architecture is more optimal for the traversal API.
/michael

Neo4j 3.1 Technology

https://neo4j.com/blog/neo4j-3-1-beta-release/?ref=home
- causal clustering (with read replicas and synced read/write clusters)
- bolt can drive load balancing (with read/write splitting)
- 4 security roles with list and terminate control
- node and id reallocation from deleted records (uuid reuse?)
- max query duration

Test Code

Run the following to bring up a fully populated hypercube and test the memory limits of your server and browser
package org.obrienlabs.nbi.graph.test;

import static org.neo4j.driver.v1.Values.parameters;

import org.neo4j.driver.v1.AuthTokens;
import org.neo4j.driver.v1.Driver;
import org.neo4j.driver.v1.GraphDatabase;
import org.neo4j.driver.v1.Session;
import org.neo4j.driver.v1.StatementResult;
import org.neo4j.driver.v1.Transaction;

public class BoltUnitTest {

public static long linkStep = 0;
public static long linkCount = 0;
public static long pk = 0;
public static long pair = 0;
public static long linkMax = 0;
    private static final String url = "bolt://192.168.0.26:7687";
    private static final String username =  "neo4j";
    private static final String password = "mainstreet"; 
private void link(String label, long pair0, long pair1, long pk0, long pk1, Session session) {
         try (Transaction writeTransaction = session.beginTransaction()) {
        writeTransaction.success();
        writeTransaction.run( 
        "MATCH (a:Node"+ pair0 + "),(b:Node"+ pair1 + ") WHERE a.name = {p0} AND b.name = {p1} CREATE (a)-[r:" + label + "]->(b)",
                        parameters( "p0", pk0, "p1", pk1));
        StatementResult edge1  = writeTransaction.run( 
        "MATCH (a:Node"+ pair0 + "),(b:Node"+ pair1 + ") WHERE a.name = {p0} AND b.name = {p1} CREATE (a)-[r:" + label + "]->(b) RETURN r",
                        parameters( "p0", pk1, "p1", pk0));
        writeTransaction.success();
        linkStep--;
        linkCount++;
        if(linkStep < 0) {
        linkStep = 16;
        System.out.println("link: " + pk0 + "-" + pk1 + " : " + linkCount + " of " + linkMax);
        }
        }
}

private void create(long pk0, long pair, Session session) {
        try (Transaction writeTransaction = session.beginTransaction()) {
        StatementResult left  = writeTransaction.run( "CREATE (a:Node" + pair + " {name: {pk}}) return a",
                    parameters( "pk", pk));
        pk++;
        writeTransaction.success();
        }
}

private void createL(int power, long id, Session session) {
// base
if(power == 0) {
        create(id,  0, session);
        create(id + 1, 0,  session);
        link("L0", 0, 0, id ,id + 1, session);
        //pair++;
        return;
}
// recursive
int ext = 1 << power;
createL(power - 1, id, session);
createL(power - 1, id + ext, session);
for(int i=0; i<ext; i++) {
link("L" + power, 0, 0, id + i, id + ext + i, session);
}
}
// match(n:Node0) where n.name=0 return(n);
public void populateHypercube() throws Exception {
  long count = 0;
  int lastPower = 16;
  int power = 16;
  linkMax = (1 << power) * (power + 1);
        Driver driver = GraphDatabase.driver(url,  AuthTokens.basic(username, password)); 
        try (Session session = driver.session()) {
            try (Transaction writeTransaction = session.beginTransaction()) {
            for(int i=0;i<lastPower + 1;i++) {
            writeTransaction.run(" match (:Node0)-[r:L" + i + "]->(:Node0) delete r");
            }
        writeTransaction.run(" match(n) delete(n)");
        writeTransaction.success();
            }
        createL(power, count, session);
        }
        driver.close();
    }
 
// https://github.com/neo4j/neo4j-java-driver/blob/1.1/examples/src/main/java/org/neo4j/docs/driver/MinimalWorkingExample.java
public static void main(String[] args) {
BoltUnitTest test = new BoltUnitTest();
try {
test.populateHypercube();
} catch(Exception e) {
e.printStackTrace();
}
}

}

Troubleshooting

./neo4j start does not work
Fix: make sure your java 8 home is set like

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home

Links



https://neo4j.com/docs/java-reference/current/javadocs/deprecated-list.html
references deprecated wrapper



No comments:

Total Pageviews

Followers