JPA Persistence

Saturday, October 1, 2016

Machine Learning using Apache Spark on top of Apache Hadoop and Kafka

Hadoop processes historical data where Spark processes real time iOT data as it comes in for trends.

http://www.cs.toronto.edu/~hinton/science.pdf

MapReduce Batch Processing
http://hadoop.apache.org/
http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.3/hadoop-2.7.3-src.tar.gz
http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
https://hadoop.apache.org/docs/r2.7.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html

Realtime Streaming Processing
runs on Hadoop
http://spark.apache.org/

Distributed Streaming Platform
runs on Scala on top of Zookeeper
http://kafka.apache.org/
https://kafka.apache.org/08/quickstart.html

output

[root@nuc12-i7 hadoop-2.7.3]# vi /home/hadoop-2.7.3/etc/hadoop/hadoop-env.sh

[root@nuc12-i7 hadoop-2.7.3]# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep input output 'dfs[a-z.]+'

16/10/02 00:16:47 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id

16/10/02 00:16:47 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=

16/10/02 00:16:48 INFO input.FileInputFormat: Total input paths to process : 8

16/10/02 00:16:48 INFO mapreduce.JobSubmitter: number of splits:8

16/10/02 00:16:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1447651192_0001

16/10/02 00:16:48 INFO mapreduce.Job: The url to track the job: http://localhost:8080/

16/10/02 00:16:48 INFO mapreduce.Job: Running job: job_local1447651192_0001

16/10/02 00:16:48 INFO mapred.LocalJobRunner: OutputCommitter set in config null

16/10/02 00:16:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:48 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Waiting for map tasks

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1447651192_0001_m_000000_0

16/10/02 00:16:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:48 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:48 INFO mapred.MapTask: Processing split: file:/home/hadoop-2.7.3/input/hadoop-policy.xml:0+9683

16/10/02 00:16:48 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

16/10/02 00:16:48 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

16/10/02 00:16:48 INFO mapred.MapTask: soft limit at 83886080

16/10/02 00:16:48 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

16/10/02 00:16:48 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

16/10/02 00:16:48 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

16/10/02 00:16:48 INFO mapred.LocalJobRunner:

16/10/02 00:16:48 INFO mapred.MapTask: Starting flush of map output

16/10/02 00:16:48 INFO mapred.MapTask: Spilling map output

16/10/02 00:16:48 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600

16/10/02 00:16:48 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600

16/10/02 00:16:48 INFO mapred.MapTask: Finished spill 0

16/10/02 00:16:48 INFO mapred.Task: Task:attempt_local1447651192_0001_m_000000_0 is done. And is in the process of committing

16/10/02 00:16:48 INFO mapred.LocalJobRunner: map

16/10/02 00:16:48 INFO mapred.Task: Task 'attempt_local1447651192_0001_m_000000_0' done.

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Finishing task: attempt_local1447651192_0001_m_000000_0

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1447651192_0001_m_000001_0

16/10/02 00:16:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:48 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:48 INFO mapred.MapTask: Processing split: file:/home/hadoop-2.7.3/input/kms-site.xml:0+5511

16/10/02 00:16:48 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

16/10/02 00:16:48 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

16/10/02 00:16:48 INFO mapred.MapTask: soft limit at 83886080

16/10/02 00:16:48 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

16/10/02 00:16:48 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

16/10/02 00:16:48 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

16/10/02 00:16:48 INFO mapred.LocalJobRunner:

16/10/02 00:16:48 INFO mapred.MapTask: Starting flush of map output

16/10/02 00:16:48 INFO mapred.Task: Task:attempt_local1447651192_0001_m_000001_0 is done. And is in the process of committing

16/10/02 00:16:48 INFO mapred.LocalJobRunner: map

16/10/02 00:16:48 INFO mapred.Task: Task 'attempt_local1447651192_0001_m_000001_0' done.

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Finishing task: attempt_local1447651192_0001_m_000001_0

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1447651192_0001_m_000002_0

16/10/02 00:16:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:48 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:48 INFO mapred.MapTask: Processing split: file:/home/hadoop-2.7.3/input/capacity-scheduler.xml:0+4436

16/10/02 00:16:48 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

16/10/02 00:16:48 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

16/10/02 00:16:48 INFO mapred.MapTask: soft limit at 83886080

16/10/02 00:16:48 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

16/10/02 00:16:48 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

16/10/02 00:16:48 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

16/10/02 00:16:48 INFO mapred.LocalJobRunner:

16/10/02 00:16:48 INFO mapred.MapTask: Starting flush of map output

16/10/02 00:16:48 INFO mapred.Task: Task:attempt_local1447651192_0001_m_000002_0 is done. And is in the process of committing

16/10/02 00:16:48 INFO mapred.LocalJobRunner: map

16/10/02 00:16:48 INFO mapred.Task: Task 'attempt_local1447651192_0001_m_000002_0' done.

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Finishing task: attempt_local1447651192_0001_m_000002_0

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1447651192_0001_m_000003_0

16/10/02 00:16:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:48 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:48 INFO mapred.MapTask: Processing split: file:/home/hadoop-2.7.3/input/kms-acls.xml:0+3518

16/10/02 00:16:48 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

16/10/02 00:16:48 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

16/10/02 00:16:48 INFO mapred.MapTask: soft limit at 83886080

16/10/02 00:16:48 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

16/10/02 00:16:48 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

16/10/02 00:16:48 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

16/10/02 00:16:48 INFO mapred.LocalJobRunner:

16/10/02 00:16:48 INFO mapred.MapTask: Starting flush of map output

16/10/02 00:16:48 INFO mapred.Task: Task:attempt_local1447651192_0001_m_000003_0 is done. And is in the process of committing

16/10/02 00:16:48 INFO mapred.LocalJobRunner: map

16/10/02 00:16:48 INFO mapred.Task: Task 'attempt_local1447651192_0001_m_000003_0' done.

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Finishing task: attempt_local1447651192_0001_m_000003_0

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1447651192_0001_m_000004_0

16/10/02 00:16:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:48 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:48 INFO mapred.MapTask: Processing split: file:/home/hadoop-2.7.3/input/hdfs-site.xml:0+775

16/10/02 00:16:48 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

16/10/02 00:16:48 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

16/10/02 00:16:48 INFO mapred.MapTask: soft limit at 83886080

16/10/02 00:16:48 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

16/10/02 00:16:48 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

16/10/02 00:16:48 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

16/10/02 00:16:48 INFO mapred.LocalJobRunner:

16/10/02 00:16:48 INFO mapred.MapTask: Starting flush of map output

16/10/02 00:16:48 INFO mapred.Task: Task:attempt_local1447651192_0001_m_000004_0 is done. And is in the process of committing

16/10/02 00:16:48 INFO mapred.LocalJobRunner: map

16/10/02 00:16:48 INFO mapred.Task: Task 'attempt_local1447651192_0001_m_000004_0' done.

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Finishing task: attempt_local1447651192_0001_m_000004_0

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1447651192_0001_m_000005_0

16/10/02 00:16:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:48 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:48 INFO mapred.MapTask: Processing split: file:/home/hadoop-2.7.3/input/core-site.xml:0+774

16/10/02 00:16:48 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

16/10/02 00:16:48 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

16/10/02 00:16:48 INFO mapred.MapTask: soft limit at 83886080

16/10/02 00:16:48 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

16/10/02 00:16:48 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

16/10/02 00:16:48 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

16/10/02 00:16:48 INFO mapred.LocalJobRunner:

16/10/02 00:16:48 INFO mapred.MapTask: Starting flush of map output

16/10/02 00:16:48 INFO mapred.Task: Task:attempt_local1447651192_0001_m_000005_0 is done. And is in the process of committing

16/10/02 00:16:48 INFO mapred.LocalJobRunner: map

16/10/02 00:16:48 INFO mapred.Task: Task 'attempt_local1447651192_0001_m_000005_0' done.

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Finishing task: attempt_local1447651192_0001_m_000005_0

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1447651192_0001_m_000006_0

16/10/02 00:16:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:48 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:48 INFO mapred.MapTask: Processing split: file:/home/hadoop-2.7.3/input/yarn-site.xml:0+690

16/10/02 00:16:48 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

16/10/02 00:16:48 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

16/10/02 00:16:48 INFO mapred.MapTask: soft limit at 83886080

16/10/02 00:16:48 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

16/10/02 00:16:48 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

16/10/02 00:16:48 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

16/10/02 00:16:48 INFO mapred.LocalJobRunner:

16/10/02 00:16:48 INFO mapred.MapTask: Starting flush of map output

16/10/02 00:16:48 INFO mapred.Task: Task:attempt_local1447651192_0001_m_000006_0 is done. And is in the process of committing

16/10/02 00:16:48 INFO mapred.LocalJobRunner: map

16/10/02 00:16:48 INFO mapred.Task: Task 'attempt_local1447651192_0001_m_000006_0' done.

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Finishing task: attempt_local1447651192_0001_m_000006_0

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1447651192_0001_m_000007_0

16/10/02 00:16:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:48 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:48 INFO mapred.MapTask: Processing split: file:/home/hadoop-2.7.3/input/httpfs-site.xml:0+620

16/10/02 00:16:48 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

16/10/02 00:16:48 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

16/10/02 00:16:48 INFO mapred.MapTask: soft limit at 83886080

16/10/02 00:16:48 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

16/10/02 00:16:48 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

16/10/02 00:16:48 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

16/10/02 00:16:48 INFO mapred.LocalJobRunner:

16/10/02 00:16:48 INFO mapred.MapTask: Starting flush of map output

16/10/02 00:16:48 INFO mapred.Task: Task:attempt_local1447651192_0001_m_000007_0 is done. And is in the process of committing

16/10/02 00:16:48 INFO mapred.LocalJobRunner: map

16/10/02 00:16:48 INFO mapred.Task: Task 'attempt_local1447651192_0001_m_000007_0' done.

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Finishing task: attempt_local1447651192_0001_m_000007_0

16/10/02 00:16:48 INFO mapred.LocalJobRunner: map task executor complete.

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Waiting for reduce tasks

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1447651192_0001_r_000000_0

16/10/02 00:16:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:48 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:48 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@619f906a

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10

16/10/02 00:16:48 INFO reduce.EventFetcher: attempt_local1447651192_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events

16/10/02 00:16:48 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1447651192_0001_m_000003_0 decomp: 2 len: 6 to MEMORY

16/10/02 00:16:48 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1447651192_0001_m_000003_0

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2

16/10/02 00:16:48 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1447651192_0001_m_000000_0 decomp: 21 len: 25 to MEMORY

16/10/02 00:16:48 WARN io.ReadaheadPool: Failed readahead on ifile

EBADF: Bad file descriptor

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)

at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

16/10/02 00:16:48 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local1447651192_0001_m_000000_0

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->23

16/10/02 00:16:48 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1447651192_0001_m_000006_0 decomp: 2 len: 6 to MEMORY

16/10/02 00:16:48 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1447651192_0001_m_000006_0

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 23, usedMemory ->25

16/10/02 00:16:48 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1447651192_0001_m_000007_0 decomp: 2 len: 6 to MEMORY

16/10/02 00:16:48 WARN io.ReadaheadPool: Failed readahead on ifile

EBADF: Bad file descriptor

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)

at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

16/10/02 00:16:48 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1447651192_0001_m_000007_0

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 25, usedMemory ->27

16/10/02 00:16:48 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1447651192_0001_m_000004_0 decomp: 2 len: 6 to MEMORY

16/10/02 00:16:48 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1447651192_0001_m_000004_0

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 27, usedMemory ->29

16/10/02 00:16:48 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1447651192_0001_m_000001_0 decomp: 2 len: 6 to MEMORY

16/10/02 00:16:48 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1447651192_0001_m_000001_0

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 29, usedMemory ->31

16/10/02 00:16:48 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1447651192_0001_m_000005_0 decomp: 2 len: 6 to MEMORY

16/10/02 00:16:48 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1447651192_0001_m_000005_0

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 31, usedMemory ->33

16/10/02 00:16:48 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1447651192_0001_m_000002_0 decomp: 2 len: 6 to MEMORY

16/10/02 00:16:48 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1447651192_0001_m_000002_0

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 33, usedMemory ->35

16/10/02 00:16:48 WARN io.ReadaheadPool: Failed readahead on ifile

EBADF: Bad file descriptor

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)

at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

16/10/02 00:16:48 WARN io.ReadaheadPool: Failed readahead on ifile

EBADF: Bad file descriptor

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)

at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

16/10/02 00:16:48 WARN io.ReadaheadPool: Failed readahead on ifile

EBADF: Bad file descriptor

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)

at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

16/10/02 00:16:48 WARN io.ReadaheadPool: Failed readahead on ifile

EBADF: Bad file descriptor

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)

at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

16/10/02 00:16:48 WARN io.ReadaheadPool: Failed readahead on ifile

EBADF: Bad file descriptor

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)

at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

16/10/02 00:16:48 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning

16/10/02 00:16:48 INFO mapred.LocalJobRunner: 8 / 8 copied.

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: finalMerge called with 8 in-memory map-outputs and 0 on-disk map-outputs

16/10/02 00:16:48 INFO mapred.Merger: Merging 8 sorted segments

16/10/02 00:16:48 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: Merged 8 segments, 35 bytes to disk to satisfy reduce memory limit

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk

16/10/02 00:16:48 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce

16/10/02 00:16:48 INFO mapred.Merger: Merging 1 sorted segments

16/10/02 00:16:48 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes

16/10/02 00:16:48 INFO mapred.LocalJobRunner: 8 / 8 copied.

16/10/02 00:16:48 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords

16/10/02 00:16:48 INFO mapred.Task: Task:attempt_local1447651192_0001_r_000000_0 is done. And is in the process of committing

16/10/02 00:16:48 INFO mapred.LocalJobRunner: 8 / 8 copied.

16/10/02 00:16:48 INFO mapred.Task: Task attempt_local1447651192_0001_r_000000_0 is allowed to commit now

16/10/02 00:16:48 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1447651192_0001_r_000000_0' to file:/home/hadoop-2.7.3/grep-temp-965988443/_temporary/0/task_local1447651192_0001_r_000000

16/10/02 00:16:48 INFO mapred.LocalJobRunner: reduce > reduce

16/10/02 00:16:48 INFO mapred.Task: Task 'attempt_local1447651192_0001_r_000000_0' done.

16/10/02 00:16:48 INFO mapred.LocalJobRunner: Finishing task: attempt_local1447651192_0001_r_000000_0

16/10/02 00:16:48 INFO mapred.LocalJobRunner: reduce task executor complete.

16/10/02 00:16:49 INFO mapreduce.Job: Job job_local1447651192_0001 running in uber mode : false

16/10/02 00:16:49 INFO mapreduce.Job: map 100% reduce 100%

16/10/02 00:16:49 INFO mapreduce.Job: Job job_local1447651192_0001 completed successfully

16/10/02 00:16:49 INFO mapreduce.Job: Counters: 30

File System Counters

FILE: Number of bytes read=2892701

FILE: Number of bytes written=5231540

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

Map-Reduce Framework

Map input records=745

Map output records=1

Map output bytes=17

Map output materialized bytes=67

Input split bytes=877

Combine input records=1

Combine output records=1

Reduce input groups=1

Reduce shuffle bytes=67

Reduce input records=1

Reduce output records=1

Spilled Records=2

Shuffled Maps =8

Failed Shuffles=0

Merged Map outputs=8

GC time elapsed (ms)=90

Total committed heap usage (bytes)=3163029504

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=26007

File Output Format Counters

Bytes Written=123

16/10/02 00:16:49 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

16/10/02 00:16:49 INFO input.FileInputFormat: Total input paths to process : 1

16/10/02 00:16:49 INFO mapreduce.JobSubmitter: number of splits:1

16/10/02 00:16:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1445128842_0002

16/10/02 00:16:49 INFO mapreduce.Job: The url to track the job: http://localhost:8080/

16/10/02 00:16:49 INFO mapred.LocalJobRunner: OutputCommitter set in config null

16/10/02 00:16:49 INFO mapreduce.Job: Running job: job_local1445128842_0002

16/10/02 00:16:49 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:49 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

16/10/02 00:16:49 INFO mapred.LocalJobRunner: Waiting for map tasks

16/10/02 00:16:49 INFO mapred.LocalJobRunner: Starting task: attempt_local1445128842_0002_m_000000_0

16/10/02 00:16:49 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:49 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:49 INFO mapred.MapTask: Processing split: file:/home/hadoop-2.7.3/grep-temp-965988443/part-r-00000:0+111

16/10/02 00:16:49 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

16/10/02 00:16:49 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

16/10/02 00:16:49 INFO mapred.MapTask: soft limit at 83886080

16/10/02 00:16:49 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

16/10/02 00:16:49 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

16/10/02 00:16:49 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

16/10/02 00:16:49 INFO mapred.LocalJobRunner:

16/10/02 00:16:49 INFO mapred.MapTask: Starting flush of map output

16/10/02 00:16:49 INFO mapred.MapTask: Spilling map output

16/10/02 00:16:49 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600

16/10/02 00:16:49 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600

16/10/02 00:16:49 INFO mapred.MapTask: Finished spill 0

16/10/02 00:16:49 INFO mapred.Task: Task:attempt_local1445128842_0002_m_000000_0 is done. And is in the process of committing

16/10/02 00:16:49 INFO mapred.LocalJobRunner: map

16/10/02 00:16:49 INFO mapred.Task: Task 'attempt_local1445128842_0002_m_000000_0' done.

16/10/02 00:16:49 INFO mapred.LocalJobRunner: Finishing task: attempt_local1445128842_0002_m_000000_0

16/10/02 00:16:49 INFO mapred.LocalJobRunner: map task executor complete.

16/10/02 00:16:49 INFO mapred.LocalJobRunner: Waiting for reduce tasks

16/10/02 00:16:49 INFO mapred.LocalJobRunner: Starting task: attempt_local1445128842_0002_r_000000_0

16/10/02 00:16:49 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1

16/10/02 00:16:49 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]

16/10/02 00:16:49 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1673725c

16/10/02 00:16:49 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10

16/10/02 00:16:49 INFO reduce.EventFetcher: attempt_local1445128842_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events

16/10/02 00:16:49 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local1445128842_0002_m_000000_0 decomp: 21 len: 25 to MEMORY

16/10/02 00:16:49 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local1445128842_0002_m_000000_0

16/10/02 00:16:49 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->21

16/10/02 00:16:49 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning

16/10/02 00:16:49 WARN io.ReadaheadPool: Failed readahead on ifile

EBADF: Bad file descriptor

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)

at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)

at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

16/10/02 00:16:49 INFO mapred.LocalJobRunner: 1 / 1 copied.

16/10/02 00:16:49 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs

16/10/02 00:16:49 INFO mapred.Merger: Merging 1 sorted segments

16/10/02 00:16:49 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes

16/10/02 00:16:49 INFO reduce.MergeManagerImpl: Merged 1 segments, 21 bytes to disk to satisfy reduce memory limit

16/10/02 00:16:49 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk

16/10/02 00:16:49 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce

16/10/02 00:16:49 INFO mapred.Merger: Merging 1 sorted segments

16/10/02 00:16:49 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes

16/10/02 00:16:49 INFO mapred.LocalJobRunner: 1 / 1 copied.

16/10/02 00:16:49 INFO mapred.Task: Task:attempt_local1445128842_0002_r_000000_0 is done. And is in the process of committing

16/10/02 00:16:49 INFO mapred.LocalJobRunner: 1 / 1 copied.

16/10/02 00:16:49 INFO mapred.Task: Task attempt_local1445128842_0002_r_000000_0 is allowed to commit now

16/10/02 00:16:49 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1445128842_0002_r_000000_0' to file:/home/hadoop-2.7.3/output/_temporary/0/task_local1445128842_0002_r_000000

16/10/02 00:16:49 INFO mapred.LocalJobRunner: reduce > reduce

16/10/02 00:16:49 INFO mapred.Task: Task 'attempt_local1445128842_0002_r_000000_0' done.

16/10/02 00:16:49 INFO mapred.LocalJobRunner: Finishing task: attempt_local1445128842_0002_r_000000_0

16/10/02 00:16:49 INFO mapred.LocalJobRunner: reduce task executor complete.

16/10/02 00:16:50 INFO mapreduce.Job: Job job_local1445128842_0002 running in uber mode : false

16/10/02 00:16:50 INFO mapreduce.Job: map 100% reduce 100%

16/10/02 00:16:50 INFO mapreduce.Job: Job job_local1445128842_0002 completed successfully

16/10/02 00:16:50 INFO mapreduce.Job: Counters: 30

File System Counters

FILE: Number of bytes read=1248270

FILE: Number of bytes written=2321184

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

Map-Reduce Framework

Map input records=1

Map output records=1

Map output bytes=17

Map output materialized bytes=25

Input split bytes=121

Combine input records=0

Combine output records=0

Reduce input groups=1

Reduce shuffle bytes=25

Reduce input records=1

Reduce output records=1

Spilled Records=2

Shuffled Maps =1

Failed Shuffles=0

Merged Map outputs=1

GC time elapsed (ms)=0

Total committed heap usage (bytes)=695205888

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=123

File Output Format Counters

Bytes Written=23

[root@nuc12-i7 hadoop-2.7.3]# cat output/*

1 dfsadmin

[root@nuc12-i7 hadoop-2.7.3]# ls output

part-r-00000 _SUCCESS

Tuesday, June 7, 2016

JAX-RS 2.0 Client API on Tomcat 8 running Spring 4, Hibernate 4 with a Jersey 2 RestController

Rest Client

package com.obrienlabs.gps.integration;

import javax.ws.rs.client.Client;

import javax.ws.rs.client.ClientBuilder;

import javax.ws.rs.client.Entity;

import javax.ws.rs.client.WebTarget;

import javax.ws.rs.core.Response;

import com.obrienlabs.gps.business.entity.Record;

import com.obrienlabs.gps.business.entity.User;

public class JAXRSClient {

public void run() {

Long id = 1L;

Client client = ClientBuilder.newClient();

WebTarget rootTarget = client.target("http://127.0.0.1:8080/biometric/rest/read");

WebTarget latestTarget = rootTarget.path("json/latest");

WebTarget finalTarget = latestTarget.path(id.toString());

Record record = finalTarget.request().get(Record.class);

System.out.println(record);

client.close();

}

public static void main(String[] args) {

JAXRSClient client = new JAXRSClient();

client.run();

}

Rest Backend

pom.xml

RestController

@Path("/read")

////@Produces("text/plain")

//@Consumes("application/json")

//@RestController

//@Service("daoFacade")

//@TransactionConfiguration(transactionManager = "transactionManager", defaultRollback = true)

@RequestMapping("/read")

@Component

public class RestServiceImpl extends Application {

// https://jersey.java.net/documentation/latest/spring.html

// http://stackoverflow.com/questions/3027834/inject-a-ejb-into-jax-rs-restfull-service

// not supported until 2.0 https://java.net/jira/browse/JERSEY-517

//@EJB(name="ejb/ApplicationService") // not for JAX-RS

// 20160606: spring injection requires jersey-spring3 from org.glassfish.jersey.ext with spring 4 exclusions in the pom

@Inject

//@Autowired

@Qualifier("daoFacade")

private ApplicationServiceLocal applicationServiceLocal;// = new ApplicationService();

@GET

@Path("/json/record/{identifier}")

@Produces(MediaType.APPLICATION_JSON)

public String getRecordJSON(@PathParam("identifier")String identifier) {

return Integer.toString(read(identifier).size());

}

Friday, April 29, 2016

VM creation for Microservices HA infrastructure

Amazon EC2, RDS and ElasticBeanstalk work very well but are a bit expensive when you have your own 12 core systems at home that can be used for most backend services.

VM Boostrap

Normally I would use Chef to provision the VM. This is the manual configuration list to get a repeatable VM snapshot built.

We are running a RHEL 7.3 VM on top of either Windows 10 or MacOS 10

Register System
Attach Subscription

1. Change Networks and hostname (some services need differentiation in an HA environment)
[root@localhost ~]# vi /etc/hosts
[rootRHEL7@localhost ~]# hostnamectl set-hostname biometric-vm01
[rootRHEL6@localhost ~]# vi /etc/sysconfig/network
[root@localhost ~]# systemctl restart systemd-hostnamed

verify /etc/hosts has the correct ip set for the VM

 If you have 2 NIC's in this case the wired nic is on a private internal subnet with no internet access and the wireless nic is our internet access - you will need to change the METRIC order.  You need to set the internet live nic as the first in the routing sequence by setting the METRIC lowest on wireless and turning off DEFROUTE, IPV4_DEFROUTE, PEERDNS etc on the private wired nic.

Check regenerated
[root@nuc14-i3 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.100.1

Make the non-internet nic secondary
[root@nuc14-i3 ~]# cat /etc/sysconfig/network-scripts/ifcfg-enp0s25 
TYPE="Ethernet"
BOOTPROTO="dhcp"
DEFROUTE=no
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="enp0s25"
UUID="938325b5-6c08-49eb-8044-8fd5dfe931fe"
DEVICE="enp0s25"
ONBOOT="yes"
PEERDNS=no
PEERROUTES=yes
IPV6_PEERDNS=no
IPV6_PEERROUTES=yes
METRIC=200

2. Install Java 8
[root@biometric-vm01 ~]# alternatives --config java
Selection Command
-----------------------------------------------
1 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64/jre/bin/java
*+ 2 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64/jre/bin/java

Ideally install the SUN/Oracle JDK
download from http://java.sun.com

3. yum update -y

4. Install Tomcat (In the past i would install a full EE server like WebLogic 12c but a Tomcat + Spring container is now the way forward)

firewall-cmd --permanent --zone=public --add-port=8080/tcp

sudo groupadd tomcat

sudo useradd -M -s /bin/nologin -g tomcat -d /opt/tomcat tomcat

cd ~

http://apache.mirror.rafal.ca/tomcat/tomcat-8/v8.0.39/bin/apache-tomcat-8.0.39.tar.gz

sudo mkdir /opt/tomcat

sudo tar xvf apache-tomcat-8*tar.gz -C /opt/tomcat --strip-components=1

cd /opt/tomcat

sudo chgrp -R tomcat conf

sudo chmod g+rwx conf

sudo chmod g+r conf/*
cd /opt

sudo chown -R tomcat tomcat/

sudo vi /etc/systemd/system/tomcat.service

[Unit]
    Description=Apache Tomcat8
    After=syslog.target network.target

[Service]
    Type=forking
    User=tomcat
    Group=tomcat
    Environment=JAVA_HOME=/usr/lib/jvm/jre

    Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid
    Environment=CATALINA_HOME=/opt/tomcat
    Environment=CATALINA_BASE=/opt/tomcat
    Environment='CATALINA_OPTS=-Xms512M -Xmx4096M -server -XX:+UseParallelGC'
    Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'
    ExecStart=/opt/tomcat/bin/startup.sh
    ExecStop=/bin/kill -15 $MAINPID

[Install]
    WantedBy=multi-user.target

72 sudo systemctl daemon-reload

73 sudo systemctl start tomcat

74 sudo systemctl enable tomcat

81 systemctl restart firewalld.service

[root@rhel73os0prod opt]# systemctl status tomcat.service
● tomcat.service - Apache Tomcat8
Loaded: loaded (/etc/systemd/system/tomcat.service; disabled; vendor preset: disabled)
Active: active (running) since Sat 2016-12-10 22:07:05 EST; 12s ago
Process: 4731 ExecStart=/opt/tomcat/bin/startup.sh (code=exited, status=0/SUCCESS)
Main PID: 4738 (java)
CGroup: /system.slice/tomcat.service
└─4738 /usr/lib/jvm/jre/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava...

Dec 10 22:07:05 rhel73os0prod systemd[1]: Starting Apache Tomcat8...

Dec 10 22:07:05 rhel73os0prod systemd[1]: Started Apache Tomcat8.

5. Setup developer services

6. PostgreSQL install - for ORM services via Hibernate or better EclipseLink

7. MySQL install

http://dev.mysql.com/doc/refman/5.7/en/linux-installation-yum-repo.html
cd Downloads
yum repolist enabled | grep "mysql.*-community.*"
yum localinstall mysql57-community-release-el7-8.noarch.rpm
yum install mysql-community-server

Installing : mysql-community-common-5.7.12-1.el7.x86_64 1/6
Installing : mysql-community-libs-5.7.12-1.el7.x86_64 2/6
Installing : mysql-community-client-5.7.12-1.el7.x86_64 3/6
Installing : mysql-community-server-5.7.12-1.el7.x86_64 4/6
Installing : mysql-community-libs-compat-5.7.12-1.el7.x86_64 5/6
Erasing : 1:mariadb-libs-5.5.47-1.el7_2.x86_64 6/6
Verifying : mysql-community-client-5.7.12-1.el7.x86_64 1/6
Verifying : mysql-community-server-5.7.12-1.el7.x86_64 2/6
Verifying : mysql-community-common-5.7.12-1.el7.x86_64 3/6
Verifying : mysql-community-libs-5.7.12-1.el7.x86_64 4/6
Verifying : mysql-community-libs-compat-5.7.12-1.el7.x86_64 5/6
Verifying : 1:mariadb-libs-5.5.47-1.el7_2.x86_64 6/6

service mysqld start
grep 'temporary password' /var/log/mysqld.log
mysql -uroot -p
mysql> ALTER User 'root'@'localhost' IDENTIFIED BY 'S';
/usr/bin/mysql_secure_installation
mysql -u root -p

mysql>create database biometric;

here

mysql>create user 'obrienlabs'@'192.168.0.21' IDENTIFIED BY 'OC';

mysql>grant all on biometric.* to 'obrienlabs'@'192.168.0.21';
mysql> create user 'obriensystems'@'%' IDENTIFIED BY 'S';
mysql> grant all on biometric.* to 'obriensystems'@'%';
mysql>flush privileges;

firewall-cmd --permanent --zone=public --add-service=mysql
firewall-cmd --permanent --zone=public --add-port=3306/tcp
systemctl restart firewalld.service

help hibernate run the following

private TypedQuery<Record> getTypedCriteriaQuery(EntityManager entityManager, String user) {

CriteriaBuilder cb = entityManager.getCriteriaBuilder();

CriteriaQuery<Record> query = cb.createQuery(Record.class);

Root<Record> target = query.from(Record.class);

// workaround for http://stackoverflow.com/questions/16348354/how-do-i-write-a-max-query-with-a-where-clause-in-jpa-2-0

SingularAttribute<? super Record, Long> anAttribute = entityManager.getMetamodel()

.entity(Record.class).getSingularAttribute("tsStop", Long.class);

query.orderBy(

cb.desc(target.get(anAttribute)));

query.where(

cb.equal(target.get("userId"), user));

TypedQuery<Record> typedQuery = entityManager.createQuery(query);

return typedQuery;

}

#https://github.com/Elgg/Elgg/issues/8121
# remove ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES
SET GLOBAL sql_mode = 'NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
SET SESSION sql_mode = 'NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';

SELECT @@GLOBAL.sql_mode;
SELECT @@SESSION.sql_mode;

8. Neo4J install - for graph based nosql (a geohash tree for example)

9. JPA

Hibernate on Spring Repository


<?xml version="1.0"?>

<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xmlns:task="http://www.springframework.org/schema/task"

       xmlns:context="http://www.springframework.org/schema/context"

       xmlns:tx="http://www.springframework.org/schema/tx"

       xmlns:p="http://www.springframework.org/schema/p"

       xmlns:util="http://www.springframework.org/schema/util"

       xmlns:mvc="http://www.springframework.org/schema/mvc"

       xsi:schemaLocation="http://www.springframework.org/schema/beans

                           http://www.springframework.org/schema/beans/spring-beans.xsd

                           http://www.springframework.org/schema/context

                           http://www.springframework.org/schema/context/spring-context.xsd

                           http://www.springframework.org/schema/task

                           http://www.springframework.org/schema/task/spring-task.xsd

                           http://www.springframework.org/schema/mvc

                           http://www.springframework.org/schema/mvc/spring-mvc.xsd

                           http://www.springframework.org/schema/tx

                           http://www.springframework.org/schema/tx/spring-tx.xsd

                           http://www.springframework.org/schema/util

                           http://www.springframework.org/schema/util/spring-util.xsd">



    <context:component-scan base-package="com....orm"/>

    <context:annotation-config/>



    <!-- mvc:annotation-driven/-->

    <!-- tx:annotation-driven/-->

    

    <!-- bean id="contextAwareMBeanServer" class="com.common.DefaultServerMBean" />



    <bean id="dbConfig" class="com.common.util.NspConfig" depends-on="contextAwareMBeanServer">

        <constructor-arg type="java.lang.String" index="0" name="aInFileName" value="pgsql.conf" />

    </bean-->



    <!-- bean class="com..app.ts.bwcal.config.SwaggerConfig"/-->



    <bean class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor"/>

    <bean class="org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor"/>



    <bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">

    <!-- bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource" destroy-method="close"-->    

        <property name="driverClass" value="org.postgresql.Driver" />

        <property name="jdbcUrl" value="jdbc:postgresql://127.0.0.1:6432/orbwdb?ssl=true&amp;sslfactory=org.postgresql.ssl.NonValidatingFactory" />

        <property name="user" value="taskscheduler" />

        <property name="password" value="taskscheduler" />

        <property name="minPoolSize" value="10"/>

        <property name="maxPoolSize" value="20"/>

        <property name="acquireIncrement" value="2"/>

        <property name="maxStatementsPerConnection" value="100"/>

        <property name="testConnectionOnCheckin" value="true" />

        <property name="testConnectionOnCheckout" value="true" />

        <property name="idleConnectionTestPeriod" value="90" />

        <property name="preferredTestQuery" value="SELECT 1" />        

    </bean>

    

    <!-- bean id="liquibase2" class="liquibase.integration.spring.SpringLiquibase">

        <property name="dataSource" ref="dataSource" />

        <property name="changeLog" value="classpath:db/changelog/db-changelog-master.xml" />

        <property name="defaultSchema" value="taskscheduler" />

    </bean-->  



    <bean id="jpaAdapter" class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">

        <property name="databasePlatform" value="org.hibernate.dialect.PostgreSQLDialect"/>

        <property name="generateDdl" value="true"/>

        <property name="showSql" value="true"/>

    </bean>



    <bean id="emf" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean"

          p:jpaVendorAdapter-ref="jpaAdapter">

        <property name="persistenceUnitName" value="openflow"></property>

        <property name="dataSource" ref="dataSource"></property>

        <property name="jpaProperties">

            <props>

                <prop key="hibernate.hbm2ddl.auto">create-drop</prop>

                <prop key="hibernate.flushMode">FLUSH_AUTO</prop>

                <prop key="hibernate.temp.use_jdbc_metadata_defaults">false</prop>

            </props>

        </property>

    </bean>



    <bean id="txManager" class="org.springframework.orm.jpa.JpaTransactionManager"

          p:entityManagerFactory-ref="emf"/>



    <tx:annotation-driven transaction-manager="txManager"/>



</beans>

JPA Persistence

Saturday, October 1, 2016

Machine Learning using Apache Spark on top of Apache Hadoop and Kafka

Tuesday, June 7, 2016

JAX-RS 2.0 Client API on Tomcat 8 running Spring 4, Hibernate 4 with a Jersey 2 RestController

Rest Client

Rest Backend

pom.xml

RestController

Friday, April 29, 2016

VM creation for Microservices HA infrastructure

VM Boostrap

9. JPA

Hibernate on Spring Repository

EclipseLink on Spring Repository

Hibernate on Spring-data-jpa

EclipseLink on Spring-data-jpa

Total Pageviews

Related Blogs

Blog Archive

About Me

Followers