Raspberry PI cluster of 32 boards - Under Construction:
Disclaimer: For serious computational power - build your own i7-5820 and use CUDA on an nVidia GTX-970. The raspberry PI cluster is more of a "build it - and they will come" exploration exercise.http://eclipsejpa.blogspot.ca/2015/03/haswell-intel-i7-5820k-overclock-pc.html
This article is an ongoing discussion of how to get a cluster of 32 raspberry pi boards up and running.
 |
| 32 node/board Raspberry PI cluster for parallel processing experimentation |
The ARM based Raspberry PI board is an excellent platform to investigate various parallel processing configurations. If we are looking for pure performance then I would stick with an Intel core i7 and a CUDA based NVidia GPU because a single core raspberry PI is about 40 times slower than a single core of an 2nd gen i7-2600 or 3rd gen i7-3610 (about 140 times slower than an 8-thread ForkJoin implementation). This however is not our goal, we need an efficient and accessible way to run multiple servers - and the pi does this at about $70 per server (board + connectors + 16GB SD) and 4 watts/node. For example: it would take $9800 of raspberry PI boards with 70G ram to equal one 3rd gen i7 at $1300 with 24G ram. But we can build a cluster of 8 raspberry PI servers with $560 as opposed to 8 i7 boxes with $10000.
 |
| 8 board Raspberry PI cluster |
This tutorial details how to get a networked cluster (bramble) of (eight for now) Raspberry PI boards running as a single distributed auxillary processing unit to a controlling Java EE server - ideally using Hadoop. The primary goal of this exercise is for distributed experimentation. As I configure and acquire multiple raspberry pi boards and work out power distribution issues my cluster will increase in size. I currently work with 8 boards and 8 spares. The cluster of raspberry PI s can be distributed work using a custom RPC API like remote stateless sessions beans on top of RMI or they can use a formal MapReduce implementation like Hadoop or even MPI.
After running the Oracle embedded ARM JVM with no problems on the REV A raspberry PI using the distribution from Element14, I was not immediately successfull running the JVM on the new REV B (512Mb) version because the default Debian distribution from Element 14 no longer uses the soft float version. I get the following missing library error.
pi@raspberrypi ~/java/ejre1.7.0_06 $ java -version
java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
Download a new OS compatible with the Java 7 JDK here
http://www.raspberrypi.org/downloads
Choose "Soft-float Debian “wheezy”" = 2012-08-08-wheezy-armel.zip
Write it to your SD card
https://launchpad.net/win32-image-writer/+download
Reinstall Java (curently 1.7.0_10 from Oracle)
You are good to go [Java 7 SE on the Raspberry PI Rev B]. So we can now use Fork-Join, JAXB and JAX-WS webservices (using a single thread however).
DI 1: Powering your Raspberry PI cluster
Using a good Agilent power supply we use from 3 (idle) to 3.5A (startup) (6-7W @ 5V) for 8 boards.
A good ATX power supply will suffice to power a cluster of raspberry pi boards. In this example I have a 450W supply which supplies 30A of 5V power (make sure you put some load on the 5V and 12V rails as well).
As you can see I have yet to fully integrate the power supply interface between the ATX supply and the breadboard bus for the 8 pi's - but we are functioning fine and are no longer limited by the bench supplies or individual 5V USB connectors. (the blue LED boards are Parallax Propeller 8-core microcontrollers uses a per/core output indicator for now.
I need some sort of protection fuse - in case of a short circuit. It was very stressful connecting up my 8 raspberry pi boards up to the ATX after testing on one. I recommend working with all GPIO pins (one 1 header is populated instead of 2 on the latest rev B board) covered by a flat cable connector.
20130126: I now have 24 of 32 raspberry pi boards powered up however running the full peak 15A off an ATX power supply is not practical as it requires some serious wire guage as my 3 wire 24 guage setup is overheating. Also if you accidently short the power supply you will use the full 15-40A and burn your wire. I accidentally shorted the leads on a 5A supply on my metal breadboard and the supply wire started to smell and melt. This brings us to the recommend way to power a large cluster of raspberry pi boards - separate bench power supplies. When I shorted the supply the bench supply held stead at 5.2A which is safe enough not to burn your house down before you notice it.
Recommended power supply setup for 32 raspberry pi board cluster
No more than 8 raspberry pi boards per 5A power supply will allow you to add some peripherals like an adafruit display or a propeller 8-core coprocessor on an SPI bus.So this is kind of expensive but instead of using a 40A bench supply at around $350 I use 4 separate 5A bench supplies (3 Circuit-Test PSC-520 supplies @ 3 x $225 and 1 Agilent U8002A supply @ $450).
DI 2: Updating your board for 512mb RAM (470Mb from 224Mb)
The Rev 2 board has double the ram but will require updated firmware to enable it.https://github.com/Hexxeh/rpi-update
sudo wget http://goo.gl/1BOfJ -O /usr/bin/rpi-update && sudo chmod +x /usr/bin/rpi-update
sudo apt-get install git-core
sudo rpi-update
- reboot after firmware update
DI 3: Overclocking
The lan chip heats up to 52 degrees celsius from a normal 45 when the raspberry pi is overclocked from 700 to 800 MHz.
DI 4: Setup Networking
Wireless is kind of unreliable, I recommend wired.The WiPi module from Element 14 works essentially out of the box
Wired networking
After duplicating all the 32 SD cards, put one at a time into one of the raspberry pi boards and change the hostname, hosts and static network interfaces settingssudo nano /etc/hostname
sudo nano /etc/hosts
sudo nano /etc/network/interfaces
iface eth0 inet static
address 192.168.4.101
netmask 255.255.255.0
gateway 192.168.4.1
nameserver 4.2.2.1
# here we do not rely on our internet providers's DNS servers - we use the google server at 4.2.2.1 as more reliable DNS server
sudo nano /etc/resolv.conf
nameserver 4.2.2.1
DI 5: Setup Java
Setup Tomcat
Setup Fortran and MPICH
Issue is that I lower performance (likely network overhead) when I increase the number of nodes (currently 6 pi's)
pi@rpi0 ~ $ mpiexec -f machinefile -n 1 ~/mpich_build/examples/cpi
Process 0 of 1 is on rpi0
pi is approximately 3.1415926544231341, Error is 0.0000000008333410
wall clock time = 0.017286
pi@rpi0 ~ $ mpiexec -f machinefile -n 2 ~/mpich_build/examples/cpi
Process 0 of 2 is on rpi0
Process 1 of 2 is on rpi1
pi is approximately 3.1415926544231318, Error is 0.0000000008333387
wall clock time = 0.020435
pi@rpi0 ~ $ mpiexec -f machinefile -n 4 ~/mpich_build/examples/cpi
Process 1 of 4 is on rpi1
Process 0 of 4 is on rpi0
Process 2 of 4 is on rpi2
Process 3 of 4 is on rpi3
pi is approximately 3.1415926544231239, Error is 0.0000000008333307
wall clock time = 0.037727
pi@rpi0 ~ $ mpiexec -f machinefile -n 6 ~/mpich_build/examples/cpi
Process 2 of 6 is on rpi0
Process 1 of 6 is on rpi1
Process 0 of 6 is on rpi2
Process 3 of 6 is on rpi3
Process 4 of 6 is on rpi4
Process 5 of 6 is on rpi5
pi is approximately 3.1415926544231239, Error is 0.0000000008333307
wall clock time = 0.043331
Log:
20121121: Setup 4 networked PIs
20130127: power up of 24 raspberry pi boards
BOM:
32 Raspberry PI boards from Element 14 @ $35 = $1120
32 Sandisk Ultra 16GB SD cards @ $10-18 = $320-576
0 micro USB cables = $0
1 HDMI cable from Apple = $20
8 power supply cables @ 10 = $80
4 bench 5A power supplies from Agilent or Circuit-Test @224-450 = $896-1800
4 large breadboards (that fit 8 raspberry pi boards) @ 45 = $180
64 bendable arduino headers from www.evilmadscience.com @ $1 = $64
5 Gigabit 8 node network hubs or 2 16 node hubs from Dlink @ 65 = $325
32 belkin flexible network cables from the Apple store @ 15 = $480
Total = $3885.00
Copies of this article
http://www.framboise314.fr/32-raspberry-pi-pour-du-calcul-parallele-en-java/
Links
Objective C on Raspberry PI - http://www.techotopia.com/index.php/Installing_and_Using_GNUstep_and_Objective-C_on_Linux
