07/17/2016: Docker Swarm 1.12 on PicoCluster
This post shows the steps I followed to run Docker Swarm 1.12 on my 5-node Raspberry PI PicoCluter.
I followed the directions at https://medium.com/@bossjones/how-i-setup-a-raspberry-pi-3-cluster-using-the-new-docker-swarm-mode-in-29-minutes-aa0e4f3b1768#.ma06iyonf but tweaked them a bit.
First off, I wanted to have my cluster using eth0 to connect to my laptop and then share its WiFi connection. Using this technique means that my WiFi network name and password are not on the cluster. So the cluster should be able to plug into any laptop or server without changes. Follow instructions at https://t.co/2jRbNAOiCU to share your eth0 connection.
Use lsblk to umount any directories on the SD cards you’ll be using.
Now flash (https://github.com/hypriot/flash) the SD cards using the flash tool from hypriot. Notice that no network information is provided.
I used piX naming convention so that I can easily loop over all five RPI in the PicoCluster.
flash --hostname pi1 --device /dev/mmcblk0 https://github.com/hypriot/image-builder-rpi/releases/download/v0.8.1/hypriotos-rpi-v0.8.1.img.zip
flash --hostname pi2 --device /dev/mmcblk0 https://github.com/hypriot/image-builder-rpi/releases/download/v0.8.1/hypriotos-rpi-v0.8.1.img.zip
flash --hostname pi3 --device /dev/mmcblk0 https://github.com/hypriot/image-builder-rpi/releases/download/v0.8.1/hypriotos-rpi-v0.8.1.img.zip
flash --hostname pi4 --device /dev/mmcblk0 https://github.com/hypriot/image-builder-rpi/releases/download/v0.8.1/hypriotos-rpi-v0.8.1.img.zip
flash --hostname pi5 --device /dev/mmcblk0 https://github.com/hypriot/image-builder-rpi/releases/download/v0.8.1/hypriotos-rpi-v0.8.1.img.zip
Using this function, you can find the IP addresses for the RPI.
function getip() { (traceroute $1 2>&1 | head -n 1 | cut -d\( -f 2 | cut -d\) -f 1) }
List the IP addresses.
for i in `seq 1 5`; do
echo "HOST: pi$i; IP: $(getip pi$i.local)";
done
Remove any fingerprints for the RPI.
for i in `seq 1 5`; do
ssh-keygen -R pi${i}.local 2>/dev/null;
done
Create variable to hold ssh options you’ll use frequently
export SSH_OPTIONS="-oStrictHostKeyChecking=no -oCheckHostIP=no"
Copy your PKI identity to the RPI.
for i in `seq 1 5`; do
ssh-copy-id $SSH_OPTOINS pirate@pi${i}.local;
done
Download the deb file for Docker v1.12
curl -O https://jenkins.hypriot.com/job/armhf-docker/17/artifact/bundles/latest/build-deb/raspbian-jessie/docker-engine_1.12.0%7Erc4-0%7Ejessie_armhf.deb
Copy the deb file to the RPI
for i in `seq 1 5`; do
scp $SSH_OPTIONS docker-engine_1.12.0%7Erc4-0%7Ejessie_armhf.deb pirate@pi$i.local:.;
done
Remove older Docker version from the RPI
for i in `seq 1 5`; do
ssh $SSH_OPTIONS pirate@pi$i.local sudo apt-get purge -y docker-hypriot;
done
Install Docker
for i in `seq 1 5`; do
ssh $SSH_OPTIONS pirate@pi$i.local sudo dpkg -i docker-engine_1.12.0%7Erc4-0%7Ejessie_armhf.deb;
done
Initialize the Swarm
ssh $SSH_OPTIONS pirate@pi1.local docker swarm init
Join slaves to Swarm - replace the join command below with the specific one displayed by the init command.
for i in `seq 2 5`; do
ssh $SSH_OPTIONS pirate@pi$i.local docker swarm join --secret ceuok9jso0klube8m3ih9gcsv --ca-hash sha256:f0864eb57963e3f9cd1756e691d0b609903e3a0bb48785272ea53155809025ee 10.42.0.49:2377;
done
Exercise the Swarm
ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi1.local
docker service create --name ping hypriot/rpi-alpine-scratch ping 8.8.8.8
docker service tasks ping
docker service update --replicas 10 ping
docker service tasks ping
docker service rm ping
Shutdown the Swarm
for i in `seq 1 5`; do
ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi$i.local sudo shutdown -h +0 "Shutting down";
done
07/05/2016: How I Got Apache Spark to Sort Of (Not Really) Work on my PicoCluster of 5 Raspberry PI
I've read several blog posts about people running Apache Spark on a Raspberry PI. It didn't seem too hard so I thought I've have a go at it. But the results were disappointing. Bear in mind that I am a Spark novice so some setting is probably. I ran into two issues - memory and heartbeats.
So, this what I did.
I based my work on these pages:
* https://darrenjw2.wordpress.com/2015/04/17/installing-apache-spark-on-a-raspberry-pi-2/
* https://darrenjw2.wordpress.com/2015/04/18/setting-up-a-standalone-apache-spark-cluster-of-raspberry-pi-2/
* http://www.openkb.info/2014/11/memory-settings-for-spark-standalone_27.html
I created five SD cards according to my previous blog post (see http://affy.blogspot.com/2016/06/how-did-i-prepare-my-picocluster-for.html).
Installation of Apache Spark
* install Oracle Java and Python
for i in `seq 1 5`; do (ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local sudo apt-get install -y oracle-java8-jdk python2.7 &); done
* download Spark
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.6.2-bin-hadoop2.6.tgz
* Copy Spark to all RPI
for i in `seq 1 5`; do (scp -q -oStrictHostKeyChecking=no -oCheckHostIP=no spark-1.6.2-bin-hadoop2.6.tgz pirate@pi0${i}.local:. && echo "Copy complete to pi0${i}" &); done
* Uncompress Spark
for i in `seq 1 5`; do (ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local tar xfz spark-1.6.2-bin-hadoop2.6.tgz && echo "Uncompress complete to pi0${i}" &); done
* Remove tgz file
for i in `seq 1 5`; do (ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local rm spark-1.6.2-bin-hadoop2.6.tgz); done
* Add the following to your .bashrc file on each RPI. I can't figure out how to put this into a loop.
export SPARK_LOCAL_IP="$(ip route get 1 | awk '{print $NF;exit}')"
* Run Standalone Spark Shell
ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi01.local
cd spark-1.6.2-bin-hadoop2.6
bin/run-example SparkPi 10
bin/spark-shell --master local[4]
# This takes several minutes to display a prompt.
# While the shell is running, visit http://pi01.local:4040/
scala> sc.textFile("README.md").count
# After the job is complete, visit the monitor page.
scala> exit
* Run PyShark Shell
bin/pyspark --master local[4]
>>> sc.textFile("README.md").count()
>>> exit()
CLUSTER
Now for the clustering...
* Enable password-less SSH between nodes
ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi01.local
for i in `seq 1 5`; do avahi-resolve --name pi0${i}.local -4 | awk ' { t = $1; $1 = $2; $2 = t; print; } ' | sudo tee --append /etc/hosts; done
echo "$(ip route get 1 | awk '{print $NF;exit}') $(hostname).local" | sudo tee --append /etc/hosts
ssh-keygen
for i in `seq 1 5`; do ssh-copy-id pirate@pi0${i}.local; done
* Configure Spark for Cluster
cd spark-1.6.2-bin-hadoop2.6/conf
create a slaves file with the following contents
pi01.local
pi02.local
pi03.local
pi04.local
pi05.local
cp spark-env.sh.template spark-env.sh
In spark-env.sh
Set SPARK_MASTER_IP the results of "ip route get 1 | awk '{print $NF;exit}'"
SPARK_WORKER_MEMORY=512m
* Copy the spark environment script to the other RPI
for i in `seq 2 5`; do scp spark-env.sh pirate@pi0${i}.local:spark-1.6.2-bin-hadoop2.6/conf/; done
* Start the cluster
cd ..
sbin/start-all.sh
* Visit the monitor page
http://192.168.1.8:8080
And everything is working so far! But ...
* Start a Spark Shell
bin/spark-shell --executor-memory 500m --driver-memory 500m --master spark://pi01.local:7077 --conf spark.executor.heartbeatInterval=45s
And this fails...
So, this what I did.
I based my work on these pages:
* https://darrenjw2.wordpress.com/2015/04/17/installing-apache-spark-on-a-raspberry-pi-2/
* https://darrenjw2.wordpress.com/2015/04/18/setting-up-a-standalone-apache-spark-cluster-of-raspberry-pi-2/
* http://www.openkb.info/2014/11/memory-settings-for-spark-standalone_27.html
I created five SD cards according to my previous blog post (see http://affy.blogspot.com/2016/06/how-did-i-prepare-my-picocluster-for.html).
Installation of Apache Spark
* install Oracle Java and Python
for i in `seq 1 5`; do (ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local sudo apt-get install -y oracle-java8-jdk python2.7 &); done
* download Spark
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.6.2-bin-hadoop2.6.tgz
* Copy Spark to all RPI
for i in `seq 1 5`; do (scp -q -oStrictHostKeyChecking=no -oCheckHostIP=no spark-1.6.2-bin-hadoop2.6.tgz pirate@pi0${i}.local:. && echo "Copy complete to pi0${i}" &); done
* Uncompress Spark
for i in `seq 1 5`; do (ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local tar xfz spark-1.6.2-bin-hadoop2.6.tgz && echo "Uncompress complete to pi0${i}" &); done
* Remove tgz file
for i in `seq 1 5`; do (ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local rm spark-1.6.2-bin-hadoop2.6.tgz); done
* Add the following to your .bashrc file on each RPI. I can't figure out how to put this into a loop.
export SPARK_LOCAL_IP="$(ip route get 1 | awk '{print $NF;exit}')"
* Run Standalone Spark Shell
ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi01.local
cd spark-1.6.2-bin-hadoop2.6
bin/run-example SparkPi 10
bin/spark-shell --master local[4]
# This takes several minutes to display a prompt.
# While the shell is running, visit http://pi01.local:4040/
scala> sc.textFile("README.md").count
# After the job is complete, visit the monitor page.
scala> exit
* Run PyShark Shell
bin/pyspark --master local[4]
>>> sc.textFile("README.md").count()
>>> exit()
CLUSTER
Now for the clustering...
* Enable password-less SSH between nodes
ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi01.local
for i in `seq 1 5`; do avahi-resolve --name pi0${i}.local -4 | awk ' { t = $1; $1 = $2; $2 = t; print; } ' | sudo tee --append /etc/hosts; done
echo "$(ip route get 1 | awk '{print $NF;exit}') $(hostname).local" | sudo tee --append /etc/hosts
ssh-keygen
for i in `seq 1 5`; do ssh-copy-id pirate@pi0${i}.local; done
* Configure Spark for Cluster
cd spark-1.6.2-bin-hadoop2.6/conf
create a slaves file with the following contents
pi01.local
pi02.local
pi03.local
pi04.local
pi05.local
cp spark-env.sh.template spark-env.sh
In spark-env.sh
Set SPARK_MASTER_IP the results of "ip route get 1 | awk '{print $NF;exit}'"
SPARK_WORKER_MEMORY=512m
* Copy the spark environment script to the other RPI
for i in `seq 2 5`; do scp spark-env.sh pirate@pi0${i}.local:spark-1.6.2-bin-hadoop2.6/conf/; done
* Start the cluster
cd ..
sbin/start-all.sh
* Visit the monitor page
http://192.168.1.8:8080
And everything is working so far! But ...
* Start a Spark Shell
bin/spark-shell --executor-memory 500m --driver-memory 500m --master spark://pi01.local:7077 --conf spark.executor.heartbeatInterval=45s
And this fails...