At the end of this article, I have a working Docker Swarm running on a five-node PicoCluster. Please flash your SD cards according to http://affy.blogspot.com/2016/06/how-did-i-prepare-my-picocluster-for.html. Stop following that article after copying the SSH ids to the RPI.

I am controlling the PicoCluster using my laptop. Therefore, my laptop is the HOST in the steps below.

There is no guarantee this commands are correct. They just seem to work for me. And please don't ever, ever depend on this information for anything non-prototype without doing your own research.

* On the HOST, create the Docker Machine to hold the consul service.

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --generic-ip-address=$(getip pi01.local) \
  --generic-ssh-user "pirate" \
  consul-machine

* Connect to the consul-machine Docker Machine

eval $(docker-machine env consul-machine)

* Start Consul.

docker run \
  -d \
  -p 8500:8500 \
  hypriot/rpi-consul \
  agent -dev -client 0.0.0.0

* Reset docker environment to talk with host docker.

unset DOCKER_TLS_VERIFY DOCKER_HOST DOCKER_CERT_PATH DOCKER_MACHINE_NAME

* Visit the consul dashboard to provide it is working and accessible.

firefox http://$(getip pi01.local):8500

* Create the swarm-master machine. Note that eth0 is being used instead of eth1.

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-master \
  --swarm-image hypriot/rpi-swarm:latest \
  --swarm-discovery="consul://$(docker-machine ip consul-machine):8500" \
  --generic-ip-address=$(getip pi02.local) \
  --generic-ssh-user "pirate" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" \
  --engine-opt="cluster-advertise=eth0:2376" \
  swarm-master

* Create the first slave node.

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --swarm-discovery="consul://$(docker-machine ip consul-machine):8500" \
  --generic-ip-address=$(getip pi03.local) \
  --generic-ssh-user "pirate" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" \
  --engine-opt="cluster-advertise=eth0:2376" \
  swarm-slave01

* List nodes in the swarm. I don't know why, but this command must be run from one of the RPI. Otherwise, I see a "malformed HTTP response" message.

eval $(docker-machine env swarm-master)

docker -H $(docker-machine ip swarm-master):3376 run \
  --rm \
  hypriot/rpi-swarm:latest \
  list consul://$(docker-machine ip consul-machine):8500

* Create the second slave node.

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --swarm-discovery="consul://$(docker-machine ip consul-machine):8500" \
  --generic-ip-address=$(getip pi04.local) \
  --generic-ssh-user "pirate" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" \
  --engine-opt="cluster-advertise=eth0:2376" \
  swarm-slave02

* Create the first third node.

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --swarm-discovery="consul://$(docker-machine ip consul-machine):8500" \
  --generic-ip-address=$(getip pi05.local) \
  --generic-ssh-user "pirate" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" \
  --engine-opt="cluster-advertise=eth0:2376" \
  swarm-slave03

* Check that docker machine sees all of the nodes

$ docker-machine ls
NAME             ACTIVE   DRIVER    STATE     URL                      SWARM                   DOCKER    ERRORS
consul-machine   -        generic   Running   tcp://192.168.1.8:2376                           v1.11.1   
swarm-master     -        generic   Running   tcp://192.168.1.7:2376   swarm-master (master)   v1.11.1   
swarm-slave01    -        generic   Running   tcp://192.168.1.2:2376   swarm-master            v1.11.1   
swarm-slave02    -        generic   Running   tcp://192.168.1.5:2376   swarm-master            v1.11.1   
swarm-slave03    -        generic   Running   tcp://192.168.1.4:2376   swarm-master            v1.11.1   

* List the swarm nodes in Firefox using Consul.

firefox http://$(docker-machine ip consul-machine):8500/ui/#/dc1/kv/docker/swarm/nodes/

* Is my cluster working? First, switch to the swarm-master environment. Then view it's information. You should see the slaves listed. Next run the hello-world container. And finally, list the containers.

eval $(docker-machine env swarm-master)
docker -H $(docker-machine ip swarm-master):3376 info
docker -H $(docker-machine ip swarm-master):3376 run hypriot/armhf-hello-world
docker -H $(docker-machine ip swarm-master):3376 ps -a

This post tells how I attached a USB Thumb drive to my Raspberry PI and used it to hold Docker's Root Directory.

The first step is to connect to the RPI.

$ ssh -o 'StrictHostKeyChecking=no' -o 'CheckHostIP=no' 'pirate@pi02.local'

Now create a mount point. This is just a directory, nothing fancy. It should be owned by root because Docker runs as root. Don't try to use "pirate" as the owner. I tried that. It failed. Leave the owner as root.

$ sudo mkdir /media/usb

Then look at the attached USB devices.

$ sudo blkid
/dev/mmcblk0: PTTYPE="dos"
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="HypriotOS" UUID="D6D9-1D76" TYPE="vfat"
/dev/mmcblk0p2: LABEL="root" UUID="81e5bfc7-0701-4a09-80aa-fe5bc3eecbcf" TYPE="ext4"
/dev/sda1: LABEL="STORE N GO" UUID="F171-FAE6" TYPE="vfat" PARTUUID="f11d6f2b-01"

Note that the USB thumb drive is /dev/sda1. The information above is for the original formatting of the drive. After formatting the drive to use "ext3" the information looks like:

/dev/sda1: LABEL="PI02" UUID="801b666c-ea47-4f6f-ab6b-b88acceff08f" TYPE="ext3" PARTUUID="f11d6f2b-01"

This is the command that I used to format the drive to use ext3. Notiice that I named the drive the same as the hostname. I have no particular reason to do this. It just seemed right. Only run this formatting command once.

$ sudo mkfs.ext3 -L "PI02" /dev/sda1

Now it's time to mount the thumb drive. Here we connect the device (/dev/sda1) to the mount point. After this command is run you'll be able to use /media/usb as a normal directory.

$ sudo mount /dev/sda1 /media/usb

Next we setup the thumb drive to be available whenever the RPI is rebooted. First, find the UUID. It's whatever UUID is associated with sda1.

$ sudo ls -l /dev/disk/by-uuid
total 0
lrwxrwxrwx 1 root root 10 Jul  3  2014 801b666c-ea47-4f6f-ab6b-b88acceff08f -> ../../sda1
lrwxrwxrwx 1 root root 15 Jul  3  2014 81e5bfc7-0701-4a09-80aa-fe5bc3eecbcf -> ../../mmcblk0p2
lrwxrwxrwx 1 root root 15 Jul  3  2014 D6D9-1D76 -> ../../mmcblk0p1

Now add that UUID to the /etc/fstab file so it will be recognized across reboots. If you re-flash your SD card, you'll need to execute this step again.

$ echo "UUID=801b666c-ea47-4f6f-ab6b-b88acceff08f /media/usb nofail 0 0" | sudo tee -a /etc/fstab

Some images already on the Hypriot SD card. We'll make sure they are available after we move the Docker Root directory.

$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
hypriot/rpi-swarm    1.2.2               f13b7205f2db        5 weeks ago         13.97 MB
hypriot/rpi-consul   0.6.4               879ac05d5353        6 weeks ago         19.71 MB

Stop Docker to ensure that the Docker root directory does not change.

$ sudo systemctl stop docker

Copy files to the new location. Don't bother deleting the original files.

$ sudo cp --no-preserve=mode --recursive /var/lib/docker /media/usb/docker

If you are paranoid, you can compare the two directory trees.

$ sudo diff /var/lib/docker /media/usb/docker
Common subdirectories: /var/lib/docker/containers and /media/usb/docker/containers
Common subdirectories: /var/lib/docker/image and /media/usb/docker/image
Common subdirectories: /var/lib/docker/network and /media/usb/docker/network
Common subdirectories: /var/lib/docker/overlay and /media/usb/docker/overlay
Common subdirectories: /var/lib/docker/tmp and /media/usb/docker/tmp
Common subdirectories: /var/lib/docker/trust and /media/usb/docker/trust
Common subdirectories: /var/lib/docker/volumes and /media/usb/docker/volumes

Edit the docker service file to add --graph "/media/usb/docker" to the end of the ExecStart line.

$ sudo vi /etc/systemd/system/docker.service

Now reload the systemctl daemon and start docker.

sudo systemctl daemon-reload
sudo systemctl start docker

Confirm that the ExecStart is correct - that is has the graph parameter.

$ sudo systemctl show docker | grep ExecStart

Confirm that the Docker Root Directory has changed.

$ docker info | grep "Root Dir"

And finally, confirm that you can see docker images.

$ docker images