This post tells how I attached a USB Thumb drive to my Raspberry PI and used it to hold Docker's Root Directory.

The first step is to connect to the RPI.

$ ssh -o 'StrictHostKeyChecking=no' -o 'CheckHostIP=no' 'pirate@pi02.local'

Now create a mount point. This is just a directory, nothing fancy. It should be owned by root because Docker runs as root. Don't try to use "pirate" as the owner. I tried that. It failed. Leave the owner as root.

$ sudo mkdir /media/usb

Then look at the attached USB devices.

$ sudo blkid
/dev/mmcblk0: PTTYPE="dos"
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="HypriotOS" UUID="D6D9-1D76" TYPE="vfat"
/dev/mmcblk0p2: LABEL="root" UUID="81e5bfc7-0701-4a09-80aa-fe5bc3eecbcf" TYPE="ext4"
/dev/sda1: LABEL="STORE N GO" UUID="F171-FAE6" TYPE="vfat" PARTUUID="f11d6f2b-01"

Note that the USB thumb drive is /dev/sda1. The information above is for the original formatting of the drive. After formatting the drive to use "ext3" the information looks like:

/dev/sda1: LABEL="PI02" UUID="801b666c-ea47-4f6f-ab6b-b88acceff08f" TYPE="ext3" PARTUUID="f11d6f2b-01"

This is the command that I used to format the drive to use ext3. Notiice that I named the drive the same as the hostname. I have no particular reason to do this. It just seemed right. Only run this formatting command once.

$ sudo mkfs.ext3 -L "PI02" /dev/sda1

Now it's time to mount the thumb drive. Here we connect the device (/dev/sda1) to the mount point. After this command is run you'll be able to use /media/usb as a normal directory.

$ sudo mount /dev/sda1 /media/usb

Next we setup the thumb drive to be available whenever the RPI is rebooted. First, find the UUID. It's whatever UUID is associated with sda1.

$ sudo ls -l /dev/disk/by-uuid
total 0
lrwxrwxrwx 1 root root 10 Jul  3  2014 801b666c-ea47-4f6f-ab6b-b88acceff08f -> ../../sda1
lrwxrwxrwx 1 root root 15 Jul  3  2014 81e5bfc7-0701-4a09-80aa-fe5bc3eecbcf -> ../../mmcblk0p2
lrwxrwxrwx 1 root root 15 Jul  3  2014 D6D9-1D76 -> ../../mmcblk0p1

Now add that UUID to the /etc/fstab file so it will be recognized across reboots. If you re-flash your SD card, you'll need to execute this step again.

$ echo "UUID=801b666c-ea47-4f6f-ab6b-b88acceff08f /media/usb nofail 0 0" | sudo tee -a /etc/fstab

Some images already on the Hypriot SD card. We'll make sure they are available after we move the Docker Root directory.

$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
hypriot/rpi-swarm    1.2.2               f13b7205f2db        5 weeks ago         13.97 MB
hypriot/rpi-consul   0.6.4               879ac05d5353        6 weeks ago         19.71 MB

Stop Docker to ensure that the Docker root directory does not change.

$ sudo systemctl stop docker

Copy files to the new location. Don't bother deleting the original files.

$ sudo cp --no-preserve=mode --recursive /var/lib/docker /media/usb/docker

If you are paranoid, you can compare the two directory trees.

$ sudo diff /var/lib/docker /media/usb/docker
Common subdirectories: /var/lib/docker/containers and /media/usb/docker/containers
Common subdirectories: /var/lib/docker/image and /media/usb/docker/image
Common subdirectories: /var/lib/docker/network and /media/usb/docker/network
Common subdirectories: /var/lib/docker/overlay and /media/usb/docker/overlay
Common subdirectories: /var/lib/docker/tmp and /media/usb/docker/tmp
Common subdirectories: /var/lib/docker/trust and /media/usb/docker/trust
Common subdirectories: /var/lib/docker/volumes and /media/usb/docker/volumes

Edit the docker service file to add --graph "/media/usb/docker" to the end of the ExecStart line.

$ sudo vi /etc/systemd/system/docker.service

Now reload the systemctl daemon and start docker.

sudo systemctl daemon-reload
sudo systemctl start docker

Confirm that the ExecStart is correct - that is has the graph parameter.

$ sudo systemctl show docker | grep ExecStart

Confirm that the Docker Root Directory has changed.

$ docker info | grep "Root Dir"

And finally, confirm that you can see docker images.

$ docker images

How Did I prepare my PicoCluster?

DOCKER VERSION:  1.11.1
HYPRIOT VERSION: 0.8
RASPBERRY PI:    3

From my Linux laptop, I created five SD cards using the flash utility from Hypriot.

As I plugged each SD card into my laptop, I ran 'lsblk'. Then I used 'umount' for anything mounted to the SD card. For example.

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 111.8G  0 disk 
├─sda1        8:1    0  79.9G  0 part /
├─sda2        8:2    0     1K  0 part 
└─sda5        8:5    0  31.9G  0 part [SWAP]
sdb           8:16   0 894.3G  0 disk 
└─sdb1        8:17   0 894.3G  0 part /data
sr0          11:0    1  1024M  0 rom  
mmcblk0     179:0    0    15G  0 disk 
├─mmcblk0p1 179:1    0    64M  0 part /media/medined/3ABE-55E4
└─mmcblk0p2 179:2    0  14.9G  0 part /media/medined/root

umount any mount points for mmcblk0 (or your SD card). For example,

umount /media/medined/3ABE-55E4
umount /media/medined/root

If the SD cards were flashed in the past then you'll need to run

umount /media/medined/HypriotOS
umount /media/medined/root

Here are the five flash commands that I used. Of course, I used my real SSID and PASSWORD. Note that this command leaves your password in your shell history. If this is a concern, please research alternatives.

As you flash the SD cards, use a gold sharpie to indicate the hostname of the SD card. This will make it much easier to make sure they are in the right RPI.

flash --hostname pi01 --ssid NETWORK --password PASSWORD --device /dev/mmcblk0 https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip
flash --hostname pi02 --ssid NETWORK --password PASSWORD --device /dev/mmcblk0 https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip
flash --hostname pi03 --ssid NETWORK --password PASSWORD --device /dev/mmcblk0 https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip
flash --hostname pi04 --ssid NETWORK --password PASSWORD --device /dev/mmcblk0 https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip
flash --hostname pi05 --ssid NETWORK --password PASSWORD --device /dev/mmcblk0 https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip

Next after the SD cards are plaeced into the PicoCluster, I plugged it into power.

As a sidenote, each time you restart the RPIs, their SSH fingerprint changes. You'll need to remove the old fingerprint. One technique is the following:

for i in `seq 1 5`; do ssh-keygen -R pi0${i}.local 2>/dev/null; done

I dislike questions about server fingerprint's when connecting. Therefore, you'll see me using the "StrictHostKeyChecking=no" option with SSH. I take no stance on the security ramifications of this choice. I'm connecting to my local PicoCluster not some public server. Make your own security decisions.

Ensure that you have a SSH key set. Look for "~/.ssh/id_rsa". If you don't have that file, use ssh-keygen to make one.

Now copy your PKI credential to the five PRI to enable password-less SSH. You be asked for the password, which should be "hypriot", five times.

for i in `seq 1 5`; do ssh-copy-id -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local; done

Next you can check that password-less SSH is working. After each SSH, you'll see a prompt like "HypriotOS/armv7: pirate@pi01 in ~". Just check the hostname is correct and then type exit to move onto the next RPI.

for i in `seq 1 5`; do ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local; done

You can use the following shell function to determine the IP address of an RPI. I also found it happy to log into my router to see the list of attached devices. By the way, if you haven't changed the default password for the admin user of your router, do it. This article will wait...

function getip() { (traceroute $1 2>&1 | head -n 1 | cut -d\( -f 2 | cut -d\) -f 1) }

It's probably a good idea to place that function in your .bashrc file so that you'll always have it handy.

for i in `seq 1 5`; do echo "PI0${i}.local: $(getip pi0${i}.local)"; done

Now comes the fun part, setting up the Docker Swarm. Fair warning. I don't know if these steps are correct.

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-master \
  --swarm-image hypriot/rpi-swarm:latest \
  --generic-ip-address=$(getip pi01.local) \
  --generic-ssh-user "pirate" \
  --swarm-discovery="token://01" \
  swarm

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --generic-ip-address=$(getip pi02.local) \
  --generic-ssh-user "pirate" \
  --swarm-discovery="token://01" \
  swarm-slave01

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --generic-ip-address=$(getip pi03.local) \
  --generic-ssh-user "pirate" \
  --swarm-discovery="token://01" \
  swarm-slave02

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --generic-ip-address=$(getip pi04.local) \
  --generic-ssh-user "pirate" \
  --swarm-discovery="token://01" \
  swarm-slave03

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --generic-ip-address=$(getip pi05.local) \
  --generic-ssh-user "pirate" \
  --swarm-discovery="token://01" \
  swarm-slave04

Now you can run list the nodes in the cluster using Docker Machine:

$ docker-machine ls
NAME            ACTIVE   DRIVER    STATE     URL                       SWARM            DOCKER    ERRORS
swarm           -        generic   Running   tcp://192.168.1.12:2376   swarm (master)   v1.11.1   
swarm-slave01   -        generic   Running   tcp://192.168.1.7:2376    swarm            v1.11.1   
swarm-slave02   -        generic   Running   tcp://192.168.1.11:2376   swarm            v1.11.1   
swarm-slave03   -        generic   Running   tcp://192.168.1.23:2376   swarm            v1.11.1   
swarm-slave04   -        generic   Running   tcp://192.168.1.22:2376   swarm            v1.11.1   

Notice that a master node is indicated but it is not marked as active. I don't know why.

Before moving on, let's look at what containers are being run. There should be six.

for i in `seq 1 5`; do echo "RPI ${i}"; ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local docker ps -a; done
RPI 1
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                              NAMES
ceb4a5255dc2        hypriot/rpi-swarm:latest   "/swarm join --advert"   About an hour ago   Up About an hour    2375/tcp                           swarm-agent
e9d3bf308284        hypriot/rpi-swarm:latest   "/swarm manage --tlsv"   About an hour ago   Up About an hour    2375/tcp, 0.0.0.0:3376->3376/tcp   swarm-agent-master
RPI 2
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
e2dca97c23fe        hypriot/rpi-swarm:latest   "/swarm join --advert"   About an hour ago   Up About an hour    2375/tcp            swarm-agent
RPI 3
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
07d0b4fc4490        hypriot/rpi-swarm:latest   "/swarm join --advert"   11 minutes ago      Up 11 minutes       2375/tcp            swarm-agent
RPI 4
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
88712d8df693        hypriot/rpi-swarm:latest   "/swarm join --advert"   6 minutes ago       Up 6 minutes        2375/tcp            swarm-agent
RPI 5
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
b7738fb8c4b8        hypriot/rpi-swarm:latest   "/swarm join --advert"   2 minutes ago       Up 2 minutes        2375/tcp            swarm-agent

Currently, when you type "docker ps" you're looking at containers running on your local computer. You can switch so that "docker" connects to one of the "docker machines" using this command:

eval $(docker-machine env swarm)

Now "docker ps" returns information about containers running on pi01.

$ docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                              NAMES
ceb4a5255dc2        hypriot/rpi-swarm:latest   "/swarm join --advert"   About an hour ago   Up About an hour    2375/tcp                           swarm-agent
e9d3bf308284        hypriot/rpi-swarm:latest   "/swarm manage --tlsv"   About an hour ago   Up About an hour    2375/tcp, 0.0.0.0:3376->3376/tcp   swarm-agent-master

One neat "trick" is to look at the information from the "swarm-agent-master" container. This is done using Docker's -H option. Notice that the results indicate there are six containers running. Count the number of containers found using the "for..loop" earlier. They are the same number.

$ docker -H $(docker-machine ip swarm):3376 info
Containers: 6
 Running: 6
 Paused: 0
 Stopped: 0
Images: 15
Server Version: swarm/1.2.3
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 5
 swarm: 192.168.1.12:2376
  └ ID: P4OH:AB7Q:T2T3:P6OK:BW5F:YSIB:NACW:Q2F3:FKU4:IJFD:AUJQ:74CZ
  └ Status: Healthy
  └ Containers: 2
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 971.7 MiB
  └ Labels: executiondriver=, kernelversion=4.4.10-hypriotos-v7+, operatingsystem=Raspbian GNU/Linux 8 (jessie), provider=generic, storagedriver=overlay
  └ UpdatedAt: 2016-06-22T01:39:56Z
  └ ServerVersion: 1.11.1
 swarm-slave01: 192.168.1.7:2376
  └ ID: GDQI:WYHS:OD2W:EE67:CKMU:A2PW:6K5T:YZSK:B5KL:SPCZ:6GVX:5MCO
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 971.7 MiB
  └ Labels: executiondriver=, kernelversion=4.4.10-hypriotos-v7+, operatingsystem=Raspbian GNU/Linux 8 (jessie), provider=generic, storagedriver=overlay
  └ UpdatedAt: 2016-06-22T01:39:45Z
  └ ServerVersion: 1.11.1
 swarm-slave02: 192.168.1.11:2376
  └ ID: CA7H:C7UA:5V5N:NY4C:KECT:JK57:HDGN:2DNH:ASXQ:UJFQ:A5A4:US3Y
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 971.7 MiB
  └ Labels: executiondriver=, kernelversion=4.4.10-hypriotos-v7+, operatingsystem=Raspbian GNU/Linux 8 (jessie), provider=generic, storagedriver=overlay
  └ UpdatedAt: 2016-06-22T01:39:32Z
  └ ServerVersion: 1.11.1
 swarm-slave03: 192.168.1.23:2376
  └ ID: 6H6D:P6EN:PTBL:Q5E3:MP32:T6CI:XU33:PCQV:KT6H:KRJ4:LYSN:76EJ
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 971.7 MiB
  └ Labels: executiondriver=, kernelversion=4.4.10-hypriotos-v7+, operatingsystem=Raspbian GNU/Linux 8 (jessie), provider=generic, storagedriver=overlay
  └ UpdatedAt: 2016-06-22T01:39:25Z
  └ ServerVersion: 1.11.1
 swarm-slave04: 192.168.1.22:2376
  └ ID: 2ZBK:3DJE:D23C:7QAB:TLFS:L7EO:L4L4:IQ6Y:EC7D:UG7S:3WU6:QJ5D
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 971.7 MiB
  └ Labels: executiondriver=, kernelversion=4.4.10-hypriotos-v7+, operatingsystem=Raspbian GNU/Linux 8 (jessie), provider=generic, storagedriver=overlay
  └ UpdatedAt: 2016-06-22T01:39:32Z
  └ ServerVersion: 1.11.1
Plugins: 
 Volume: 
 Network: 
Kernel Version: 4.4.10-hypriotos-v7+
Operating System: linux
Architecture: arm
CPUs: 20
Total Memory: 4.745 GiB
Name: e9d3bf308284
Docker Root Dir: 
Debug mode (client): false
Debug mode (server): false
WARNING: No kernel memory limit support

And that's as far as I've gotten.