How Did I prepare my PicoCluster?

DOCKER VERSION:  1.11.1
HYPRIOT VERSION: 0.8
RASPBERRY PI:    3

From my Linux laptop, I created five SD cards using the flash utility from Hypriot.

As I plugged each SD card into my laptop, I ran 'lsblk'. Then I used 'umount' for anything mounted to the SD card. For example.

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 111.8G  0 disk 
├─sda1        8:1    0  79.9G  0 part /
├─sda2        8:2    0     1K  0 part 
└─sda5        8:5    0  31.9G  0 part [SWAP]
sdb           8:16   0 894.3G  0 disk 
└─sdb1        8:17   0 894.3G  0 part /data
sr0          11:0    1  1024M  0 rom  
mmcblk0     179:0    0    15G  0 disk 
├─mmcblk0p1 179:1    0    64M  0 part /media/medined/3ABE-55E4
└─mmcblk0p2 179:2    0  14.9G  0 part /media/medined/root

umount any mount points for mmcblk0 (or your SD card). For example,

umount /media/medined/3ABE-55E4
umount /media/medined/root

If the SD cards were flashed in the past then you'll need to run

umount /media/medined/HypriotOS
umount /media/medined/root

Here are the five flash commands that I used. Of course, I used my real SSID and PASSWORD. Note that this command leaves your password in your shell history. If this is a concern, please research alternatives.

As you flash the SD cards, use a gold sharpie to indicate the hostname of the SD card. This will make it much easier to make sure they are in the right RPI.

flash --hostname pi01 --ssid NETWORK --password PASSWORD --device /dev/mmcblk0 https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip
flash --hostname pi02 --ssid NETWORK --password PASSWORD --device /dev/mmcblk0 https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip
flash --hostname pi03 --ssid NETWORK --password PASSWORD --device /dev/mmcblk0 https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip
flash --hostname pi04 --ssid NETWORK --password PASSWORD --device /dev/mmcblk0 https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip
flash --hostname pi05 --ssid NETWORK --password PASSWORD --device /dev/mmcblk0 https://downloads.hypriot.com/hypriotos-rpi-v0.8.0.img.zip

Next after the SD cards are plaeced into the PicoCluster, I plugged it into power.

As a sidenote, each time you restart the RPIs, their SSH fingerprint changes. You'll need to remove the old fingerprint. One technique is the following:

for i in `seq 1 5`; do ssh-keygen -R pi0${i}.local 2>/dev/null; done

I dislike questions about server fingerprint's when connecting. Therefore, you'll see me using the "StrictHostKeyChecking=no" option with SSH. I take no stance on the security ramifications of this choice. I'm connecting to my local PicoCluster not some public server. Make your own security decisions.

Ensure that you have a SSH key set. Look for "~/.ssh/id_rsa". If you don't have that file, use ssh-keygen to make one.

Now copy your PKI credential to the five PRI to enable password-less SSH. You be asked for the password, which should be "hypriot", five times.

for i in `seq 1 5`; do ssh-copy-id -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local; done

Next you can check that password-less SSH is working. After each SSH, you'll see a prompt like "HypriotOS/armv7: pirate@pi01 in ~". Just check the hostname is correct and then type exit to move onto the next RPI.

for i in `seq 1 5`; do ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local; done

You can use the following shell function to determine the IP address of an RPI. I also found it happy to log into my router to see the list of attached devices. By the way, if you haven't changed the default password for the admin user of your router, do it. This article will wait...

function getip() { (traceroute $1 2>&1 | head -n 1 | cut -d\( -f 2 | cut -d\) -f 1) }

It's probably a good idea to place that function in your .bashrc file so that you'll always have it handy.

for i in `seq 1 5`; do echo "PI0${i}.local: $(getip pi0${i}.local)"; done

Now comes the fun part, setting up the Docker Swarm. Fair warning. I don't know if these steps are correct.

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-master \
  --swarm-image hypriot/rpi-swarm:latest \
  --generic-ip-address=$(getip pi01.local) \
  --generic-ssh-user "pirate" \
  --swarm-discovery="token://01" \
  swarm

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --generic-ip-address=$(getip pi02.local) \
  --generic-ssh-user "pirate" \
  --swarm-discovery="token://01" \
  swarm-slave01

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --generic-ip-address=$(getip pi03.local) \
  --generic-ssh-user "pirate" \
  --swarm-discovery="token://01" \
  swarm-slave02

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --generic-ip-address=$(getip pi04.local) \
  --generic-ssh-user "pirate" \
  --swarm-discovery="token://01" \
  swarm-slave03

docker-machine create \
  -d generic \
  --engine-storage-driver=overlay \
  --swarm \
  --swarm-image hypriot/rpi-swarm:latest \
  --generic-ip-address=$(getip pi05.local) \
  --generic-ssh-user "pirate" \
  --swarm-discovery="token://01" \
  swarm-slave04

Now you can run list the nodes in the cluster using Docker Machine:

$ docker-machine ls
NAME            ACTIVE   DRIVER    STATE     URL                       SWARM            DOCKER    ERRORS
swarm           -        generic   Running   tcp://192.168.1.12:2376   swarm (master)   v1.11.1   
swarm-slave01   -        generic   Running   tcp://192.168.1.7:2376    swarm            v1.11.1   
swarm-slave02   -        generic   Running   tcp://192.168.1.11:2376   swarm            v1.11.1   
swarm-slave03   -        generic   Running   tcp://192.168.1.23:2376   swarm            v1.11.1   
swarm-slave04   -        generic   Running   tcp://192.168.1.22:2376   swarm            v1.11.1   

Notice that a master node is indicated but it is not marked as active. I don't know why.

Before moving on, let's look at what containers are being run. There should be six.

for i in `seq 1 5`; do echo "RPI ${i}"; ssh -oStrictHostKeyChecking=no -oCheckHostIP=no pirate@pi0${i}.local docker ps -a; done
RPI 1
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                              NAMES
ceb4a5255dc2        hypriot/rpi-swarm:latest   "/swarm join --advert"   About an hour ago   Up About an hour    2375/tcp                           swarm-agent
e9d3bf308284        hypriot/rpi-swarm:latest   "/swarm manage --tlsv"   About an hour ago   Up About an hour    2375/tcp, 0.0.0.0:3376->3376/tcp   swarm-agent-master
RPI 2
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
e2dca97c23fe        hypriot/rpi-swarm:latest   "/swarm join --advert"   About an hour ago   Up About an hour    2375/tcp            swarm-agent
RPI 3
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
07d0b4fc4490        hypriot/rpi-swarm:latest   "/swarm join --advert"   11 minutes ago      Up 11 minutes       2375/tcp            swarm-agent
RPI 4
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
88712d8df693        hypriot/rpi-swarm:latest   "/swarm join --advert"   6 minutes ago       Up 6 minutes        2375/tcp            swarm-agent
RPI 5
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
b7738fb8c4b8        hypriot/rpi-swarm:latest   "/swarm join --advert"   2 minutes ago       Up 2 minutes        2375/tcp            swarm-agent

Currently, when you type "docker ps" you're looking at containers running on your local computer. You can switch so that "docker" connects to one of the "docker machines" using this command:

eval $(docker-machine env swarm)

Now "docker ps" returns information about containers running on pi01.

$ docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                              NAMES
ceb4a5255dc2        hypriot/rpi-swarm:latest   "/swarm join --advert"   About an hour ago   Up About an hour    2375/tcp                           swarm-agent
e9d3bf308284        hypriot/rpi-swarm:latest   "/swarm manage --tlsv"   About an hour ago   Up About an hour    2375/tcp, 0.0.0.0:3376->3376/tcp   swarm-agent-master

One neat "trick" is to look at the information from the "swarm-agent-master" container. This is done using Docker's -H option. Notice that the results indicate there are six containers running. Count the number of containers found using the "for..loop" earlier. They are the same number.

$ docker -H $(docker-machine ip swarm):3376 info
Containers: 6
 Running: 6
 Paused: 0
 Stopped: 0
Images: 15
Server Version: swarm/1.2.3
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 5
 swarm: 192.168.1.12:2376
  └ ID: P4OH:AB7Q:T2T3:P6OK:BW5F:YSIB:NACW:Q2F3:FKU4:IJFD:AUJQ:74CZ
  └ Status: Healthy
  └ Containers: 2
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 971.7 MiB
  └ Labels: executiondriver=, kernelversion=4.4.10-hypriotos-v7+, operatingsystem=Raspbian GNU/Linux 8 (jessie), provider=generic, storagedriver=overlay
  └ UpdatedAt: 2016-06-22T01:39:56Z
  └ ServerVersion: 1.11.1
 swarm-slave01: 192.168.1.7:2376
  └ ID: GDQI:WYHS:OD2W:EE67:CKMU:A2PW:6K5T:YZSK:B5KL:SPCZ:6GVX:5MCO
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 971.7 MiB
  └ Labels: executiondriver=, kernelversion=4.4.10-hypriotos-v7+, operatingsystem=Raspbian GNU/Linux 8 (jessie), provider=generic, storagedriver=overlay
  └ UpdatedAt: 2016-06-22T01:39:45Z
  └ ServerVersion: 1.11.1
 swarm-slave02: 192.168.1.11:2376
  └ ID: CA7H:C7UA:5V5N:NY4C:KECT:JK57:HDGN:2DNH:ASXQ:UJFQ:A5A4:US3Y
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 971.7 MiB
  └ Labels: executiondriver=, kernelversion=4.4.10-hypriotos-v7+, operatingsystem=Raspbian GNU/Linux 8 (jessie), provider=generic, storagedriver=overlay
  └ UpdatedAt: 2016-06-22T01:39:32Z
  └ ServerVersion: 1.11.1
 swarm-slave03: 192.168.1.23:2376
  └ ID: 6H6D:P6EN:PTBL:Q5E3:MP32:T6CI:XU33:PCQV:KT6H:KRJ4:LYSN:76EJ
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 971.7 MiB
  └ Labels: executiondriver=, kernelversion=4.4.10-hypriotos-v7+, operatingsystem=Raspbian GNU/Linux 8 (jessie), provider=generic, storagedriver=overlay
  └ UpdatedAt: 2016-06-22T01:39:25Z
  └ ServerVersion: 1.11.1
 swarm-slave04: 192.168.1.22:2376
  └ ID: 2ZBK:3DJE:D23C:7QAB:TLFS:L7EO:L4L4:IQ6Y:EC7D:UG7S:3WU6:QJ5D
  └ Status: Healthy
  └ Containers: 1
  └ Reserved CPUs: 0 / 4
  └ Reserved Memory: 0 B / 971.7 MiB
  └ Labels: executiondriver=, kernelversion=4.4.10-hypriotos-v7+, operatingsystem=Raspbian GNU/Linux 8 (jessie), provider=generic, storagedriver=overlay
  └ UpdatedAt: 2016-06-22T01:39:32Z
  └ ServerVersion: 1.11.1
Plugins: 
 Volume: 
 Network: 
Kernel Version: 4.4.10-hypriotos-v7+
Operating System: linux
Architecture: arm
CPUs: 20
Total Memory: 4.745 GiB
Name: e9d3bf308284
Docker Root Dir: 
Debug mode (client): false
Debug mode (server): false
WARNING: No kernel memory limit support

And that's as far as I've gotten.

It took me a bit of time to get this simple program working so I'm sharing for other people new to Go.

package main

import (
    "fmt"
    "io"
    "net"
)

func reader(r io.Reader) {
    buf := make([]byte, 1024)
    for {
        n, err := r.Read(buf[:])
        if err != nil {
            return
        }
        println(string(buf[0:n]))
    }
}

func main() {
    c, err := net.Dial("unix", "/var/run/docker.sock")
    if err != nil {
        panic(err)
    }
    defer c.Close()

    fmt.Fprintf(c, "GET /images/json HTTP/1.0\r\n\r\n")

    reader(c)
}