Cluster computing with a raspberry pi bramble

My goal here was to simply create a cluster of raspberry pi 3s (colloquially referred to as a bramble).

Why? It seems like a pretty cheap way to build a test bed for deploying to distributed systems (such as AWS). Overall this setup cost around $240 but could be made much cheaper by using Raspberry Pi Zeros and using USB OTG.

Pretty straight forward equipment list:

• 4x Raspberry Pi 3
• 5 Port Switch
• 4x 16 GB Micro SD Cards
• 4x USB cables
• 4x Patch Cables
• And one neat stackable case to house them all

The result:

OS

Initially I went with HypriotOS 1.0.1. Partially because of the awesome work they've done into getting Docker (specifically 1.12) to run well on this Raspbian derivitive. My understanding now is that Raspbian has good support for Docker 1.12 out of the box (with a kernel update).

First Steps To a Distributed Task Queue

My first goal was to create distributed compute task queue using Celery and Redis. All running under Docker swarm mode such that I could easily spin up containers of celery workers in the cloud. You can checkout my armv7 compatible docker image at https://hub.docker.com/r/deldreth/. The hypriot/redis image works well enough out of the box (for testing purposes). I will have another article at a later point detailing more information about the rpi-python-celery image.

The good news is that everything works well enough considering the clock speeds of the rpi3. Redis is a bit of memory hog (it tends to consume everything it can--immediately). I've yet to try building an image for a celery task broker using rabbitmq.

I'll have further updates on this project shortly. Right now I've been playing around with running Apache Spark on a single master:three slave setup with the cluster, but I want to continue working on the distributed task queue.

Further Comments

If this is something that really interests you then I strongly suggest checking out Docker Captain Alex Ellis' talk for Container Camp.