Kevin Boone

Deploying the Mosquitto MQTT message broker on OpenShift (part 1)

OpenShift logo I originally wrote this article for Red Hat Developer. It is reprinted here with permission.

Mosquitto is a lightweight message broker that supports the Message Queuing Telemetry Transport (MQTT) protocol. Mosquitto is widely used in Internet of Things (IoT) and telemetry applications, where a fully-featured message broker like Red Hat AMQ would be unnecessarily burdensome. Mosquitto also finds a role as a message bus for interprocess communication in distributed systems. Because it avoids complex features, Mosquitto is easy to tune and handles substantial application workloads with relatively modest memory and CPU resources.

There are essentially two stages to making Mosquitto available on Red Hat OpenShift. First, you need to containerize the application in a way that is broadly compatible with OpenShift. Part of containerization involves installing the container image in a repository, from which OpenShift can download it. Second, you need to deploy the containerized image in a pod, providing whatever properties and configuration are necessary for the specific installation. The first half of this article shows how to build Mosquitto into an image suitable for use in a container. The second half will show you how to configure and deploy the Mosquitto image on OpenShift.

Mosquitto basics

Mosquitto's messaging model is publish-subscribe, like a "topic" in Java Message Service (JMS) terminology. In fact, the Mosquitto documentation uses the term "topic" for its message destinations. In the terminology of AMQ 7 (and ActiveMQ Artemis, on which AMQ 7 is based), Mosquitto destinations are non-durable anycast addresses by default. Mosquitto communicates only over MQTT: a lightweight, payload-neutral, non-transactional wire protocol.

Note: A detailed description of Mosquitto is beyond the scope of this article. For more information, please see the Mosquitto website.

It's relatively easy to deploy Mosquitto in an OpenShift pod: Mosquitto has (and needs) no clustering support, so it lends itself to simple deployment methods. While you might find uses for Mosquitto on OpenShift, this article focuses as much on software packaging in an OpenShift-compatible way as it does on Mosquitto itself. Many widely-used software packages were not designed for cloud or container operation; as a result, they present challenges to creating a maintainable, configurable installation.

Using Mosquitto with OpenShift

For this article, I make certain assumptions about how Mosquitto (or any other lightweight application) will be used on OpenShift:

In the instructions, I'm also assuming you have a local workstation installation of Mosquitto--or at least, the Mosquitto test clients--for testing. On Fedora and Red Hat Enterprise Linux (RHEL) systems, typing dnf install mosquitto in the shell should do the trick.

For the sake of brevity, I won't display all the relevant files in their entirety. You can obtain all of these files from my GitHub repository.

Note that I'll use Podman to build and run images in my demonstration. Docker and Buildah should also work without modifying the example.

Configuration files for containerization

Mosquitto needs at least four files, or sets of files, to provide a rudimentary service on OpenShift. You might need to configure these files at installation time:

A standard Mosquitto installation provides default locations for configuration files. In this example, however, I've chosen to put all the files that might need to be modified by the deployer into a single directory in the container image. I've chosen the home directory for the user account that will own the broker. I'm using the directory /myuser. The directory's name is arbitrary, but administration is simpler if all the files that an installer will need to change are in the same directory.

In the source repository, all the relevant configuration files and certificates are in the files directory, and will be copied to /myuser in the image when it is built.

We'll also need a Dockerfile to build the container image from a base image, the Mosquitto binaries, and the configuration files. It will be possible (perhaps necessary is a better word) to test the generated image on the local workstation before installing it on OpenShift.

Credentials file

Mosquitto provides a simple user-password authentication mechanism based on a credentials file. The credentials file is in a proprietary format, containing hashed passwords. Mosquitto provides a utility named mosquitto_passwd for editing the file.

The default image provides one user named admin with the password admin, in a credentials file called passwd. I created this file as follows: 

$ touch files/passwd
$ mosquitto_passwd -b files/passwd admin admin

Note: In Part 2, I will explain how to provide different credentials when deploying the image on OpenShift.

To create multiple users with different privileges, edit the main configuration file and add users to the credentials file.

TLS certificates

Mosquitto requires at least three certificates for a practical installation. Like all the other files that an installer may override, these will be installed into the image in the /myuser directory. These certificates are:

All these certificates must be in PEM format and are easily generated using the openssl utility.

The following example shows the commands that I used to generate the certificates in the default image. You might want to use similar commands to bake different certificates into the image; however, I envisage all these files being overridden in OpenShift at installation time, from a secret or configmap (more about this in Part 2). 

$ openssl req -new -x509 -days 3650 -extensions v3_ca -keyout files/ca.key \
     -out files/ca.crt -subj "/O=acme/CN=com"
$ openssl genrsa -out files/server.key 2048
$ openssl req -new -out files/server.csr -key files/server.key -subj "/O=acme2/CN=com"
$ openssl x509 -req -in files/server.csr -CA files/ca.crt -CAkey files/ca.key \
    -CAcreateserial -out files/server.crt -days 3650
$ openssl rsa -in files/server.key -out files/server.key
$ rm files/ca.key files/ca.srl files/server.csr
$ chmod 644 files/server.key

Main configuration file

The main configuration file specifies the locations of the credentials files and certificate files and creates TCP listeners on ports 1883 (plaintext) and 8883 (TLS). These port numbers are conventional for MQTT, and the Mosquitto utilities assume they are in use unless otherwise specified. Here is the mosquitto.confconfiguration file for this example. 

# Port for plaintext communication
port 1883

# Location of the credentials file
password_file /myuser/passwd

# Port and certificates for TLS encrypted communication
listener 8883
certfile /myuser/server.crt
cafile /myuser/ca.crt
keyfile /myuser/server.key

Note that all the referenced files are in /myuser. This directory will be copied and populated during the image build.

In practice, this configuration file will probably need to be overridden. If it is, the new version will probably also need to specify a credentials file and certificate files.

Startup script

Starting up is trivially simple in this example. The script startup.sh just runs the mosquitto command, specifying the main configuration file: 

#!/bin/sh
mosquitto -c /myuser/mosquitto.conf

Building, testing, and publishing the image

To create the smallest possible image, I'm using Alpine Linux as a base. Alpine is a minimal Linux specifically designed for containers. To further reduce its size and complexity, Alpine uses the MUSL C library rather than the Glibc that is almost ubiquitous in Linux. This decision mostly limits Alpine's use to applications that are available in the Alpine repository or can be built from source.

Note: At the time of writing, the latest Alpine release is 3.12. The corresponding Alpine repository doesn't have a binary package for Mosquitto, so to simplify the image build, I'm specifying Alpine 3.11. To use a later version of Alpine, if the package is not present in the repository, you'll need to build Mosquitto from source. Building Mosquitto is not difficult, but to build a version for Alpine, you must build against the MUSL C library. A straightforward way to do this is to install a desktop version of Alpine Linux, perhaps in a virtual machine, and do the build there. Another approach is to build Mosquitto in-image, using a multistage build process to eliminate all the build tools from the final image. There's a lot more information on this multistage build process in my article Developing micro-microservices in C on Red Hat OpenShift. Here, I'm happy to use the binary package from the Alpine repository.

Here is a simple Dockerfile that creates the image, using Alpine Linux as a base: 

FROM alpine:3.11
RUN addgroup -g 1000 mygroup && \
     adduser -G mygroup -u 1000 -h /myuser -D myuser && \
     chown -R myuser:mygroup /myuser && \
     apk --no-cache add mosquitto
WORKDIR /myuser
COPY files/* /myuser/
USER myuser
EXPOSE 1883 8883
CMD ["/myuser/start.sh"]

There's nothing remarkable about this Dockerfile. It creates a single user, exposes the ports that Mosquitto uses, and specifies our startup script as the entry point.

The total size of the final image is only about 7 MB.

Building and testing the image locally

Build the image using: 

$ podman build .

Get the ID of the new image (using, for example, podman image list) and then run it, specifying the port mappings. In the following command, replace the italicized word image with the ID of your image: 

$ podman run -it -port 1883:1883 -p 8883:8883 image

This command exposes the plaintext and TLS ports. We can test the image using the Mosquitto test clients mosquitto_pub (publish) and/or mosquitto_sub (subscribe). For example, to publish a message using the plaintext listener, enter: 

$ mosquitto_pub -t foo -m "Some text" -u admin -P admin

The -t foo option specifies the Mosquitto topic, while -m specifies the data to send. The user (-u option) and password (-P option) must match the values given in the mosquitto_passwd command when creating the credentials file earlier. Note that the hostname and port default to localhost and 1883, which are appropriate in this case.

To test the TLS listener, you'll need to specify the CA certificate. The certificate is in the source repository's files directory and in the image (I'll discuss how to obtain the certificate from the OpenShift pod in Part 2): 

$ mosquitto_sub -t foo --cafile files/ca.crt --insecure -u admin -P admin

Again, the defaults for host and port will be appropriate. You need the --insecure switch to override certificate hostname checks; this is because the server certificate in the image has the hostname acme.com, not localhost.

Note: Mosquitto is non-durable by default. So if you want to test that messages can be produced and consumed. you'll need to start the consumer before the producer.

Publishing the image

If the image works adequately on a workstation, you must publish it to a repository from which OpenShift can download it. The procedures for doing this are outside the scope of this article. I've published my image to quay.io using the following podman procedure: 

$ podman tag <image-id> mosquitto-ephemeral:0.1a
$ podman login quay.io...
$ podman push mosquitto-ephemeral:0.1a quay.io/kboone/mosquitto-ephemeral

This is a public repository, so feel free to use my image for testing if you don't want to build your own. I'm using the tag ephemeral in the image name to indicate that this implementation does not support durable messaging.

Of course, you could publish the image to another public image repository, or a private, institutional repository, or directly to the OpenShift internal repository if you're willing to expose it as a route. For detailed steps, see the article Using Podman to deploy an image directly to OpenShift 4 on my website.

However you publish the image, you'll need to know the repository URI for the OpenShift deployment. In this example using the quay.io repository, that URI is: 

quay.io/kboone/mosquitto-ephemeral:latest

Conclusion to Part 1

This article described some of the design considerations that are relevant to deploying a lightweight application on OpenShift. I don't claim that my approach is the only one you could use, or even particularly optimal. It is, however, straightforward enough for a demonstration, and might highlight a variety of issues that packagers and deployers should consider. Look for Part 2, where we will deploy the Mosquitto image we built in this article on OpenShift.