Using Quarkus to provide a REST interface to a message broker

Middleware message brokers usually use specific protocols for communication with their clients. AMQP and MQTT are common non-proprietary protocols, but most products have proprietary protocols as well. A problem with all these protocols is that they are not HTTP. Firewalls, proxies, and routers are generally well-adapted to HTTP, and usually to little else. So, while it's no problem to run a message broker on the same network as the applications that use it, problems related to network routing and access control arise when they are on distinct networks. With the rise of hybrid cloud installations, software components in the same application are on different networks as often as not.

The STOMP messaging protocol is HTTP-like, and can sometimes be handled like HTTP. Usually, though, what integrators are looking for is a way to interoperate with message brokers using real HTTP. Since it's now very popular, integrators usually ask for a REST-style interface, where the request URL indicates the action to be performed, and the HTTP request and response bodies carry the data for the request. This means that we need, as a minimum, a way to send an HTTP request that produces a message to a broker, and a request that consumes a message from a broker. That's the minimum functionality; a lot more could be provided -- and that's why there's a problem.

The classic version of Apache ActiveMQ (5.x) has a REST interface -- at least for demonstration purposes -- but the more recent 'Artemis' version does not. The Artemis maintainers do not seem keen to provide this functionality, and there's a good reason, which I'll describe shortly. In reality, developers or integrators who want to provide REST-style access to their message brokers really need to implement it themselves. In this article I will describe a simple way to do that, using an application for the Camel extension for Quarkus. I'll show that the basic functionality -- send and receive messages -- can be implemented in about twenty lines of Java, using this platform. Of course, a production-quality application might need more than this.

In this article I will describe my simple implementation, but first I'll explain why it's more complicated than it might first appear for the broker vendor to do this in a generic way.

What a REST interface to a broker might look like

In the simplest case, we might receive the next message available on a specified queue my_queue by issuing an HTTP GET request with a URL like this:

/broker/consume/my_queue

The interface might provide the message body just as a stream of raw data, or it might wrap it in some other format (JSON is popular). In the REST paradigm, GET requests are usual for retrieving data without changing anything. However, there's no reason why a POST request could not be used.

To send a message to the broker, we might issue a POST request of this form:

/broker/produce/my_queue

POST is appropriate here, both because we're changing the data on the target server, and because we will use the request body to carry the message that is to be stored on the broker. As with retrieving the message, we might just send the raw message body, or wrap it up in some other format.

We could use the HTTP Content-Type header to indicate the type of the data that is to be stored or retrieved; but most message brokers recognize a very limited number of types (typically text and binary).

Why not a generic REST interface?

Message brokers typically don't provide a generic HTTP interface, although a few do. One reason for this is that operations on message brokers are often more complex than HTTP interactions. HTTP is a very simple protocol, designed for situations where a single request is completely self-contained and stateless.

Messaging protocols, however, have notions of connection, session, and address, all with different lifetimes. Typically they have mechanisms for flow control and read-ahead which can be used to fine-tune data transfer speeds.

These protocol differences raise a number of questions. For example, should the HTTP interface maintain long-lived connections to the message broker, or create them according to demand from its own clients? What about sessions -- if these are distinct from connections, should the interface maintain a pool of session? And if so, how large should it be? How will the interface deal with the distinction between persistent and non-persistent messaging? Message batches? Setting custom headers? How should data flow be regulated? How should the system handle errors? If multiple HTTP clients connect to the same address on the same message broker, should the clients share the messages, or receive their own copies?

These questions are easy to answer in a specific application, but it is difficult to provide a general approach that will work with all applications. If we try to provide a general-purpose interface, most likely it will be so complex that developing clients will be very burdensome.

There's another, more subtle, reason why message brokers typically don't provide an HTTP interface of their own -- other than (sometimes) for administration. The HTTP interface will amount to a web server, which will create its own load on the broker when it is used by HTTP clients. This load will typically be of a different level, and often of a completely different kind, from the load created by clients using messaging protocols. Having these two kinds of load in the same process makes it difficult to scale the installation properly.

In this article I'll show how easy it is to provide an HTTP interface to a message broker using Quarkus. My simple application requires only about twenty lines of Java. If a real application needed ten times as much code, that would still be relatively straightforward. There's no magic here -- the Quarks and Camel libraries consists of millions of lines of code. This code just happens to be very appropriate for integration tasks.

Quarkus and Camel

The Quarkus framework is a Java development platform with many similarities to Spring Boot -- it is a highly opinionated framework, well-suited for doing the kinds of things that are typically done in the middleware world. An interesting feature is that it is based on libraries that can be compiled to native executable code, rather than Java bytecode. The use of native code leads to very rapid start-up times, which can be important in cloud environments.

Apache Camel is an integration toolkit for Java, that provides a language for managing communication between many different systems with many different protocols. Naturally, HTTP/HTTPS is supported, as is the AMQP messaging protocol. However, there are hundreds of other components. At the time of writing, not all are supported on Quarkus, but the ones we need for this application are.

The REST interface application

The full source of the application quarkus-broker-rest is available in my GitHub repository. Although there's not much to it, only about twenty lines are relevant to the specific application -- the rest are boilerplate. The boilerplate code -- like the Maven pom.xml that controls the build process -- is often generated using tools or archetypes.

The source bundle also includes a document explaining how to build the application, and test it using a message broker and the curl utility.

The application consists of a Camel route (integration flow), which starts like this:

rest("/broker/produce/{queue}")
  .post()
  .to ("direct:produce");

.rest("/broker/consume/{queue}")
  .get()
  .to ("direct:consume");

These calls set up two REST endpoints -- the first for sending messages, and the second for receiving. The token {queue} indicates that the client will supply a queue name here, as part of the URL, which will end up stored in Camel as a named parameter header.queue (more on this later). The calls post() and get() indicate that the endpoints respond to POST and GET requests. These endpoints forward requests to two other endpoints, called produce and consume. produce is defined like this:

from ("direct:produce")
  .toD ("amqp:queue:${header.queue}?disableReplyTo=true&jmsMessageType=Text")
  .setBody().constant ("OK\r\n");

The Camel amqp endpoint defines a connection to a message broker; the broker's hostname, port, etc., are defined in a configuration file. Note that ${header.name} substitutes the broker queue name, that was provided in the client's URL. We need disableReplyTo here because Camel will not wait for a response from the broker (apart from its basic acknowledgement). Once the client has sent its message, it's finished.

Note:
Camel note: We need toD here, rather than to because the amqp endpoint has a value that is substituted at runtime. Camel is more efficient if all the endpoint parameters are static, but that's not the case here

The consume endpoint is a bit more complicated:

from ("direct:consume")
      .pollEnrich()
      .simple ("amqp:queue:${header.queue}?disableReplyTo=true");

pollEnrich() indicates that, when this endpoint is invoked (when an HTTP request has been received) Camel will wait, whilst it issues another request. This further request is to consume from the message broker -- again using the ampq endpoint. The message retrieved from the broker is not processed in any way, so eventually it just finds its way back to the HTTP client, in the form of a response body.

To be fair, I should point out that I have not show the exception handling -- but this only adds five lines of code. So I have implemented the complete REST interface to the broker using only twenty lines of Java. Apart from the exception handling and the pom.xml build specification, what I've shown is the complete application.

Testing the application

Build the application using mvn package. This creates a self-contained (very large) JAR file containing the application and its dependencies in the target/ directory. To run the application:

  java -jar target/quarkus-broker-rest-1.0.0-runner.jar

By default, the application listens for HTTP clients on port 8080. It assumes that the message broker is accessible on the local host, on port 61616, and is not authenticated. These settings can be changed in application.properties. Assuming that the broker is running, we can produce a message to the address my_queue by issuing this HTTP request:

  curl -X POST --data-binary "This is a test" \
      localhost:8080/broker/produce/my_queue -H "content-type: text/plain"

The curl command is a little fiddly, because curl will by default send a request body as if it were an HTML form submission, and that's not at all what is required here. quarkus-broker-rest could be adapted to handle this kind of request if necessary, but it would require a bit more work.

To consume the message from the same address:

curl hostname:8080/broker/consume/my_queue
This is a test

Closing remarks

It's quite easy to implement a REST interface to a message broker using Camel and Quarkus. A real application will need encryption and security, but these can be added to Quarkus without any code changes -- see my trivial authentication example to see how this might be done, in a completely declarative (configuration-based) way.

What I haven't touched on here is any notion of tuning. The Camel components used in this example are highly tunable. For example, you can configure the pool of connections that is used for HTTP clients, and separately the pool used for connections to the message broker. You can also control more subtle aspects, such as the way that JMS sessions are cached between requests. All of this is reasonably well documented for Quarkus.