RabbitMQ clustering

What is a RabbitMQ cluster

A RabbitMQ cluster is a logical grouping of one or more nodes, each sharing users, virtual hosts, queues, exchanges, bindings, runtime parameters and other distributed states.

A cluster is made of N nodes and a proxy to enable seamless addressing in the stack.

How to cluster

RabbitMQ

Service creation

The first step in using a RabbitMQ cluster is creating one service per new RabbitMQ node joining the cluster. For example, if you want to update the default configuration containing only one rabbit service to a cluster containing three nodes, start by creating two new RabbitMQ service instances.

${{service-name}}:
    image: ${{image-repository}}osp-rabbitmq${{image-version}}
    ${{ports}}:
        - ${{remote-port:"5671:5671/tcp"}}
        - ${{debug-port:15672}}
    networks:
        back:
        aliases:
            - rabbit
    hostname: ${{service-name}}
    volumes:
        - "rabbitmq_data:/var/lib/rabbitmq/mnesia"
    secrets:
        - ${{auto-generated-secret-access}}

Every node of a cluster must share the same RABBITMQ_ERLANG_COOKIE to be able to communicate.

Note

Do no forget to change (and create in the stack configuration) the volume for each node.

Once the new services are added to the configuration you can push the new configuration to be able to create the cluster.

Cluster creation

Connect the newly added services to the existing RabbitMQ node by running the following commands on every (new) node:

root@modules_rabbitmq_rabbitmq-2:/# rabbitmqctl stop_app
Stopping rabbit application on node rabbit@modules_rabbitmq_rabbitmq-2 ...

root@modules_rabbitmq_rabbitmq-2:/# rabbitmqctl reset
Resetting node rabbit@modules_rabbitmq_rabbitmq-2 ...

root@modules_rabbitmq_rabbitmq-2:/# rabbitmqctl join_cluster rabbit@modules_rabbitmq_rabbitmq-1
Clustering node rabbit@modules_rabbitmq_rabbitmq-2 with rabbit@modules_rabbitmq_rabbitmq-1

root@modules_rabbitmq_rabbitmq-2:/# rabbitmqctl start_app
Starting node rabbit@modules_rabbitmq_rabbitmq-2 ...

Check the cluster is correctly formed by running the following command on one of the nodes.

root@modules_rabbitmq_rabbitmq-3:/# rabbitmqctl cluster_status
Cluster status of node rabbit@modules_rabbitmq_rabbitmq-3 ...
Basics

Cluster name: rabbit@modules_rabbitmq_rabbitmq-1

Disk Nodes

rabbit@modules_rabbitmq_rabbitmq-1
rabbit@modules_rabbitmq_rabbitmq-2
rabbit@modules_rabbitmq_rabbitmq-3

Running Nodes

rabbit@modules_rabbitmq_rabbitmq-1
rabbit@modules_rabbitmq_rabbitmq-2
rabbit@modules_rabbitmq_rabbitmq-3
...

Note

After this step the cluster is created but the behavior will not change compared to before. Indeed, other modules will continue communicating only with one of the cluster node.

Envoy (proxy)

The RabbitMQ cluster works properly without a proxy. However, this means we must choose which node we use, therefore loosing the reliability provided by a cluster. Using a proxy allows us to use a single entry point - the proxy - that will route requests to an available RabbitMQ node.

Update the RabbitMQ configuration

The default configuration uses an alias to be able to speak with the RabbitMQ service using only the ‘rabbit’ hostname. However, we now want to speak with the proxy - and therefore any of the cluster node - instead of only the first RabbitMQ service. In order to do so we must remove the alias configuration from the module.service file.

In the same way, the RabbitMQ service exposes its connection port when a remote connector is configured. In case of a cluster the port should be exposed by the proxy so that the remote modules can communicate with any of the cluster node.

Updated module.service file for RabbitMQ instances :

${{service-name}}:
    image: ${{image-repository}}osp-rabbitmq${{image-version}}
    networks:
        - "back"
    environment:
        - RABBITMQ_ERLANG_COOKIE=rabbit-cluster-cookie
    hostname: ${{service-name}}
    volumes:
        - "rabbitmq_data-1:/var/lib/rabbitmq/mnesia"
    secrets:
        - ${{auto-generated-secret-access}}

Note

Note that module.service file should most likely be the same for every service of the cluster.

After modifying the module.service file of the first RabbitMQ service, the rabbitmq.conf configuration file of every cluster node should be updated to include its affiliation to the same cluster. Add the following snippet in the rabbitmq.conf configuration files.

This snippet will be removed from the final rabbit configuration file but will be used while generating the Envoy configuration to identify which RabbitMQ service belongs to which cluster.

Creating the Envoy service

We can create a proxy by adding the following service.

${{service-name}}:
    image: ${{image-repository}}osp-envoy${{image-version}}
    networks:
        back:
        aliases:
            - rabbit
    ${{ports}}:
        - ${{remote-port:"5671:5671/tcp"}}
    secrets:
        - ${{auto-generated-secret-access}}

Note

The Envoy service contains the configuration - network alias and remote port - that we removed from the RabbitMQ service in the previous step. Therefore, every module - local or remote - communicating with only one RabbitMQ service will now seamlessly communicate with the proxy.

The Envoy configuration must contain information about the cluster nodes to be able to route requests to available nodes. The configuration is done inside the envoy.yaml file

static_resources:
    # Define envoy TCP proxy listener.
    listeners:
        - address:
            socket_address:
                address: 0.0.0.0
                port_value: 5671
          filter_chains:
            - filters:
                - name: envoy.filters.tcp_proxy
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                    stat_prefix: ingress_rabbit
                    cluster: rabbit
        - address:
            socket_address:
                address: 0.0.0.0
                port_value: 15672
          filter_chains:
            - filters:
                - name: envoy.filters.tcp_proxy
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                    stat_prefix: ingress_rabbit
                    cluster: rabbit-admin

    # Cluster configuration.
    clusters:
        - name: rabbit
          connect_timeout: 1s
          type: strict_dns

          # Health check to ensure RabbitMQ nodes are available.
          health_checks:
              - timeout: 0.25s
                interval: 60s
                interval_jitter: 1s
                unhealthy_threshold: 1
                healthy_threshold: 1
                reuse_connection: false
                tcp_health_check:
                  # Empty data = only test if it is possible to establish connection.
                  send:
                  receive:

          # Load balancing of requests between the nodes.
          lb_policy: round_robin
          load_assignment:
              cluster_name: rabbit

The cluster configuration is done in the envoy.conf file.

Note

The ${{cluster-endpoints:CLUSTER-NAME|PORT}} tag should use the same CLUSTER-NAME used in the rabbitmq.conf configuration files.