Version: Next

Mosquitto PROXY protocol support

[TOC]

Introduction

Using Mosquitto as an entirely standalone broker works very well, but sometimes you may wish to place it behind a load balancer/proxy such as HAProxy. This provides certain advantages, such as carrying out TLS termination in the proxy to reduce load on the broker.

It does have one disadvantage which is that only the proxy IP address is found in the broker logs - the client IP addresses are not seen. Since Mosquitto 2.1, this can be fixed using the PROXY protocol v2 support, which can be enabled using the enable_proxy_protocol_v2 option. This is the recommended mode when using HAProxy.

Example configuration of a port:

listener 1883
enable_proxy_protocol_v2 true
# Further listener settings

In the following some different ways you can combine Mosquitto and HAProxy are described. It is not a complete guide to HAProxy.

Important: Enabling PROXY protocol support requires that the broker itself is not directly accessible on its network port. All communication must go through the broker. If a client is able to connect to the broker directly, it is trivial to spoof connection information, and this is especially important when using client certificates for mutual TLS on HAProxy. In that case, the contents of the PROXY header can directly indicate whether a client is allowed to connect so it must be protected.

It may be desirable to use a firewall to restrict access to the broker port.

General setup

All examples presented will be of the haproxy.cfg file, typically located at /etc/haproxy/haproxy.cfg on a native Linux installation.

The first part of the config file contains the global and defaults sections which are going to be common to all of the examples and not repeated.

Global section

This is a fairly standard global section, presented without comment.

global
        log /dev/log    local0
        log /dev/log    local1 notice
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

Defaults section

defaults
        mode    tcp
        timeout connect 5000
        timeout client  60000
        timeout server  60000

mode tcp - set TCP mode rather than HTTP mode by default. MQTT is not HTTP.
timeout connect 5000 - the timeout allowed for a client to connect, in milliseconds.
timeout client 60000 - if a client does not communicate in this interval, the connection will be closed by HAProxy.
timeout server 60000 - if the broker does not communicate in this interval, the connection will be closed by HAProxy.

The client and server timeout intervals should be chosen based on the intervals you expect to have communication occurring in your clients. If your communication is typically very sparse, the timeout should be chosen based on your keepalive interval. Otherwise, the quiet clients will be disconnected before they have a chance to send a keepalive request.

Direct pass-through, without PROXY protocol

The most basic approach is to pass connections directly through HAProxy to Mosquitto without being modified. This means that if an unencrypted MQTT connection is made, it will pass through as an unencrypted connection, and if an encrypted MQTT connection is made, it will pass through as an encrypted MQTT connection. Likewise, websockets connections are passed through unaffected.

Create a frontend, which is where HAProxy will listen for connections and a backend, which is where the broker that HAProxy will connect to is defined.

frontend mqtt_frontend
         bind *:1883
         default_backend mqtt_backend

backend mqtt_backend
        server server1 127.0.0.1:1884 check on-marked-down shutdown-sessions

In this case, HAProxy is listening on all interfaces on port 1883 and will attempt to connect to the broker on address 127.0.0.1 on port 1884. In other words, HAProxy and Mosquitto are running on the same instance. This is not required.

The broker configuration could be with an encrypted or unencrypted listener:

Unencrypted:

listener 1884
# Further listener settings

Encrypted:

listener 1884
certfile <path/to/server.crt>
keyfile <path/to/server.key>
# Further listener settings

TLS termination with server certificate only, without PROXY protocol

Place your server certificate and private key in /etc/haproxy/certs/.

frontend mqtts_frontend
         bind 0.0.0.0:8883 ssl crt /etc/haproxy/certs/

backend mqtt_backend
        server server1 127.0.0.1:1883 check on-marked-down shutdown-sessions

The broker configuration should declare an unencrypted listener:

listener 1883
# Further listener settings

TLS termination with mutual TLS - client and server certificates, without PROXY protocol

In addition to the server certificate, you must also provide the CA certificate that will sign the client certificates, ask for verification of the client certificate and make the certificate required.

frontend mqtts_frontend
         bind *:8883 ssl crt /etc/haproxy/certs/ verify required ca-file /etc/haproxy/client-ca.crt
         default_backend mqtt_backend

backend mqtt_backend
        server server1 127.0.0.1:1883 check on-marked-down shutdown-sessions

The broker configuration should declare an unencrypted listener:

listener 1883
# Further listener settings

Direct pass through, with PROXY protocol

To enable PROXY protocol v2 on HAProxy, add the send-proxy-v2 option to the backend.

frontend mqtt_frontend
         bind *:1883
         default_backend mqtt_backend

backend mqtt_backend
        server server1 127.0.0.1:1884 check on-marked-down shutdown-sessions send-proxy-v2

The broker configuration should declare an unencrypted listener and enable PROXY protocol v2 support. It is not possible to have a direct pass-through with encrypted connections on the broker.

listener 1883
enable_proxy_protocol_v2 true
# Further listener settings

TLS termination with server certificate only, with PROXY protocol

For TLS connections, use send-proxy-v2-ssl instead of send-proxy-v2. This ensures that TLS information is added to the PROXY header.

frontend mqtts_frontend
         bind *:8883 ssl crt /etc/haproxy/certs/
         default_backend mqtt_backend

backend mqtt_backend
        server server1 127.0.0.1:1883 check on-marked-down shutdown-sessions send-proxy-v2-ssl

On the broker side, use proxy_protocol_v2_require_tls true to ensure that only connections that were made using TLS are accepted on the broker. No other TLS configuration is required.

listener 1883
enable_proxy_protocol_v2 true
proxy_protocol_v2_require_tls true

TLS termination with mutual TLS - client and server certificate, with PROXY protocol

For TLS connections, use send-proxy-v2-ssl instead of send-proxy-v2. This ensures that TLS information is added to the PROXY header.

frontend mqtts_frontend
         bind *:8883 ssl crt /etc/haproxy/certs/ verify required ca-file /etc/haproxy/client-ca.crt
         default_backend mqtt_backend

backend mqtt_backend
        server server1 127.0.0.1:1883 check on-marked-down shutdown-sessions send-proxy-v2-ssl

The broker configuration uses require_certificate true to indicate that the broker should check the PROXY protocol header for the valid certificate result. No other TLS configuration is required.

listener 1883
enable_proxy_protocol_v2 true
proxy_protocol_v2_require_tls true
require_certificate true

TLS termination with mutual TLS - client and server certificate, with username and PROXY protocol

The Mosquitto option use_identity_as_username true can be used with the PROXY protocol support. This requires that the send-proxy-v2-ssl-cn option is used on HAProxy.

It is not possible to use use_subject_as_username with the PROXY protocol.

frontend mqtts_frontend
         bind *:8883 ssl crt /etc/haproxy/certs/ verify required ca-file /etc/haproxy/client-ca.crt
         default_backend mqtt_backend

backend mqtt_backend
        server server1 127.0.0.1:1883 check on-marked-down shutdown-sessions send-proxy-v2-ssl-cn

The broker configuration uses require_certificate and use_identity_as_username. No other TLS configuration is required.

listener 1883
enable_proxy_protocol_v2 true
proxy_protocol_v2_require_tls true
require_certificate true
use_identity_as_username true

Premium

Pro Mosquitto provides a Prometheus exporter and a InfluxDB exporter which both expose the metrics available in the core broker and any metrics provided by other plugins.

Plugin configuration

Prometheus Metrics Exporter

To enable the Prometheus Metrics Exporter plugin it must be loaded into the broker with, by adding the following to your mosquitto.conf:

global_plugin /usr/lib/cedalo_metrics_prometheus.so

The plugin has some options which can be configured.

bind_address : by default, the exporter will listen on all available network interfaces. You can bind it to a specific address using this option. For example, plugin_opt_bind_address 192.0.2.1
port : by default, the exporter will listen on port 8000. You can use another port with this option. For example plugin_opt_port 8100
update_interval : metrics stored in the exporter are updated on a set interval, by default 15 seconds. It is important to match this to the scrape interval of your Prometheus setup. If you are using a 60 second scrape interval, then use plugin_opt_update_interval 60

A sample configuration could look like:

global_plugin /usr/lib/cedalo_metrics_prometheus.so
plugin_opt_update_interval 60
plugin_opt_bind_address 192.0.2.1
plugin_opt_port 8100

InfluxDB Metrics Exporter

To enable the InfluxDB Metrics Exporter plugin it must be loaded into the broker with, by adding the following to your mosquitto.conf:

global_plugin /usr/lib/cedalo_metrics_influxdb.so

Connection and authentication

The InfluxDB exporter can operate with InfluxDB servers 2.0 upwards and 1.x. These are configured differently to one another.

Influx 2.x

To use InfluxDB 2.0 or above, you must configure a host, organization, bucket, and authentication token:

plugin_opt_host https://example.hoster.cloud2.influxdata.com
plugin_opt_organisation my-organisation
plugin_opt_bucket my-bucket
plugin_opt_token <token>

If you are using InfluxDB cloud, these values should all be easily discoverable. The plugin_opt_token option can contain either your literal authentication token, or the special value env. If set to env, the token will be read from the INFLUXDB_TOKEN environment variable instead.

Influx 1.6/8

To use InfluxDB 1.8 or 1.6, you must configure at least a hostname and database:

global_plugin /usr/lib/cedalo_metrics_influxdb.so
plugin_opt_hostname influxdb.example.com
plugin_opt_database my-database
plugin_opt_port 8086
plugin_opt_use_tls true

Basic Authentication

Basic HTTP authentication is supported for both connection modes by specifing a username and password:

...
plugin_opt_username my-username
plugin_opt_password my-password

Other options

plugin_opt_hostlabel - use a specific label for this host when sending data. If not specified, the local hostname will be used. plugin_opt_measurement - the measurement string to use when sending data. If not specified, mosquitto will be used. plugin_opt_cafile - a path to an x509 encoded certificate that should be trusted for making encrypted connections to the server. plugin_opt_update_interval - the interval between metrics submissions, in seconds.

A sample configuration could look like:

global_plugin /usr/lib/cedalo_metrics_influxdb.so
plugin_opt_update_interval 60
plugin_opt_hostname influxdb.example.com
plugin_opt_port 8100
plugin_opt_hostlabel broker1
plugin_opt_database edge_data
plugin_opt_use_tls true
plugin_opt_username username
plugin_opt_password password

Optional configuration (applies to both exporters)

You may also consider turning off the $SYS updates from the broker:

sys_interval 0

Available metrics

label	metric type	description
`mosquitto_sessions`	Gauge	Current client sessions (includes offline clients)
`mosquitto_clients_offline`	Gauge	Current offline client count
`mosquitto_clients_online`	Gauge	Current online client count
`mosquitto_clients_expired`	Counter	Total clients expired due to keepalive
`mosquitto_subscriptions`	Gauge	Current subscription count
`mosquitto_shared_subscriptions`	Gauge	Current shared subscription count
`mosquitto_retained_messages`	Gauge	Current retained message count
`mosquitto_bytes_received`	Counter	Total bytes received
`mosquitto_bytes_sent`	Counter	Total bytes sent
`mosquitto_pub_bytes_received`	Counter	Total PUBLISH payload bytes received
`mosquitto_pub_bytes_sent`	Counter	Total PUBLISH payload bytes sent
`mosquitto_message_store_count`	Gauge	Current stored message count
`mosquitto_message_store_bytes`	Gauge	Current stored message bytes
`mosquitto_out_packets`	Gauge	Current queued outgoing packet count (includes offline clients)
`mosquitto_out_packet_bytes`	Gauge	Current queued outgoing packet bytes (includes offline clients)
`mosquitto_socket_connections`	Counter	Total incoming connections
`mosquitto_mqtt_packets_received`	Counter	Total MQTT packets received
`mosquitto_mqtt_packets_sent`	Counter	Total MQTT packets sent
`mosquitto_mqtt_connect_received`	Counter	MQTT CONNECT received
`mosquitto_mqtt_connect_sent`	Counter	MQTT CONNECT sent
`mosquitto_mqtt_connack_received`	Counter	MQTT CONNACK received
`mosquitto_mqtt_connack_sent`	Counter	MQTT CONNACK sent
`mosquitto_mqtt_publish_dropped`	Counter	MQTT PUBLISH dropped
`mosquitto_mqtt_publish_received`	Counter	MQTT PUBLISH received
`mosquitto_mqtt_publish_sent`	Counter	MQTT PUBLISH sent
`mosquitto_mqtt_puback_received`	Counter	MQTT PUBACK received
`mosquitto_mqtt_puback_sent`	Counter	MQTT PUBACK sent
`mosquitto_mqtt_pubrec_received`	Counter	MQTT PUBREC received
`mosquitto_mqtt_pubrec_sent`	Counter	MQTT PUBREC sent
`mosquitto_mqtt_pubrel_received`	Counter	MQTT PUBREL received
`mosquitto_mqtt_pubrel_sent`	Counter	MQTT PUBREL sent
`mosquitto_mqtt_pubcomp_received`	Counter	MQTT PUBCOMP received
`mosquitto_mqtt_pubcomp_sent`	Counter	MQTT PUBCOMP sent
`mosquitto_mqtt_subscribe_received`	Counter	MQTT SUBSCRIBE received
`mosquitto_mqtt_subscribe_sent`	Counter	MQTT SUBSCRIBE sent
`mosquitto_mqtt_suback_received`	Counter	MQTT SUBACK received
`mosquitto_mqtt_suback_sent`	Counter	MQTT SUBACK sent
`mosquitto_mqtt_unsubscribe_received`	Counter	MQTT UNSUBSCRIBE received
`mosquitto_mqtt_unsubscribe_sent`	Counter	MQTT UNSUBSCRIBE sent
`mosquitto_mqtt_unsuback_received`	Counter	MQTT UNSUBACK received
`mosquitto_mqtt_unsuback_sent`	Counter	MQTT UNSUBACK sent
`mosquitto_mqtt_pingreq_received`	Counter	MQTT PINGREQ received
`mosquitto_mqtt_pingreq_sent`	Counter	MQTT PINGREQ sent
`mosquitto_mqtt_pingresp_received`	Counter	MQTT PINGRESP received
`mosquitto_mqtt_pingresp_sent`	Counter	MQTT PINGRESP sent
`mosquitto_mqtt_disconnect_received`	Counter	MQTT DISCONNECT received
`mosquitto_mqtt_disconnect_sent`	Counter	MQTT DISCONNECT sent
`mosquitto_mqtt_auth_received`	Counter	MQTT AUTH received
`mosquitto_mqtt_auth_sent`	Counter	MQTT AUTH sent
`mosquitto_basic_auth_success`	Counter	Successful username/password authentication attempts
`mosquitto_basic_auth_fail`	Counter	Failed username/password authentication attempts
`mosquitto_basic_auth_error`	Counter	Errored username/password authentication attempts
`mosquitto_extended_auth_success`	Counter	Successful extended authentication attempts
`mosquitto_extended_auth_fail`	Counter	Failed extended authentication attempts
`mosquitto_extended_auth_error`	Counter	Errored extended authentication attempts

The broker process related metrics are exported by the Prometheus exporter only.

label	metric type	description
`process_max_fds`	Gauge	Maximum number of allowed open file descriptors (including network sockets).
`process_virtual_memory_max_bytes`	Gauge	Maximum amount of virtual memory available in bytes
`process_cpu_seconds_total`	Gauge	Total user and system CPU time spent in seconds.
`process_virtual_memory_bytes`	Gauge	Virtual memory size in bytes.
`process_resident_memory_bytes`	Gauge	Resident memory size in bytes.
`process_start_time_seconds`	Gauge	Start time of the process since unix epoch in seconds.
`process_open_fds`	Gauge	Number of open file descriptors (including network sockets).

Introduction​

General setup​

Global section​

Defaults section​

Direct pass-through, without PROXY protocol​

TLS termination with server certificate only, without PROXY protocol​

TLS termination with mutual TLS - client and server certificates, without PROXY protocol​

Direct pass through, with PROXY protocol​

TLS termination with server certificate only, with PROXY protocol​

TLS termination with mutual TLS - client and server certificate, with PROXY protocol​

TLS termination with mutual TLS - client and server certificate, with username and PROXY protocol​

Plugin configuration​

Prometheus Metrics Exporter​

InfluxDB Metrics Exporter​

Connection and authentication​

Influx 2.x​

Influx 1.6/8​

Basic Authentication​