Payload Offloading

Introduction

The Payload Offloading plugin offloads MQTT payloads to an external storage backend instead of keeping them fully in memory, if they exceed a configurable size threshold and are either retained or have QoS > 0. Any storage backend increases latency due to overhead in the broker's message processing when offloading payloads to storage and retrieving them again for dispatch to clients. To mitigate this, the plugin has an additional LRU (Least Recently Used) cache with configurable size. This cache is held in-memory and always stores at least one message regardless of its configured size. This only violates the configured cache size in cases where a single message's payload size is greater than the cache size.

The current version of the plugin supports a file based storage backend including using a directory hierarchy based on the internal store ids to not slow down the filesystem when offloading loads of files. For clustered setups the current approach will only work, if the persistence location of the broker is located on a synchronized filesystem (e.g., NFS or GlusterFS) that all cluster nodes are able to access. Moreover, the numParallelWrites configuration parameter currently only applies for adding data to the storage, not for deletion.

Plugin Activation

To load and enable the plugin into the broker, the "mosquitto.conf" must be extended by:

Docker:

plugin /usr/lib/cedalo_payload_offloading.so

persistence_location /mosquitto/data

CentOS / RHEL:

plugin /opt/cedalo/mosquitto/lib/cedalo_payload_offloading.so

persistence_location /var/opt/cedalo/mosquitto

Configuration

The config file name must be Payload-Offloading.json (case sensitive) and placed inside the configured persistence_location.

Example configuration (Payload-Offloading.json):

{
  "minOffloadPayloadSize": 1024,
  "maxCachedPayloadMemory": 1048576,
  "numParallelWrites": 3,
  "storage": {
    "type": "file"
  }
}

See the JSON schema below for details.

JSON Schema

{
  "type": "object",
  "description": "Config object representing the Payload Offloading plugin configuration.",
  "properties": {
    "minOffloadPayloadSize": {
      "type": "integer",
      "minimum": 0,
      "description": "Minimum payload size in bytes, at or above which a message payload is offloaded to the configured storage backend instead of being kept in memory. A value of 0 essentially means no offloading will happen."
    },
    "maxCachedPayloadMemory": {
      "type": "integer",
      "minimum": 0,
      "description": "Maximum amount of memory in bytes that may be used by the plugin's LRU cache. At least one message is always held in the cache, regardless of the configured size."
    },
    "numParallelWrites": {
      "type": "integer",
      "minimum": 0,
      "description": "Maximum number of concurrent write operations to the storage backend. If not specified, no explicit limit is applied."
    },
    "storage": {
      "type": "object",
      "description": "Configuration of the storage backend used to offload payloads to. The concrete set of properties depends on the selected \"type\".",
      "properties": {
        "type": {
          "type": "string",
          "description": "Selects the storage backend implementation.",
          "enum": ["file"],
          "default": "file"
        },
        "offloadingDir": {
          "type": "string",
          "description": "Directory where offloaded payloads are stored by the file storage backend. If not specified, a default location inside the broker's persistence_location is used."
        }
      },
      "required": ["type"]
    }
  },
  "required": ["minOffloadPayloadSize", "maxCachedPayloadMemory", "storage"]
}