hgerstung/data2mqtt repository overview

⁠Introduction

This image runs a python script named data2mqtt which, based on a YAML configuration file, pulls data from URLs (http, https and local file read supported), detects the format automatically (JSON, XML, CSV and YAML are currently supported), parses the data and sends it to an MQTT broker.

Most home automation platforms offer an MQTT broker plugin/integration/adapter and this tool allows to feed data from most RESTful APIs or other sources into these systems without requiring a dedicated adapter.

The container requires a mapping for the configuration file (/opt/config.yaml) and supports various log levels which can be defined with the LOGLEVEL environment variable.

In the newest version, this image contains a web based configuration editor which is started automatically with the data2mqtt program. Per default it listens on port 8833, this can be changed by specifying the port in a WEBPORT environment variable. Please ensure that you map the port if you want to access the (simple) web UI.

⁠Configuration File

The configuration file used in this image is defined as /opt/config.yaml and therefore a mapping needs to exist that provides at least this file (or the whole /opt directory for the container).

⁠Example

The configuration file format looks like this:

configurations:
  - name: "default"
    url: "https://example.com/data.json"
    mqtt_server: "192.168.1.100"
    mqtt_port: 1883
    prefix: "my/prefix"
    username: "user"
    password: "pass"
    mqttuser: "mqtt_user"
    mqttpassword: "mqtt_pass"
    verify: "/path/to/ca_bundle.pem"
    interval: 10
  - name: "backup"
    url: "https://backup.example.com/data.json"
    mqtt_server: "192.168.1.101"
    mqtt_port: 1884
    prefix: "backup/prefix"
    interval: 20

⁠Explanation:

The configuration file starts with a "configurations:" line. A file can hold multiple configuration sets, each starts with a "- name:" and needs to be indented as required by YAML.

Each configuration set must have the following mandatory "field: value" pairs:

• name: a unique name for a configuration set, will be used for logging purposes
• url: the URL from which data2mqtt should request the data. This can be a REST-API node or a static file that is created by some other tool etc.
• mqtt_server: the IP address or hostname of the MQTT broker to which the parsed data should be published , e.g. "192.168.1.100"

Additional optional parameters are:

• mqtt_port: the TCP port for the MQTT broker, default is 1883
• prefix: "my/prefix"
• username: if the data source requires authentication, this defines the user name. For example, if a REST-API node can only be accessed with provided login credentials, this is used to define the user name for the REST-API login.
• password: the clear-text password used to access the data source
• mqttuser: in case the MQTT broker requires authentication, this defines the user name
• mqttpassword: password for MQTT broker authentication
• verify: for HTTPS URLs, this defines how to verify the SSL certifcate of the server. This can be "false" if no verification should be done, or a path to your own CA bundle, e.g. "/path/to/ca_bundle.pem". If this parameter is missing, the host SSL certificate is checked using standard root CA certificates
• interval: number of seconds (numeric value, e.g. 10) to wait between two consecutive data requests from the URL. If this is not defined, data2mqtt will only try to request data once after starting up

⁠File Mappings

It is required to map the configuration file to /opt/config.yaml (or map a directory to /opt).

⁠Port mappings

The TCP port for the configuration editor requires a port mapping, if it should be made accessible. There is currently no security implemented, i.e. no password required to access and change configuration. It is recommended to change the configuration file using an editor or temporarily map the port, edit the configuration and then, before restarting the container, remove the mapping again.

⁠Environment variables

WEBPORT defines the port on which the configuration editor is running (configeditor.py), default is 8833. CFGFILE defines the path to the configuration file used in the container, default here is "/opt/config.yaml". LOGLEVEL defines the log verbosity.

The following log levels are supported:

• 0 - no output at all
• 1 - only errors
• 2 - log config set name each time a request to the URL is attempted
• 3 - log config set name and URL for each request
• 4 - log every MQTT topic including value that is published to the MQTT broker
• 10 - print all parameters for each request (URL, MQTT host etc. etc.)
• 15 - debugging output (logging all steps of processing a configuration set)
• 25 - extended debugging (dumping received data)

Everything after 3 can result in a log of log entries and should be avoided until necessary. In production mode, a loglevel of 1 should be used to avoid creating log entries for every request.