HTTP#

Introduction#

Measurement readings can be acquired through HTTP using different formats like x-www-form-urlencoded, JSON and CSV.

The former both are full-qualified serialization formats containing named sensor readings of key/value pairs, CSV is an unqualified serialization format. Unqualified serialization formats require field names to be announced upfront by sending a special CSV header. See CSV format for an example.

Both single shot readings as well as bulk transfer modes are supported. There is a special field name time which is reserved for submitting data readings including timestamps. This is specifically important for bulk data submissions, see Readings with timestamp for an example and the whole list of available Timestamp formats as reference.

Setup#

HTTP data acquisition works using a HTTP-to-MQTT convergence handler. Please have a look at HTTP acquisition API configuration about how to configure a HTTP endpoint as an addon to a MQTTKit application application.

Basic examples#

Data acquisition with curl#

A basic example about sending telemetry data using curl:

# Define channel address
export DEVICE_REALM="mqttkit-1"
export DEVICE_TOPIC="testdrive/area-42/node-1"

# Compute HTTP URI
export HTTP_URI=http://localhost:24642/api/$DEVICE_REALM/$DEVICE_TOPIC/data

# Send readings in JSON format
echo '{"temperature": 42.84, "humidity": 83}' | curl --request POST --header 'Content-Type: application/json' --data @- $HTTP_URI

# Send readings in x-www-form-urlencoded format
echo 'temperature=42.84&humidity=83' | curl --request POST --header 'Content-Type: application/x-www-form-urlencoded' --data @- $HTTP_URI

Data acquisition with HTTPie#

HTTPie is a cURL-like tool for humans and even more convenient.

Setup#

Choose one of these ways to install HTTPie, depending on your platform:

# Debian/Ubuntu
apt install httpie

# Mac OSX: Homebrew
brew install httpie

# Mac OSX: Macports
port install httpie

# Vanilla Python
pip install httpie

# Windows
# Install the Chocolatey package manager: https://chocolatey.org/install
cinst httpie -source python

Single readings#

# Send readings in JSON format
http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data temperature:=42.84 humidity:=83

# Send readings in x-www-form-urlencoded format
http --form POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data temperature:=42.84 humidity:=83

Readings with timestamp#

When sending data readings using JSON or x-www-form-urlencoded, a timestamp may be supplied. Example:

# Send readings with human-readable timestamp in UTC
http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data time=2016-12-07T17:30:15.842428Z temperature:=42.84 humidity:=83

See also the whole list of Timestamp formats.

Events#

Events can be submitted to HTTP at the endpoint URI suffix /event. Grafana Annotations will be created automatically. Read more about it at HTTP events.

CSV format#

CSV is an universal but unqualified serialization format, so field names have to be announced upfront by sending a special CSV header.

These registered field names are retained across server restarts and will be mapped against the value-only readings before propagating them to the storage subsystem and other downstream data channels. The list of registered field names can be updated any time.

Single readings#

Transmit a single reading using CSV serialization. To register field names with a data channel before sending any sensor readings, use a comma separated list of field names prefixed by ## as the data line format, e.g. ## weight, temperature, humidity.

Example:

# 1. Send field names once
echo '## weight, temperature, humidity' | http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv

# 2. Send readings
echo '42.42, 34.02, 82.82' | http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv

Note

The line format is whitespace agnostic.

Readings with timestamp#

When sending data readings using CSV, a timestamp may be supplied. Examples:

# 1. Send field names
echo '## time, weight, temperature, humidity' | http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv

# 2.a Send readings with human-readable timestamp in UTC
echo '2016-12-07T17:00:00.842428Z, 50.42,40.02,82.82' | http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv

# 2.b Send readings with timestamp in nanoseconds since Epoch
echo '1478021421000000000, 50.42,40.02,82.82' | http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv

# 2.c Sending an empty timestamp will use current server time
echo ', 50.42,40.02,82.82' | http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv

See also the whole list of Timestamp formats.

Bulk readings#

The field name announcement header and multiple readings can be sent at once for HTTP bulk upload. Each line should be separated by a newline \n.

Let’s assume a file data.csv containing:

## time, weight, temperature, humidity, voltage
2016-08-14T21:02:06, 58.697, 19.6, 56.1, 4.13
2016-08-14T21:22:06, 58.663, 19.4, 58.3, 4.13
2016-08-14T21:42:06, 58.601, 19.1, 57.7, 4.12

Upload the file all at once:

cat data.csv | http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv

Integrations#

The HTTP data acquisition interface offers integrations with other upstream systems.

The Things Stack + Network#

The The Things Stack & Network (TTS, TTN) provides efficient integration with the TTS server and TTN network, in both single-device and multi-tenant scenarios.

Periodic acquisition example#

See also

Sawtooth signal with HTTP.

Language bindings#

See HTTP libraries.

Troubleshooting#

Some guidelines about troubleshooting the HTTP interface when sending invalid data acquisition requests.

See also

Please read about general MQTT error signalling.

Empty request body#

Empty request bodies don’t make sense:

http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:text/csv

so the service will reject them appropriately:

HTTP/1.1 400 Bad Request
Channel-Id: /mqttkit-1/testdrive/area-42/node-1
Content-Type: application/json
Date: Sat, 10 Dec 2016 19:16:44 GMT

[
    {
        "message": "Empty request body",
        "type": "error"
    }
]

Invalid content type#

The HTTP interface currently accepts the media types:

  • application/json

  • application/x-www-form-urlencoded

  • text/csv

Sending content designated with an invalid media type, e.g. unknown/format:

echo '58.697, 19.6, 56.1, 4.13' | http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-1/data Content-Type:unknown/format

will also get rejected:

HTTP/1.1 415 Unsupported Media Type
Channel-Id: /mqttkit-1/testdrive/area-42/node-1
Content-Type: application/json
Date: Sat, 10 Dec 2016 19:17:08 GMT

[
    {
        "message": "Unable to handle Content-Type 'unknown/format'",
        "type": "error"
    }
]

Readings without fields#

When using unqualified data serialization formats like CSV, field names must have been registered prior to sending sensor readings. When not doing so:

echo -e '2016-12-07T18:30:15+0100,  58.697, 19.6, 56.1, 4.13' | http POST http://localhost:24642/api/mqttkit-1/testdrive/area-42/node-unknown/data Content-Type:text/csv

the service will respond:

HTTP/1.1 400 Bad Request
Channel-Id: /mqttkit-1/testdrive/area-42/node-1
Content-Type: application/json
Date: Sat, 10 Dec 2016 19:17:25 GMT

[
    {
        "message": "Could not process data, please supply field names before sending readings",
        "type": "error"
    }
]