MQTT
This section describes how to create data migration tasks through the Explorer interface, migrating data from MQTT to the current TDengine cluster.
Overview
MQTT stands for Message Queuing Telemetry Transport. It is a lightweight messaging protocol that is easy to implement and use.
TDengine can subscribe to data from an MQTT broker via an MQTT connector and write it into TDengine, enabling real-time data streaming.
Creating a Task
1. Add a Data Source
On the data writing page, click the +Add Data Source button to enter the add data source page.
2. Configure Basic Information
Enter the task name in Name, such as: "test_mqtt";
Select MQTT from the Type dropdown list.
Broker is optional, you can select a specific broker from the dropdown list or click the +Create New Broker button on the right.
Select a target database from the Target Database dropdown list, or click the +Create Database button on the right.
3. Configure Connection and Authentication Information
Enter the MQTT broker's address in MQTT Address, for example: 192.168.1.42
Enter the MQTT broker's port in MQTT Port, for example: 1883
Enter the MQTT broker's username in User.
Enter the MQTT broker's password in Password.
4. Configure SSL Certificate
If the MQTT broker uses an SSL certificate, upload the certificate file in SSL Certificate.
5. Configure Collection Information
Fill in the collection task related configuration parameters in the Collection Configuration area.
Select the MQTT protocol version from the MQTT Protocol dropdown list. There are three options: 3.1, 3.1.1, 5.0. The default value is 3.1.
Enter the client identifier in Client ID, after which a client id with the prefix taosx will be generated (for example, if the identifier entered is foo, the generated client id will be taosxfoo). If the switch at the end is turned on, the current task's task id will be concatenated after taosx and before the entered identifier (the generated client id will look like taosx100foo). All client ids connecting to the same MQTT address must be unique.
Enter the keep alive interval in Keep Alive. If the broker does not receive any message from the client within the keep alive interval, it will assume the client has disconnected and will close the connection. The keep alive interval is the time interval negotiated between the client and the broker to check if the client is active. If the client does not send a message to the broker within the keep alive interval, the broker will disconnect.
In Clean Session, choose whether to clear the session. The default value is true.
Fill in the Topic names to be consumed in Subscription Topics and QoS Configuration. Use the following format: topic1::0,topic2::1.
Click the Check Connectivity button to check if the data source is available.
6. Configure MQTT Payload Parsing
Fill in the Payload parsing related configuration parameters in the MQTT Payload Parsing area.
taosX can use a JSON extractor to parse data and allows users to specify the data model in the database, including specifying table names and supertable names, setting ordinary columns and tag columns, etc.
6.1 Parsing
There are three methods to obtain sample data:
Click the Retrieve from Server button to get sample data from MQTT.
Click the File Upload button to upload a CSV file and obtain sample data.
Fill in the example data from the MQTT message body in Message Body.
JSON data supports JSONObject or JSONArray, and the json parser can parse the following data:
{"id": 1, "message": "hello-word"}
{"id": 2, "message": "hello-word"}
or
[{"id": 1, "message": "hello-word"},{"id": 2, "message": "hello-word"}]
The analysis results are as follows:
Click the magnifying glass icon to view the preview of the analysis results.
6.2 Field Splitting
In Extract or Split from Column, fill in the fields to extract or split from the message body, for example: split the message field into message_0 and message_1, select the split extractor, fill in the separator as -, and number as 2.
Click Delete to remove the current extraction rule.
Click Add to add more extraction rules.
Click the magnifying glass icon to view the preview of the extraction/split results.
6.3 Data Filtering
In Filter, fill in the filtering conditions, for example: write id != 1, then only data with id not equal to 1 will be written to TDengine.
Click Delete to remove the current filtering rule.
Click the magnifying glass icon to view the preview of the filtering results.
6.4 Table Mapping
In the Target Supertable dropdown, select a target supertable, or click the Create Supertable button on the right.
In Mapping, fill in the subtable name in the target supertable, for example: t_{id}. Fill in the mapping rules according to the requirements, where mapping supports setting default values.
Click Preview to view the mapping results.
7. Advanced Options
In the Log Level dropdown, select a log level. There are five options: TRACE, DEBUG, INFO, WARN, ERROR. The default is INFO.
When saving raw data, the following two parameters are effective.
Set the maximum retention days for raw data in Maximum Retention Days.
Set the storage path for raw data in Raw Data Storage Directory.
8. Completion
Click the Submit button to complete the creation of the MQTT to TDengine data synchronization task, return to the Data Source List page to view the status of the task execution.
