taosAdapter Reference

taosAdapter is a companion tool for TDengine, serving as a bridge and adapter between the TDengine cluster and applications. It provides an easy and efficient way to ingest data directly from data collection agents (such as Telegraf, StatsD, collectd, etc.). It also offers InfluxDB/OpenTSDB compatible data ingestion interfaces, allowing InfluxDB/OpenTSDB applications to be seamlessly ported to TDengine.

taosAdapter offers the following features:

• RESTful interface
• Compatible with InfluxDB v1 write interface
• Compatible with OpenTSDB JSON and telnet format writing
• Seamless connection to Telegraf
• Seamless connection to collectd
• Seamless connection to StatsD
• Supports Prometheus remote_read and remote_write
• Retrieves the VGroup ID of the virtual node group (VGroup) where the table is located

taosAdapter Architecture Diagram

Figure 1. taosAdapter architecture

Deployment Methods for taosAdapter

Installing taosAdapter

taosAdapter is part of the TDengine server software. If you are using TDengine server, you do not need any additional steps to install taosAdapter. If you need to deploy taosAdapter separately from the TDengine server, you should install the complete TDengine on that server to install taosAdapter. If you need to compile taosAdapter from source code, you can refer to the Build taosAdapter document.

Starting/Stopping taosAdapter

On Linux systems, the taosAdapter service is managed by default by systemd. Use the command systemctl start taosadapter to start the taosAdapter service. Use the command systemctl stop taosadapter to stop the taosAdapter service.

Removing taosAdapter

Use the command rmtaos to remove the TDengine server software, including taosAdapter.

Upgrading taosAdapter

taosAdapter and TDengine server need to use the same version. Please upgrade taosAdapter by upgrading the TDengine server. taosAdapter deployed separately from taosd must be upgraded by upgrading the TDengine server on its server.

taosAdapter Parameter List

taosAdapter supports configuration through command-line parameters, environment variables, and configuration files. The default configuration file is /etc/taos/taosadapter.toml.

Command-line parameters take precedence over environment variables, which take precedence over configuration files. The command-line usage is arg=val, such as taosadapter -p=30000 --debug=true, detailed list as follows:

Usage of taosAdapter:
      --collectd.db string                           collectd db name. Env "TAOS_ADAPTER_COLLECTD_DB" (default "collectd")
      --collectd.enable                              enable collectd. Env "TAOS_ADAPTER_COLLECTD_ENABLE" (default true)
      --collectd.password string                     collectd password. Env "TAOS_ADAPTER_COLLECTD_PASSWORD" (default "taosdata")
      --collectd.port int                            collectd server port. Env "TAOS_ADAPTER_COLLECTD_PORT" (default 6045)
      --collectd.ttl int                             collectd data ttl. Env "TAOS_ADAPTER_COLLECTD_TTL"
      --collectd.user string                         collectd user. Env "TAOS_ADAPTER_COLLECTD_USER" (default "root")
      --collectd.worker int                          collectd write worker. Env "TAOS_ADAPTER_COLLECTD_WORKER" (default 10)
  -c, --config string                                config path default /etc/taos/taosadapter.toml
      --cors.allowAllOrigins                         cors allow all origins. Env "TAOS_ADAPTER_CORS_ALLOW_ALL_ORIGINS" (default true)
      --cors.allowCredentials                        cors allow credentials. Env "TAOS_ADAPTER_CORS_ALLOW_Credentials"
      --cors.allowHeaders stringArray                cors allow HEADERS. Env "TAOS_ADAPTER_ALLOW_HEADERS"
      --cors.allowOrigins stringArray                cors allow origins. Env "TAOS_ADAPTER_ALLOW_ORIGINS"
      --cors.allowWebSockets                         cors allow WebSockets. Env "TAOS_ADAPTER_CORS_ALLOW_WebSockets"
      --cors.exposeHeaders stringArray               cors expose headers. Env "TAOS_ADAPTER_Expose_Headers"
      --debug                                        enable debug mode. Env "TAOS_ADAPTER_DEBUG" (default true)
      --help                                         Print this help message and exit
      --httpCodeServerError                          Use a non-200 http status code when server returns an error. Env "TAOS_ADAPTER_HTTP_CODE_SERVER_ERROR"
      --influxdb.enable                              enable influxdb. Env "TAOS_ADAPTER_INFLUXDB_ENABLE" (default true)
      --instanceId int                               instance ID. Env "TAOS_ADAPTER_INSTANCE_ID" (default 32)
      --log.compress                                 whether to compress old log. Env "TAOS_ADAPTER_LOG_COMPRESS"
      --log.enableRecordHttpSql                      whether to record http sql. Env "TAOS_ADAPTER_LOG_ENABLE_RECORD_HTTP_SQL"
      --log.level string                             log level (trace debug info warning error). Env "TAOS_ADAPTER_LOG_LEVEL" (default "info")
      --log.path string                              log path. Env "TAOS_ADAPTER_LOG_PATH" (default "/var/log/taos")
      --log.reservedDiskSize string                  reserved disk size for log dir (KB MB GB), must be a positive integer. Env "TAOS_ADAPTER_LOG_RESERVED_DISK_SIZE" (default "1GB")
      --log.rotationCount uint                       log rotation count. Env "TAOS_ADAPTER_LOG_ROTATION_COUNT" (default 30)
      --log.rotationSize string                      log rotation size(KB MB GB), must be a positive integer. Env "TAOS_ADAPTER_LOG_ROTATION_SIZE" (default "1GB")
      --log.rotationTime duration                    deprecated: log rotation time always 24 hours. Env "TAOS_ADAPTER_LOG_ROTATION_TIME" (default 24h0m0s)
      --log.sqlRotationCount uint                    record sql log rotation count. Env "TAOS_ADAPTER_LOG_SQL_ROTATION_COUNT" (default 2)
      --log.sqlRotationSize string                   record sql log rotation size(KB MB GB), must be a positive integer. Env "TAOS_ADAPTER_LOG_SQL_ROTATION_SIZE" (default "1GB")
      --log.sqlRotationTime duration                 record sql log rotation time. Env "TAOS_ADAPTER_LOG_SQL_ROTATION_TIME" (default 24h0m0s)
      --logLevel string                              log level (trace debug info warning error). Env "TAOS_ADAPTER_LOG_LEVEL" (default "info")
      --monitor.collectDuration duration             Set monitor duration. Env "TAOS_ADAPTER_MONITOR_COLLECT_DURATION" (default 3s)
      --monitor.disable                              Whether to disable monitoring. Env "TAOS_ADAPTER_MONITOR_DISABLE" (default true)
      --monitor.identity string                      The identity of the current instance, or 'hostname:port' if it is empty. Env "TAOS_ADAPTER_MONITOR_IDENTITY"
      --monitor.incgroup                             Whether running in cgroup. Env "TAOS_ADAPTER_MONITOR_INCGROUP"
      --monitor.pauseAllMemoryThreshold float        Memory percentage threshold for pause all. Env "TAOS_ADAPTER_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD" (default 80)
      --monitor.pauseQueryMemoryThreshold float      Memory percentage threshold for pause query. Env "TAOS_ADAPTER_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD" (default 70)
      --node_exporter.caCertFile string              node_exporter ca cert file path. Env "TAOS_ADAPTER_NODE_EXPORTER_CA_CERT_FILE"
      --node_exporter.certFile string                node_exporter cert file path. Env "TAOS_ADAPTER_NODE_EXPORTER_CERT_FILE"
      --node_exporter.db string                      node_exporter db name. Env "TAOS_ADAPTER_NODE_EXPORTER_DB" (default "node_exporter")
      --node_exporter.enable                         enable node_exporter. Env "TAOS_ADAPTER_NODE_EXPORTER_ENABLE"
      --node_exporter.gatherDuration duration        node_exporter gather duration. Env "TAOS_ADAPTER_NODE_EXPORTER_GATHER_DURATION" (default 5s)
      --node_exporter.httpBearerTokenString string   node_exporter http bearer token. Env "TAOS_ADAPTER_NODE_EXPORTER_HTTP_BEARER_TOKEN_STRING"
      --node_exporter.httpPassword string            node_exporter http password. Env "TAOS_ADAPTER_NODE_EXPORTER_HTTP_PASSWORD"
      --node_exporter.httpUsername string            node_exporter http username. Env "TAOS_ADAPTER_NODE_EXPORTER_HTTP_USERNAME"
      --node_exporter.insecureSkipVerify             node_exporter skip ssl check. Env "TAOS_ADAPTER_NODE_EXPORTER_INSECURE_SKIP_VERIFY" (default true)
      --node_exporter.keyFile string                 node_exporter cert key file path. Env "TAOS_ADAPTER_NODE_EXPORTER_KEY_FILE"
      --node_exporter.password string                node_exporter password. Env "TAOS_ADAPTER_NODE_EXPORTER_PASSWORD" (default "taosdata")
      --node_exporter.responseTimeout duration       node_exporter response timeout. Env "TAOS_ADAPTER_NODE_EXPORTER_RESPONSE_TIMEOUT" (default 5s)
      --node_exporter.ttl int                        node_exporter data ttl. Env "TAOS_ADAPTER_NODE_EXPORTER_TTL"
      --node_exporter.urls strings                   node_exporter urls. Env "TAOS_ADAPTER_NODE_EXPORTER_URLS" (default [http://localhost:9100])
      --node_exporter.user string                    node_exporter user. Env "TAOS_ADAPTER_NODE_EXPORTER_USER" (default "root")
      --opentsdb.enable                              enable opentsdb. Env "TAOS_ADAPTER_OPENTSDB_ENABLE" (default true)
      --opentsdb_telnet.batchSize int                opentsdb_telnet batch size. Env "TAOS_ADAPTER_OPENTSDB_TELNET_BATCH_SIZE" (default 1)
      --opentsdb_telnet.dbs strings                  opentsdb_telnet db names. Env "TAOS_ADAPTER_OPENTSDB_TELNET_DBS" (default [opentsdb_telnet,collectd_tsdb,icinga2_tsdb,tcollector_tsdb])
      --opentsdb_telnet.enable                       enable opentsdb telnet,warning: without auth info(default false). Env "TAOS_ADAPTER_OPENTSDB_TELNET_ENABLE"
      --opentsdb_telnet.flushInterval duration       opentsdb_telnet flush interval (0s means not valid) . Env "TAOS_ADAPTER_OPENTSDB_TELNET_FLUSH_INTERVAL"
      --opentsdb_telnet.maxTCPConnections int        max tcp connections. Env "TAOS_ADAPTER_OPENTSDB_TELNET_MAX_TCP_CONNECTIONS" (default 250)
      --opentsdb_telnet.password string              opentsdb_telnet password. Env "TAOS_ADAPTER_OPENTSDB_TELNET_PASSWORD" (default "taosdata")
      --opentsdb_telnet.ports ints                   opentsdb_telnet tcp port. Env "TAOS_ADAPTER_OPENTSDB_TELNET_PORTS" (default [6046,6047,6048,6049])
      --opentsdb_telnet.tcpKeepAlive                 enable tcp keep alive. Env "TAOS_ADAPTER_OPENTSDB_TELNET_TCP_KEEP_ALIVE"
      --opentsdb_telnet.ttl int                      opentsdb_telnet data ttl. Env "TAOS_ADAPTER_OPENTSDB_TELNET_TTL"
      --opentsdb_telnet.user string                  opentsdb_telnet user. Env "TAOS_ADAPTER_OPENTSDB_TELNET_USER" (default "root")
      --pool.idleTimeout duration                    Set idle connection timeout. Env "TAOS_ADAPTER_POOL_IDLE_TIMEOUT"
      --pool.maxConnect int                          max connections to server. Env "TAOS_ADAPTER_POOL_MAX_CONNECT"
      --pool.maxIdle int                             max idle connections to server. Env "TAOS_ADAPTER_POOL_MAX_IDLE"
      --pool.maxWait int                             max count of waiting for connection. Env "TAOS_ADAPTER_POOL_MAX_WAIT"
      --pool.waitTimeout int                         wait for connection timeout seconds. Env "TAOS_ADAPTER_POOL_WAIT_TIMEOUT" (default 60)
  -P, --port int                                     http port. Env "TAOS_ADAPTER_PORT" (default 6041)
      --prometheus.enable                            enable prometheus. Env "TAOS_ADAPTER_PROMETHEUS_ENABLE" (default true)
      --restfulRowLimit int                          restful returns the maximum number of rows (-1 means no limit). Env "TAOS_ADAPTER_RESTFUL_ROW_LIMIT" (default -1)
      --smlAutoCreateDB                              Whether to automatically create db when writing with schemaless. Env "TAOS_ADAPTER_SML_AUTO_CREATE_DB"
      --statsd.allowPendingMessages int              statsd allow pending messages. Env "TAOS_ADAPTER_STATSD_ALLOW_PENDING_MESSAGES" (default 50000)
      --statsd.db string                             statsd db name. Env "TAOS_ADAPTER_STATSD_DB" (default "statsd")
      --statsd.deleteCounters                        statsd delete counter cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_COUNTERS" (default true)
      --statsd.deleteGauges                          statsd delete gauge cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_GAUGES" (default true)
      --statsd.deleteSets                            statsd delete set cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_SETS" (default true)
      --statsd.deleteTimings                         statsd delete timing cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_TIMINGS" (default true)
      --statsd.enable                                enable statsd. Env "TAOS_ADAPTER_STATSD_ENABLE"
      --statsd.gatherInterval duration               statsd gather interval. Env "TAOS_ADAPTER_STATSD_GATHER_INTERVAL" (default 5s)
      --statsd.maxTCPConnections int                 statsd max tcp connections. Env "TAOS_ADAPTER_STATSD_MAX_TCP_CONNECTIONS" (default 250)
      --statsd.password string                       statsd password. Env "TAOS_ADAPTER_STATSD_PASSWORD" (default "taosdata")
      --statsd.port int                              statsd server port. Env "TAOS_ADAPTER_STATSD_PORT" (default 6044)
      --statsd.protocol string                       statsd protocol [tcp or udp]. Env "TAOS_ADAPTER_STATSD_PROTOCOL" (default "udp4")
      --statsd.tcpKeepAlive                          enable tcp keep alive. Env "TAOS_ADAPTER_STATSD_TCP_KEEP_ALIVE"
      --statsd.ttl int                               statsd data ttl. Env "TAOS_ADAPTER_STATSD_TTL"
      --statsd.user string                           statsd user. Env "TAOS_ADAPTER_STATSD_USER" (default "root")
      --statsd.worker int                            statsd write worker. Env "TAOS_ADAPTER_STATSD_WORKER" (default 10)
      --taosConfigDir string                         load taos client config path. Env "TAOS_ADAPTER_TAOS_CONFIG_FILE"
      --uploadKeeper.enable                          Whether to enable sending metrics to keeper. Env "TAOS_ADAPTER_UPLOAD_KEEPER_ENABLE" (default true)
      --uploadKeeper.interval duration               send to Keeper interval. Env "TAOS_ADAPTER_UPLOAD_KEEPER_INTERVAL" (default 15s)
      --uploadKeeper.retryInterval duration          retry interval. Env "TAOS_ADAPTER_UPLOAD_KEEPER_RETRY_INTERVAL" (default 5s)
      --uploadKeeper.retryTimes uint                 retry times. Env "TAOS_ADAPTER_UPLOAD_KEEPER_RETRY_TIMES" (default 3)
      --uploadKeeper.timeout duration                send to Keeper timeout. Env "TAOS_ADAPTER_UPLOAD_KEEPER_TIMEOUT" (default 5s)
      --uploadKeeper.url string                      Keeper url. Env "TAOS_ADAPTER_UPLOAD_KEEPER_URL" (default "http://127.0.0.1:6043/adapter_report")
  -V, --version                                      Print the version and exit

Note: When using a browser to make API calls, please set the following Cross-Origin Resource Sharing (CORS) parameters according to the actual situation:

AllowAllOrigins
AllowOrigins
AllowHeaders
ExposeHeaders
AllowCredentials
AllowWebSockets

If you are not making API calls through a browser, you do not need to worry about these configurations.

For details about the CORS protocol, please refer to: https://www.w3.org/wiki/CORS_Enabled or https://developer.mozilla.org/docs/Web/HTTP/CORS.

See the example configuration file at example/config/taosadapter.toml.

Connection Pool Parameters Description

When using the RESTful API, the system will manage TDengine connections through a connection pool. The connection pool can be configured with the following parameters:

• pool.maxConnect: The maximum number of connections allowed in the pool, default is twice the number of CPU cores. It is recommended to keep the default setting.
• pool.maxIdle: The maximum number of idle connections in the pool, default is the same as pool.maxConnect. It is recommended to keep the default setting.
• pool.idleTimeout: Connection idle timeout, default is never timeout. It is recommended to keep the default setting.
• pool.waitTimeout: Timeout for obtaining a connection from the pool, default is set to 60 seconds. If a connection is not obtained within the timeout period, HTTP status code 503 will be returned. This parameter is available starting from version 3.3.3.0.
• pool.maxWait: The maximum number of requests waiting to get a connection in the pool, default is 0, which means no limit. When the number of queued requests exceeds this value, new requests will return HTTP status code 503. This parameter is available starting from version 3.3.3.0.

Feature List

• RESTful API RESTful API
• Compatible with InfluxDB v1 write interface https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/
• Compatible with OpenTSDB JSON and telnet format writing
  • http://opentsdb.net/docs/build/html/api_http/put.html
  • http://opentsdb.net/docs/build/html/api_telnet/put.html
• Seamless connection with collectd. collectd is a system statistics collection daemon, visit https://collectd.org/ for more information.
• Seamless connection with StatsD. StatsD is a simple yet powerful daemon for gathering statistics. Visit https://github.com/statsd/statsd for more information.
• Seamless connection with icinga2. icinga2 is a software for collecting check results metrics and performance data. Visit https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer for more information.
• Seamless connection with tcollector. TCollector is a client process that collects data from local collectors and pushes it to OpenTSDB. Visit http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html for more information.
• Seamless connection with node_exporter. node_exporter is an exporter of machine metrics. Visit https://github.com/prometheus/node_exporter for more information.
• Supports Prometheus remote_read and remote_write. remote_read and remote_write are Prometheus's data read-write separation cluster solutions. Visit https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis for more information.
• Get the VGroup ID of the virtual node group (VGroup) where the table is located.

Interface

TDengine RESTful Interface

You can use any client that supports the HTTP protocol to write data to TDengine or query data from TDengine by accessing the RESTful interface URL http://<fqdn>:6041/rest/sql. For details, please refer to the REST API documentation.

InfluxDB

You can use any client that supports the HTTP protocol to write data in InfluxDB compatible format to TDengine by accessing the Restful interface URL http://<fqdn>:6041/influxdb/v1/write.

Supported InfluxDB parameters are as follows:

• db specifies the database name used by TDengine
• precision the time precision used by TDengine
• u TDengine username
• p TDengine password
• ttl the lifespan of automatically created subtables, determined by the TTL parameter of the first data entry in the subtable, which cannot be updated. For more information, please refer to the TTL parameter in the table creation document.

Note: Currently, InfluxDB's token authentication method is not supported, only Basic authentication and query parameter verification are supported. Example: curl --request POST http://127.0.0.1:6041/influxdb/v1/write?db=test --user "root:taosdata" --data-binary "measurement,host=host1 field1=2i,field2=2.0 1577836800000000000"

OpenTSDB

You can use any client that supports the HTTP protocol to write data in OpenTSDB compatible format to TDengine by accessing the Restful interface URL http://<fqdn>:6041/<APIEndPoint>. EndPoint as follows:

/opentsdb/v1/put/json/<db>
/opentsdb/v1/put/telnet/<db>

collectd

Configuring taosAdapter

Method to configure taosAdapter to receive collectd data:

• Enable the configuration item in the taosAdapter configuration file (default location is /etc/taos/taosadapter.toml)

...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...

The default database name written by taosAdapter is collectd, but you can also modify the dbs item in the taosAdapter configuration file to specify a different name. Fill in user and password with the actual TDengine configuration values. After modifying the configuration file, taosAdapter needs to be restarted.

• You can also use taosAdapter command line parameters or set environment variables to start, to enable taosAdapter to receive collectd data, for more details please refer to the taosAdapter reference manual.

Configuring collectd

collectd uses a plugin mechanism that can write the collected monitoring data to different data storage software in various forms. TDengine supports direct collection plugins and write_tsdb plugins.

Configuring to receive direct collection plugin data

Modify the related configuration items in the collectd configuration file (default location /etc/collectd/collectd.conf).

LoadPlugin network
<Plugin network>
         Server "<taosAdapter's host>" "<port for collectd direct>"
</Plugin>

Where <taosAdapter's host> should be filled with the domain name or IP address of the server running taosAdapter. <port for collectd direct> should be filled with the port used by taosAdapter to receive collectd data (default is 6045).

Example as follows:

LoadPlugin network
<Plugin network>
         Server "127.0.0.1" "6045"
</Plugin>

Configuring write_tsdb plugin data

Modify the related configuration items in the collectd configuration file (default location /etc/collectd/collectd.conf).

LoadPlugin write_tsdb
<Plugin write_tsdb>
        <Node>
                Host "<taosAdapter's host>"
                Port "<port for collectd write_tsdb plugin>"
                ...
        </Node>
</Plugin>

Where <taosAdapter's host> should be filled with the domain name or IP address of the server running taosAdapter. <port for collectd write_tsdb plugin> should be filled with the port used by taosAdapter to receive collectd write_tsdb plugin data (default is 6047).

LoadPlugin write_tsdb
<Plugin write_tsdb>
        <Node>
                Host "127.0.0.1"
                Port "6047"
                HostTags "status=production"
                StoreRates false
                AlwaysAppendDS false
        </Node>
</Plugin>

Then restart collectd:

systemctl restart collectd

StatsD

Configure taosAdapter

Method to configure taosAdapter to receive StatsD data:

• Enable the configuration item in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)

...
[statsd]
enable = true
port = 6044
db = "statsd"
user = "root"
password = "taosdata"
worker = 10
gatherInterval = "5s"
protocol = "udp"
maxTCPConnections = 250
tcpKeepAlive = false
allowPendingMessages = 50000
deleteCounters = true
deleteGauges = true
deleteSets = true
deleteTimings = true
...

The default database name written by taosAdapter is statsd, but you can also modify the db item in the taosAdapter configuration file to specify a different name. Fill in the user and password with the actual TDengine configuration values. After modifying the configuration file, taosAdapter needs to be restarted.

• You can also use taosAdapter command line arguments or set environment variables to enable the taosAdapter to receive StatsD data. For more details, please refer to the taosAdapter reference manual.

Configure StatsD

To use StatsD, download its source code. Modify its configuration file according to the example file exampleConfig.js found in the root directory of the local source code download. Replace <taosAdapter's host> with the domain name or IP address of the server running taosAdapter, and <port for StatsD> with the port that taosAdapter uses to receive StatsD data (default is 6044).

Add to the backends section "./backends/repeater"
Add to the repeater section { host:'<taosAdapter's host>', port: <port for StatsD>}

Example configuration file:

{
port: 8125
, backends: ["./backends/repeater"]
, repeater: [{ host: '127.0.0.1', port: 6044}]
}

After adding the following content, start StatsD (assuming the configuration file is modified to config.js).

npm install
node stats.js config.js &

icinga2 OpenTSDB writer

Configuring taosAdapter

Method to configure taosAdapter to receive icinga2 data:

• Enable the configuration item in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)

...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...

The default database name written by taosAdapter is icinga2, but you can also modify the dbs item in the taosAdapter configuration file to specify a different name. Fill in user and password with the actual TDengine configuration values. taosAdapter needs to be restarted after modifications.

• You can also use taosAdapter command line parameters or set environment variables to enable taosAdapter to receive icinga2 data, for more details please refer to the taosAdapter reference manual

Configuring icinga2

• Enable icinga2's opentsdb-writer (reference link https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer)
• Modify the configuration file /etc/icinga2/features-enabled/opentsdb.conf filling in <taosAdapter's host> with the domain name or IP address of the server running taosAdapter, <port for icinga2> with the corresponding port supported by taosAdapter for receiving icinga2 data (default is 6048)

object OpenTsdbWriter "opentsdb" {
  host = "<taosAdapter's host>"
  port = <port for icinga2>
}

Example file:

object OpenTsdbWriter "opentsdb" {
  host = "127.0.0.1"
  port = 6048
}

TCollector

Configuring taosAdapter

To configure taosAdapter to receive data from TCollector:

• Enable the configuration in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)

...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...

The default database name that taosAdapter writes to is tcollector, but you can specify a different name by modifying the dbs option in the taosAdapter configuration file. Fill in the user and password with the actual values configured in TDengine. After modifying the configuration file, taosAdapter needs to be restarted.

• You can also use taosAdapter command line arguments or set environment variables to enable the taosAdapter to receive tcollector data. For more details, please refer to the taosAdapter reference manual.

Configuring TCollector

To use TCollector, download its source code. Its configuration options are in its source code. Note: There are significant differences between different versions of TCollector; this only refers to the latest code in the current master branch (git commit: 37ae920).

Modify the contents of collectors/etc/config.py and tcollector.py. Change the original address pointing to the OpenTSDB host to the domain name or IP address where taosAdapter is deployed, and change the port to the corresponding port supported by taosAdapter for TCollector use (default is 6049).

Example of git diff output for source code modifications:

index e7e7a1c..ec3e23c 100644
--- a/collectors/etc/config.py
+++ b/collectors/etc/config.py
@@ -59,13 +59,13 @@ def get_defaults():
         'http_password': False,
         'reconnectinterval': 0,
         'http_username': False,
-        'port': 4242,
+        'port': 6049,
         'pidfile': '/var/run/tcollector.pid',
         'http': False,
         'http_api_path': "api/put",
         'tags': [],
         'remove_inactive_collectors': False,
-        'host': '',
+        'host': '127.0.0.1',
         'logfile': '/var/log/tcollector.log',
         'cdir': default_cdir,
         'ssl': False,
diff --git a/tcollector.py b/tcollector.py
index 21f9b23..4c71ba2 100755
--- a/tcollector.py
+++ b/tcollector.py
@@ -64,7 +64,7 @@ ALIVE = True
 # exceptions, something is not right and tcollector will shutdown.
 # Hopefully some kind of supervising daemon will then restart it.
 MAX_UNCAUGHT_EXCEPTIONS = 100
-DEFAULT_PORT = 4242
+DEFAULT_PORT = 6049
 MAX_REASONABLE_TIMESTAMP = 2209212000  # Good until Tue  3 Jan 14:00:00 GMT 2040
 # How long to wait for datapoints before assuming
 # a collector is dead and restarting it
@@ -943,13 +943,13 @@ def parse_cmdline(argv):
             'http_password': False,
             'reconnectinterval': 0,
             'http_username': False,
-            'port': 4242,
+            'port': 6049,
             'pidfile': '/var/run/tcollector.pid',
             'http': False,
             'http_api_path': "api/put",
             'tags': [],
             'remove_inactive_collectors': False,
-            'host': '',
+            'host': '127.0.0.1',
             'logfile': '/var/log/tcollector.log',
             'cdir': default_cdir,
             'ssl': False,

node_exporter

An exporter used by Prometheus that exposes hardware and operating system metrics from *NIX kernels

• Enable configuration of taosAdapter node_exporter.enable
• Set the relevant configuration for node_exporter
• Restart taosAdapter

prometheus

Configuring Prometheus is done by editing the Prometheus configuration file prometheus.yml (default location /etc/prometheus/prometheus.yml).

Configure Third-Party Database Address

Set the remote_read url and remote_write url to point to the domain name or IP address of the server running the taosAdapter service, the REST service port (taosAdapter defaults to 6041), and the name of the database you want to write to in TDengine, ensuring the URLs are formatted as follows:

• remote_read url: http://<taosAdapter's host>:<REST service port>/prometheus/v1/remote_read/<database name>
• remote_write url: http://<taosAdapter's host>:<REST service port>/prometheus/v1/remote_write/<database name>

Configure Basic Authentication

• username: <TDengine's username>
• password: <TDengine's password>

Example configuration of remote_write and remote_read in the prometheus.yml file

remote_write:
  - url: "http://localhost:6041/prometheus/v1/remote_write/prometheus_data"
    basic_auth:
      username: root
      password: taosdata

remote_read:
  - url: "http://localhost:6041/prometheus/v1/remote_read/prometheus_data"
    basic_auth:
      username: root
      password: taosdata
    remote_timeout: 10s
    read_recent: true

Getting the VGroup ID of a table

You can access the HTTP interface http://<fqdn>:6041/rest/vgid?db=<db>&table=<table> to get the VGroup ID of a table.

Memory Usage Optimization Methods

taosAdapter will monitor its memory usage during operation and adjust it through two thresholds. Valid values range from -1 to 100 as a percentage of system physical memory.

• pauseQueryMemoryThreshold
• pauseAllMemoryThreshold

When the pauseQueryMemoryThreshold threshold is exceeded, it stops processing query requests.

HTTP return content:

• code 503
• body "query memory exceeds threshold"

When the pauseAllMemoryThreshold threshold is exceeded, it stops processing all write and query requests.

HTTP return content:

• code 503
• body "memory exceeds threshold"

When memory falls below the threshold, the corresponding functions are resumed.

Status check interface http://<fqdn>:6041/-/ping

• Normally returns code 200
• No parameters If memory exceeds pauseAllMemoryThreshold, it will return code 503
• Request parameter action=query If memory exceeds either pauseQueryMemoryThreshold or pauseAllMemoryThreshold, it will return code 503

Corresponding configuration parameters

  monitor.collectDuration              Monitoring interval                                    Environment variable "TAOS_MONITOR_COLLECT_DURATION" (default value 3s)
  monitor.incgroup                     Whether it is running in cgroup (set to true in containers)      Environment variable "TAOS_MONITOR_INCGROUP"
  monitor.pauseAllMemoryThreshold      Memory threshold for stopping inserts and queries                   Environment variable "TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD" (default value 80)
  monitor.pauseQueryMemoryThreshold    Memory threshold for stopping queries                        Environment variable "TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD" (default value 70)

You can adjust according to the specific project application scenario and operational strategy, and it is recommended to use operational monitoring software to monitor the system memory status in real time. Load balancers can also check the running status of taosAdapter through this interface.

taosAdapter Monitoring Metrics

taosAdapter collects monitoring metrics related to REST/WebSocket requests. These monitoring metrics are reported to taosKeeper, which writes them into the monitoring database, by default the log database, which can be modified in the taoskeeper configuration file. Below is a detailed introduction to these monitoring metrics.

adapter_requests table

adapter_requests records taosadapter monitoring data.

field	type	is_tag	comment
ts	TIMESTAMP		timestamp
total	INT UNSIGNED		total number of requests
query	INT UNSIGNED		number of query requests
write	INT UNSIGNED		number of write requests
other	INT UNSIGNED		number of other requests
in_process	INT UNSIGNED		number of requests in process
success	INT UNSIGNED		number of successful requests
fail	INT UNSIGNED		number of failed requests
query_success	INT UNSIGNED		number of successful query requests
query_fail	INT UNSIGNED		number of failed query requests
write_success	INT UNSIGNED		number of successful write requests
write_fail	INT UNSIGNED		number of failed write requests
other_success	INT UNSIGNED		number of successful other requests
other_fail	INT UNSIGNED		number of failed other requests
query_in_process	INT UNSIGNED		number of query requests in process
write_in_process	INT UNSIGNED		number of write requests in process
endpoint	VARCHAR		request endpoint
req_type	NCHAR	tag	request type: 0 for REST, 1 for WebSocket

Result Return Limit

taosAdapter controls the number of results returned through the parameter restfulRowLimit, -1 represents no limit, default is no limit.

This parameter controls the return of the following interfaces

• http://<fqdn>:6041/rest/sql
• http://<fqdn>:6041/prometheus/v1/remote_read/:db

Configure HTTP Return Codes

taosAdapter uses the parameter httpCodeServerError to set whether to return a non-200 HTTP status code when the C interface returns an error. When set to true, it will return different HTTP status codes based on the error code returned by C. See HTTP Response Codes for details.

Configure Automatic DB Creation for Schemaless Writes

Starting from version 3.0.4.0, taosAdapter provides the parameter smlAutoCreateDB to control whether to automatically create a DB when writing via the schemaless protocol. The default value is false, which does not automatically create a DB, and requires the user to manually create a DB before performing schemaless writes.

Troubleshooting

You can check the running status of taosAdapter with the command systemctl status taosadapter.

You can also adjust the detail level of taosAdapter log output by setting the --logLevel parameter or the environment variable TAOS_ADAPTER_LOG_LEVEL. Valid values include: panic, fatal, error, warn, warning, info, debug, and trace.

How to Migrate from Older Versions of TDengine to taosAdapter

In TDengine server version 2.2.x.x or earlier, the taosd process included an embedded HTTP service. As mentioned earlier, taosAdapter is a standalone software managed by systemd, having its own process. Moreover, there are some differences in configuration parameters and behaviors between the two, as shown in the table below:

#	embedded httpd	taosAdapter	comment
1	httpEnableRecordSql	--logLevel=debug
2	httpMaxThreads	n/a	taosAdapter automatically manages the thread pool, this parameter is not needed
3	telegrafUseFieldNum	Please refer to taosAdapter telegraf configuration methods
4	restfulRowLimit	restfulRowLimit	The embedded httpd defaults to outputting 10240 rows of data, with a maximum allowable value of 102400. taosAdapter also provides restfulRowLimit but does not impose a limit by default. You can configure it according to actual scenario needs.
5	httpDebugFlag	Not applicable	httpdDebugFlag does not affect taosAdapter
6	httpDBNameMandatory	Not applicable	taosAdapter requires the database name to be specified in the URL