Skip to main content

TDengine Monitoring

After TDengine is started, it automatically writes monitoring data including CPU, memory and disk usage, bandwidth, number of requests, disk I/O speed, slow queries, into a designated database at a predefined interval through taosKeeper. Additionally, some important system operations, like logon, create user, drop database, and alerts and warnings generated in TDengine are written into the log database too. A system operator can view the data in log database from TDengine CLI or from a web console.

The collection of the monitoring information is enabled by default, but can be disabled by parameter monitor in the configuration file.

TDinsight

TDinsight is a complete solution which uses the monitoring database log mentioned previously, and Grafana, to monitor a TDengine cluster.

A script TDinsight.sh is provided to deploy TDinsight automatically.

Download TDinsight.sh with the below command:

wget https://github.com/taosdata/grafanaplugin/raw/master/dashboards/TDinsight.sh
chmod +x TDinsight.sh

Prepare:

  1. TDengine Server

    • The URL of REST service: for example http://localhost:6041 if TDengine is deployed locally
    • User name and password
  2. Grafana Alert Notification

You can use below command to setup Grafana alert notification.

An existing Grafana Notification Channel can be specified with parameter -E, the notifier uid of the channel can be obtained by curl -u admin:admin localhost:3000/api/alert-notifications |jq

 ```bash
./TDinsight.sh -a http://localhost:6041 -u root -p taosdata -E <notifier uid>
```

Launch TDinsight.sh with the command above and restart Grafana, then open Dashboard http://localhost:3000/d/tdinsight.

log database

The data of tdinsight dashboard is stored in log database (default. You can change it in taoskeeper's config file. For more infrmation, please reference to taoskeeper document). The taoskeeper will create log database on taoskeeper startup.

cluster_info table

cluster_info table contains cluster information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
first_epVARCHARfirst ep of cluster
first_ep_dnode_idINTdnode id or first_ep
versionVARCHARtdengine version. such as: 3.0.4.0
master_uptimeFLOATdays of master's uptime
monitor_intervalINTmonitor interval in second
dbs_totalINTtotal number of databases in cluster
tbs_totalBIGINTtotal number of tables in cluster
stbs_totalINTtotal number of stables in cluster
dnodes_totalINTtotal number of dnodes in cluster
dnodes_aliveINTtotal number of dnodes in ready state
mnodes_totalINTtotal number of mnodes in cluster
mnodes_aliveINTtotal number of mnodes in ready state
vgroups_totalINTtotal number of vgroups in cluster
vgroups_aliveINTtotal number of vgroups in ready state
vnodes_totalINTtotal number of vnode in cluster
vnodes_aliveINTtotal number of vnode in ready state
connections_totalINTtotal number of connections to cluster
topics_totalINTtotal number of topics in cluster
streams_totalINTtotal number of streams in cluster
protocolINTprotocol version
cluster_idNCHARTAGcluster id

d_info table

d_info table contains dnodes information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
statusVARCHARdnode status
dnode_epNCHARTAGdnode endpoint
cluster_idNCHARTAGcluster id

m_info table

m_info table contains mnode information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
roleVARCHARthe role of mnode. leader or follower
mnode_idINTTAGmaster node id
mnode_epNCHARTAGmaster node endpoint
cluster_idNCHARTAGcluster id

dnodes_info table

dnodes_info table contains dnodes information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
uptimeFLOATdnode uptime in days
cpu_engineFLOATcpu usage of tdengine. read from /proc/<taosd_pid>/stat
cpu_systemFLOATcpu usage of server. read from /proc/stat
cpu_coresFLOATcpu cores of server
mem_engineINTmemory usage of tdengine. read from /proc/<taosd_pid>/status
mem_systemINTavailable memory on the server in KB
mem_totalINTtotal memory of server in KB
disk_engineINT
disk_usedBIGINTusage of data dir in bytes
disk_totalBIGINTthe capacity of data dir in bytes
net_inFLOATnetwork throughput rate in byte/s. read from /proc/net/dev
net_outFLOATnetwork throughput rate in byte/s. read from /proc/net/dev
io_readFLOATio throughput rate in byte/s. read from /proc/<taosd_pid>/io
io_writeFLOATio throughput rate in byte/s. read from /proc/<taosd_pid>/io
io_read_diskFLOATio throughput rate of disk in byte/s. read from /proc/<taosd_pid>/io
io_write_diskFLOATio throughput rate of disk in byte/s. read from /proc/<taosd_pid>/io
req_selectINTnumber of select queries received per dnode
req_select_rateFLOATnumber of select queries received per dnode divided by monitor interval.
req_insertINTnumber of insert queries received per dnode
req_insert_successINTnumber of successfully insert queries received per dnode
req_insert_rateFLOATnumber of insert queries received per dnode divided by monitor interval
req_insert_batchINTnumber of batch insertions
req_insert_batch_successINTnumber of successful batch insertions
req_insert_batch_rateFLOATnumber of batch insertions divided by monitor interval
errorsINTdnode errors
vnodes_numINTnumber of vnodes per dnode
mastersINTnumber of master vnodes
has_mnodeINTif the dnode has mnode
has_qnodeINTif the dnode has qnode
has_snodeINTif the dnode has snode
has_bnodeINTif the dnode has bnode
dnode_idINTTAGdnode id
dnode_epNCHARTAGdnode endpoint
cluster_idNCHARTAGcluster id

data_dir table

data_dir table contains data directory information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
nameNCHARdata directory. default is /var/lib/taos
levelINTlevel for multi-level storage
availBIGINTavailable space for data directory in bytes
usedBIGINTused space for data directory in bytes
totalBIGINTtotal space for data directory in bytes
dnode_idINTTAGdnode id
dnode_epNCHARTAGdnode endpoint
cluster_idNCHARTAGcluster id

log_dir table

log_dir table contains log directory information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
nameNCHARlog directory. default is /var/log/taos/
availBIGINTavailable space for log directory in bytes
usedBIGINTused space for data directory in bytes
totalBIGINTtotal space for data directory in bytes
dnode_idINTTAGdnode id
dnode_epNCHARTAGdnode endpoint
cluster_idNCHARTAGcluster id

temp_dir table

temp_dir table contains temp dir information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
nameNCHARtemp directory. default is /tmp/
availBIGINTavailable space for temp directory in bytes
usedBIGINTused space for temp directory in bytes
totalBIGINTtotal space for temp directory in bytes
dnode_idINTTAGdnode id
dnode_epNCHARTAGdnode endpoint
cluster_idNCHARTAGcluster id

vgroups_info table

vgroups_info table contains vgroups information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
vgroup_idINTvgroup id
database_nameVARCHARdatabase for the vgroup
tables_numBIGINTnumber of tables per vgroup
statusVARCHARstatus
dnode_idINTTAGdnode id
dnode_epNCHARTAGdnode endpoint
cluster_idNCHARTAGcluster id

vnodes_role table

vnodes_role table contains vnode role information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
vnode_roleVARCHARrole. leader or follower
dnode_idINTTAGdnode id
dnode_epNCHARTAGdnode endpoint
cluster_idNCHARTAGcluster id

log_summary table

log_summary table contains log summary information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
errorINTerror count
infoINTinfo count
debugINTdebug count
traceINTtrace count
dnode_idINTTAGdnode id
dnode_epNCHARTAGdnode endpoint
cluster_idNCHARTAGcluster id

grants_info table

grants_info table contains grants information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
expire_timeBIGINTtime until grants expire in seconds
timeseries_usedBIGINTtimeseries used
timeseries_totalBIGINTtotal timeseries
dnode_idINTTAGdnode id
dnode_epNCHARTAGdnode endpoint
cluster_idNCHARTAGcluster id

keeper_monitor table

keeper_monitor table contains keeper monitor information records.

fieldtypeis_tagcomment
tsTIMESTAMPtimestamp
cpuFLOATcpu usage
memFLOATmemory usage
identifyNCHARTAG

taosadapter_restful_http_request_total table

taosadapter_restful_http_request_total table contains taosadapter rest request information record. The timestamp column of this table is _ts.

fieldtypeis_tagcomment
_tsTIMESTAMPtimestamp
gaugeDOUBLEmetric value
client_ipNCHARTAGclient ip
endpointNCHARTAGtaosadpater endpoint
request_methodNCHARTAGrequest method
request_uriNCHARTAGrequest uri
status_codeNCHARTAGstatus code

taosadapter_restful_http_request_fail table

taosadapter_restful_http_request_fail table contains taosadapter failed rest request information record. The timestamp column of this table is _ts.

fieldtypeis_tagcomment
_tsTIMESTAMPtimestamp
gaugeDOUBLEmetric value
client_ipNCHARTAGclient ip
endpointNCHARTAGtaosadpater endpoint
request_methodNCHARTAGrequest method
request_uriNCHARTAGrequest uri
status_codeNCHARTAGstatus code

taosadapter_restful_http_request_in_flight table

taosadapter_restful_http_request_in_flight table contains taosadapter rest request information record in real time. The timestamp column of this table is _ts.

fieldtypeis_tagcomment
_tsTIMESTAMPtimestamp
gaugeDOUBLEmetric value
endpointNCHARTAGtaosadpater endpoint

taosadapter_restful_http_request_summary_milliseconds table

taosadapter_restful_http_request_summary_milliseconds table contains the summary or rest information record. The timestamp column of this table is _ts.

fieldtypeis_tagcomment
_tsTIMESTAMPtimestamp
countDOUBLE
sumDOUBLE
0.5DOUBLE
0.9DOUBLE
0.99DOUBLE
0.1DOUBLE
0.2DOUBLE
endpointNCHARTAGtaosadpater endpoint
request_methodNCHARTAGrequest method
request_uriNCHARTAGrequest uri

taosadapter_system_mem_percent table

taosadapter_system_mem_percent table contains taosadapter memory usage information. The timestamp of this table is _ts.

fieldtypeis_tagcomment
_tsTIMESTAMPtimestamp
gaugeDOUBLEmetric value
endpointNCHARTAGtaosadpater endpoint

taosadapter_system_cpu_percent table

taosadapter_system_cpu_percent table contains taosadapter cup usage information. The timestamp of this table is _ts.

fieldtypeis_tagcomment
_tsTIMESTAMPtimestamp
gaugeDOUBLEmertic value
endpointNCHARTAGtaosadpater endpoint