TCollector writing
TCollector is part of openTSDB and collects client computer's logs to send to the database.
You can write the data collected by TCollector to TDengine by simply changing the configuration of TCollector to point to the domain name (or IP address) and corresponding port of the server running taosAdapter. It can take full advantage of TDengine's efficient storage query performance and clustering capability for time-series data.
Prerequisites
To write data to the TDengine via TCollector requires the following preparations.
- The TDengine cluster has been deployed and is working properly
- taosAdapter is installed and running properly. Please refer to the taosAdapter manual for details.
- TCollector has been installed. Please refer to official documentation for TCollector installation
Configuration steps
Configuring taosAdapter
To configure taosAdapter to receive TCollector data.
- Enable the configuration item in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
The taosAdapter writes to the database with the default name tcollector. You can also modify the taosAdapter configuration file dbs entry to specify a different name. Fill in the actual user and password for TDengine. After changing the configuration file, you need to restart the taosAdapter.
- You can also enable taosAdapter to receive tcollector data by using the taosAdapter command-line parameters or setting environment variables.
Configuring TCollector
To use TCollector, you need to download its source code. Its configuration items are in its source code. Note: TCollector differs significantly from version to version, so here is an example of the latest code for the current master branch (git commit: 37ae920).
Modify the contents of the collectors/etc/config.py and tcollector.py files. Change the address of the OpenTSDB host to the domain name or IP address of the server where taosAdapter is deployed, and change the port to the port on which taosAdapter supports TCollector (default is 6049).
Example of git diff output of source code changes.
index e7e7a1c..ec3e23c 100644
--- a/collectors/etc/config.py
+++ b/collectors/etc/config.py
@@ -59,13 +59,13 @@ def get_defaults():
'http_password': False,
'reconnectinterval': 0,
'http_username': False,
- 'port': 4242,
+ 'port': 6049,
'pidfile': '/var/run/tcollector.pid',
'http': False,
'http_api_path': "api/put",
'tags': [],
'remove_inactive_collectors': False,
- 'host': '',
+ 'host': '127.0.0.1',
'logfile': '/var/log/tcollector.log',
'cdir': default_cdir,
'ssl': False,
diff --git a/tcollector.py b/tcollector.py
index 21f9b23..4c71ba2 100755
--- a/tcollector.py
+++ b/tcollector.py
@@ -64,7 +64,7 @@ ALIVE = True
# exceptions, something is not right and tcollector will shutdown.
# Hopefully some kind of supervising daemon will then restart it.
MAX_UNCAUGHT_EXCEPTIONS = 100
-DEFAULT_PORT = 4242
+DEFAULT_PORT = 6049
MAX_REASONABLE_TIMESTAMP = 2209212000 # Good until Tue 3 Jan 14:00:00 GMT 2040
# How long to wait for datapoints before assuming
# a collector is dead and restarting it
@@ -943,13 +943,13 @@ def parse_cmdline(argv):
'http_password': False,
'reconnectinterval': 0,
'http_username': False,
- 'port': 4242,
+ 'port': 6049,
'pidfile': '/var/run/tcollector.pid',
'http': False,
'http_api_path': "api/put",
'tags': [],
'remove_inactive_collectors': False,
- 'host': '',
+ 'host': '127.0.0.1',
'logfile': '/var/log/tcollector.log',
'cdir': default_cdir,
'ssl': False,
Verification method
Restart taosAdapter:
sudo systemctl restart taosadapter
Run sudo ./tcollector.py:
Wait for a few seconds and then use the TDengine CLI to query whether the corresponding database has been created and data are written in TDengine.
taos> show databases;
name | created_time | ntables | vgroups | replica | quorum | days | keep | cache(MB) | blocks | minrows | maxrows | wallevel | fsync | comp | cachelast | precision | update | status |
====================================================================================================================================================================================================================================================================================
tcollector | 2022-04-20 12:44:49.604 | 88 | 1 | 1 | 1 | 10 | 3650 | 16 | 6 | 100 | 4096 | 1 | 3000 | 2 | 0 | ns | 2 | ready |
log | 2022-04-20 07:19:50.260 | 11 | 1 | 1 | 1 | 10 | 3650 | 16 | 6 | 100 | 4096 | 1 | 3000 | 2 | 0 | ms | 0 | ready |
Query OK, 2 row(s) in set (0.002679s)
taos> use tcollector;
Database changed.
taos> show stables;
name | created_time | columns | tags | tables |
============================================================================================
proc.meminfo.hugepages_rsvd | 2022-04-20 12:44:53.945 | 2 | 1 | 1 |
proc.meminfo.directmap1g | 2022-04-20 12:44:54.110 | 2 | 1 | 1 |
proc.meminfo.vmallocchunk | 2022-04-20 12:44:53.724 | 2 | 1 | 1 |
proc.meminfo.hugepagesize | 2022-04-20 12:44:54.004 | 2 | 1 | 1 |
tcollector.reader.lines_dro... | 2022-04-20 12:44:49.675 | 2 | 1 | 1 |
proc.meminfo.sunreclaim | 2022-04-20 12:44:53.437 | 2 | 1 | 1 |
proc.stat.ctxt | 2022-04-20 12:44:55.363 | 2 | 1 | 1 |
proc.meminfo.swaptotal | 2022-04-20 12:44:53.158 | 2 | 1 | 1 |
proc.uptime.total | 2022-04-20 12:44:52.813 | 2 | 1 | 1 |
tcollector.collector.lines_... | 2022-04-20 12:44:49.895 | 2 | 2 | 51 |
proc.meminfo.vmallocused | 2022-04-20 12:44:53.704 | 2 | 1 | 1 |
proc.meminfo.memavailable | 2022-04-20 12:44:52.939 | 2 | 1 | 1 |
sys.numa.foreign_allocs | 2022-04-20 12:44:57.929 | 2 | 2 | 1 |
proc.meminfo.committed_as | 2022-04-20 12:44:53.639 | 2 | 1 | 1 |
proc.vmstat.pswpin | 2022-04-20 12:44:54.177 | 2 | 1 | 1 |
proc.meminfo.cmafree | 2022-04-20 12:44:53.865 | 2 | 1 | 1 |
proc.meminfo.mapped | 2022-04-20 12:44:53.349 | 2 | 1 | 1 |
proc.vmstat.pgmajfault | 2022-04-20 12:44:54.251 | 2 | 1 | 1 |
...
