Skip to main content

Data Security

In addition to traditional user and permission management, TDengine also offers other security strategies such as IP whitelisting, audit logs, data encryption, etc., which are unique features of TDengine Enterprise. The whitelisting feature was first released in v3.2.0.0, audit logs in v3.1.1.0, and database encryption in v3.3.0.0. It is recommended to use the latest version. You can also enable secure delete through database or statement options (SECURE_DELETE). For audit capabilities, see Audit and Compliance. This section covers IP whitelisting, secure delete, and storage encryption.

Database DDL options such as ENCRYPT_ALGORITHM, IS_AUDIT, SECURE_DELETE, and SECURITY_LEVEL are documented under Databases. For SECURITY_LEVEL (MAC), see Mandatory Access Control (MAC). For IS_AUDIT constraints, see Audit and Compliance. For SECURE_DELETE, see Secure Delete below.

IP Whitelisting​

IP whitelisting is a network security technology that allows IT administrators to control "who" can access the system and resources, enhancing the security of database access and preventing external malicious attacks. IP whitelisting works by creating a list of trusted IP addresses, assigning them as unique identifiers to users, and only allowing these IP addresses to access the target server. Please note that user permissions and IP whitelisting are managed separately. Below are the specific methods for configuring IP whitelisting.

The SQL to add an IP whitelist is as follows.

create user test pass password [sysinfo value] [host host_name1[,host_name2]] 
alter user test add host host_name1

The SQL to query the IP whitelist is as follows.

SELECT TEST, ALLOWED_HOST FROM INS_USERS;
SHOW USERS;

The command to delete from the IP whitelist is as follows.

ALTER USER TEST DROP HOST HOST_NAME1

Notes

  • Both the open-source and enterprise versions can add successfully and can be queried, but the open-source version does not impose any restrictions on IPs.
  • create user u_write pass 'taosdata1' host 'iprange1','iprange2', multiple ipranges can be added at once, the server will deduplicate, the logic for deduplication requires the ipranges to be exactly the same
  • By default, 127.0.0.1 is added to the whitelist and can be queried in the whitelist
  • The IP set of cluster nodes is automatically added to the whitelist, but cannot be queried.
  • When taosadaper and taosd are not on the same machine, the taosadaper IP needs to be manually added to the taosd whitelist
  • In a cluster situation, all nodes should have enableWhiteList set the same, either all false or all true, otherwise the cluster cannot start
  • Whitelist changes take effect within 1s, no more than 2s, each change has a slight impact on transmission performance (one more judgment, which can be ignored), after the change, the impact is negligible, and there is no impact on the cluster during the change process, nor on the clients currently accessing (assuming their IPs are included in the whitelist)
  • If adding two ip ranges, 192.168.1.1/16 (suppose A), 192.168.1.1/24 (suppose B), strictly speaking, A includes B, but considering the situation too complicated, A and B will not be merged
  • When deleting, it must match exactly. That is, if added as 192.168.1.1/24, it must also be deleted as 192.168.1.1/24
  • Only root has the permission to add or delete ip white list for other users
  • Compatible with previous versions, but does not support rolling back from the current version to a previous version
  • x.x.x.x/32 and x.x.x.x are considered the same iprange, displayed as x.x.x.x
  • If the client gets 0.0.0.0/0, it means the whitelist is not enabled
  • If the whitelist changes, the client will detect it in the heartbeat.
  • For one user, the maximum number of IPs that can be added is 2048

Secure Delete​

The database option SECURE_DELETE (0 / 1, default 0) controls whether the delete path, in addition to writing a delete mark, physically overwrites on-disk data blocks. For DDL syntax, see Databases · SECURE_DELETE. A single delete can also append the SECURE_DELETE keyword; see Data Deletion.

  • Off (0): Only a delete mark is written. Queries no longer return the deleted data, but the corresponding file blocks may remain on disk until later compaction or reclamation.
  • On (1): In addition to the delete mark, data blocks that hit the (table, time range) in on-disk DATA / STT files are overwritten at the file level (secure erase), reducing the risk of reading deleted content directly from the filesystem.

Behavior notes:

  • Secure delete takes effect if any of the following is set: database-level SECURE_DELETE=1, a secure-delete flag in table/supertable metadata, or statement-level DELETE ... SECURE_DELETE (combined with bitwise OR in the implementation).
  • Physical overwrite runs after the delete mark is written. If overwrite fails, the server logs the failure; query semantics still follow the delete mark (deleted data does not become visible again because overwrite failed).
  • The current implementation targets the newer TSDB file format; older-format files skip file-level overwrite and rely on later compaction paths for reclamation.
  • In multi-replica scenarios, file-level overwrite runs on the Raft leader; followers apply logical deletes via WAL replay and do not automatically repeat the same physical overwrite.
  • WAL may still retain original write records from before the delete until WAL trimming after a checkpoint. OS page cache and SSD wear leveling may also leave old content briefly visible on physical media. This capability is not hardware-level Secure Erase / Sanitize, and is not equivalent to “static encryption plus key destruction.”
  • Enabling secure delete increases I/O and latency on the delete path; weigh residual-data requirements against performance cost.
  • It complements TDE: TDE reduces the risk of reading static files directly; secure delete focuses on overwriting residual blocks after deletion. Neither constitutes a claim of compliance with a specific regulation or certification.
  • The global parameter secureEraseMode (default 0) controls how fully overwritable blocks are filled: 0 is zero-fill, 1 is random bytes; partially overlapping blocks are always zero-filled for in-place write-back. See the taosd configuration reference.

Examples:

CREATE DATABASE db SECURE_DELETE 1;
ALTER DATABASE db SECURE_DELETE 1;
DELETE FROM meters WHERE ts < '2021-10-01 10:40:00.100' SECURE_DELETE;

Storage Security​

TDengine supports Transparent Data Encryption (TDE), which encrypts static data files to prevent potential attackers from bypassing the database and directly reading sensitive information from the file system. The database access program is completely unaware, and applications do not need to make any modifications or compilations to work with encrypted databases. Storage security features support encryption algorithms such as SM4 and AES, adopt a hierarchical key management mechanism, and provide comprehensive key backup and recovery capabilities.

Key Hierarchy​

TDengine adopts a hierarchical key management system, including the following key types:

  • SVR_KEY (Server Master Key): Used to encrypt database master keys and system-level information, bound to machine hardware to prevent cross-machine migration
  • DB_KEY (Database Master Key): Used to encrypt various derived keys
  • CFG_KEY (Configuration Encryption Key): Dedicated to encrypting configuration files, cannot be changed once generated
  • META_KEY (Metadata Encryption Key): Used to encrypt metadata files, cannot be changed once generated
  • DATA_KEY (Time-Series Data Encryption Key): Used to encrypt time-series data files and related logs, cannot be changed once generated

All keys use machine code binding technology. When data files are copied to another machine, the change in machine code prevents access to the keys, naturally preventing access to the data files. After encryption, the data compression ratio remains unchanged, and write and query performance decrease slightly.

Note: The storage security feature requires obtaining machine code, which may not be available in some virtualized environments (such as certain container environments).

Generate Keys​

Use the taosk tool to generate keys. Basic syntax:

taosk -c /etc/taos \
  --set-cfg-algorithm sm4 \
  --set-meta-algorithm sm4 \
  --encrypt-server [svr_key] \
  --encrypt-database [db_key] \
  --encrypt-config \
  --encrypt-metadata \
  --encrypt-data [data_key]

Parameter descriptions:

  • -c: Specify configuration file path, default /etc/taos
  • -d: Specify data directory (dataDir), default from configuration
  • --set-cfg-algorithm: Set configuration file encryption algorithm (sm4 or aes), default sm4
  • --set-meta-algorithm: Set metadata encryption algorithm (sm4 or aes), default sm4
  • --encrypt-server: Enable server encryption, optionally specify SVR_KEY, auto-generate if not specified
  • --encrypt-database: Enable database encryption, optionally specify DB_KEY, auto-generate if not specified
  • --encrypt-config: Enable configuration file encryption, auto-generate CFG_KEY
  • --encrypt-metadata: Enable metadata encryption, auto-generate META_KEY
  • --encrypt-data: Enable data file encryption, optionally specify DATA_KEY, auto-generate if not specified

Examples:

# Generate all keys using default SM4 algorithm
taosk -c /etc/taos \
  --encrypt-server \
  --encrypt-database \
  --encrypt-config \
  --encrypt-metadata \
  --encrypt-data

# Specify keys and use different algorithms
taosk -c /etc/taos \
  --set-cfg-algorithm aes \
  --set-meta-algorithm sm4 \
  --encrypt-server mysvr123 \
  --encrypt-database mydb4567 \
  --encrypt-config \
  --encrypt-metadata \
  --encrypt-data oldkey123

After keys are generated, they will be saved in the following locations:

  • {dataDir}/dnode/config/master.bin: Stores SVR_KEY and DB_KEY
  • {dataDir}/dnode/config/derived.bin: Stores CFG_KEY, META_KEY, and DATA_KEY

View Encrypted Configuration Files​

Use the taosk tool to view encrypted configuration file content:

taosk -d /var/lib/taos --view-config /path/to/encrypted_config.json

This command automatically loads keys from the data directory, decrypts and displays the configuration file content.

Edit Encrypted Configuration Files​

Use the taosk tool to directly edit encrypted configuration files:

taosk -d /var/lib/taos --edit-file /path/to/encrypted_config.json

This command will:

  1. Load CFG_KEY from the data directory
  2. Decrypt the configuration file to a temporary file (permissions 0600)
  3. Open the file with the system editor ($EDITOR or vi)
  4. Detect file changes via SHA-256 hash comparison
  5. If modified, automatically re-encrypt and write back to the original file
  6. Clean up temporary files

Notes:

  • Keys containing CFG_KEY must be generated first (using --encrypt-config option)
  • Specify editor via EDITOR environment variable, e.g., EDITOR=nano taosk -d /var/lib/taos --edit-file ...
  • If you exit the editor without saving, the file will not be modified

View Encryption Algorithms​

Users can view all built-in available encryption algorithms.

show encrypt_algorithms;
id      |          algorithm_id          |              name              |              desc              |              type              |             source             |         ossl_algr_name         |
1 | SM4-CBC                        | SM4                            | SM4 symmetric encryption       | Symmetric Ciphers CBC mode     | build-in                       | SM4-CBC:SM4                    |
2 | AES-128-CBC                    | AES                            | AES symmetric encryption       | Symmetric Ciphers CBC mode     | build-in                       | AES-128-CBC                    |
  • id: Numeric identifier of the algorithm; built-in algorithms start from 1, custom algorithms start from 101
  • algorithm_id: Global unique identifier of the algorithm
  • name: Algorithm name
  • desc: Description of the algorithm
  • type: Algorithm type, including: Symmetric Ciphers CBC mode - symmetric encryption algorithm in CBC mode, used for database encryption; Asymmetric Ciphers - asymmetric encryption algorithm; Digests - hash algorithm
  • source: Algorithm source, including: built-in - built-in algorithm; customized - user-defined algorithm
  • ossl_algr_name: Algorithm name in OpenSSL; for built-in algorithms, it's the name in the default provider, refer to OSSL_PROVIDER-default for custom algorithms, it's user-defined in the program

Add Customized Algorithms​

Users can add their own custom algorithms.

create encrypt_algr 'vigenere' algr_name 'vigenere' desc 'my custom algr' algr_type 'Symmetric_Ciphers_CBC_mode' ossl_algr_name 'vigenere';

For user-defined algorithms, users need to develop an SO library according to the interface. When taosd starts, it will load this SO library, and after the SO library is loaded, the user-defined algorithms can be used. In this SO library, users can include multiple algorithms, each with its own naming, specified through the ossl_algr_name field in create encrypt_algr. The custom algorithm interface adopts the OpenSSL implementation and follows the OpenSSL interface definition. For the OpenSSL interface definition, refer to OpenSSL provider. The parameter encryptExtDir specifies the path to the custom algorithm library SO file. Currently, only a single file can be loaded.

Delete Customized Algorithms​

Users can delete their own custom algorithms.

drop encrypt_algr 'vigenere';

Before deleting a custom algorithm, you must ensure that the algorithm is not in use. For example, databases using that algorithm must be deleted in advance.

Built-in algorithms (whose source is build-in, such as SM4-CBC, AES-128-CBC, etc.) cannot be deleted; attempting to delete them returns an error.

Create Encrypted Database​

TDengine supports specifying encryption algorithms when creating databases, SQL as follows:

CREATE DATABASE [IF NOT EXISTS] db_name [database_options]
database_options:
  database_option ...
database_option: {
  ENCRYPT_ALGORITHM {'none' | 'SM4-CBC' | 'AES-128-CBC'}
}

The main parameters are explained as follows.

  • encrypt_algorithm: Specifies the encryption algorithm used for the data. The default is none, meaning no encryption is used. If you want to set up encrypted data, you need to specify the algorithm_id from show encrypt_algorithms, and the type must be a symmetric cipher in CBC mode.

Examples:

-- Create database using SM4 encryption
CREATE DATABASE db1 ENCRYPT_ALGORITHM 'SM4-CBC';

-- Create database using AES encryption
CREATE DATABASE db2 ENCRYPT_ALGORITHM 'AES-128-CBC';

-- Create unencrypted database
CREATE DATABASE db3;

Notes:

  • The ENCRYPT_ALGORITHM of a database cannot be modified after creation
  • Before creating an encrypted database, you must first generate keys containing DATA_KEY using taosk

View Encryption Status​

View System Encryption Status​

View overall encryption status through system tables:

SELECT * FROM information_schema.ins_encrypt_status;
         encrypt_scope          |           algorithm            |       status       |
=======================================================================================
 config                         | AES-128-CBC                    | enabled            |
 metadata                       | AES-128-CBC                    | enabled            |
 data                           | SM4-CBC:SM4                    | enabled            |

Field descriptions:

  • encrypt_scope: Encryption scope (config, metadata, data)
  • algorithm: Encryption algorithm used
  • status: Encryption status (enabled or disabled)

View Database Encryption Configuration​

View encryption algorithms for each database through system tables:

SELECT name,`encrypt_algorithm` FROM information_schema.ins_databases;
              name              | encrypt_algorithm  |
======================================================
 information_schema             | NULL               |
 performance_schema             | NULL               |
 db2                            | AES-128-CBC        |
 db1                            | SM4-CBC            |

Update Keys​

You can update SVR_KEY and DB_KEY through the taosk tool or SQL commands (other keys cannot be changed once generated).

Update Using taosk​

# Stop taosd
systemctl stop taosd

# Update keys
taosk -c /etc/taos --update-svrkey new_svr_key --update-dbkey new_db_key

# Start taosd
systemctl start taosd

Update Using SQL​

While taosd is running, you can update keys through SQL (requires administrator privileges):

-- Update SVR_KEY
ALTER SYSTEM SET SVR_KEY 'new_svr_key';

-- Update DB_KEY
ALTER SYSTEM SET DB_KEY 'new_db_key';

Key Backup and Recovery​

Backup Keys​

Use taosk to create portable backups (without machine code binding, can be restored on other machines):

taosk -c /etc/taos --backup --svr-key your_svr_key

The backup file will be generated in the {dataDir}/dnode/config/ directory with the filename format master.bin.backup.{timestamp}.

Note: Backup requires providing the correct SVR_KEY for verification.

Restore Keys​

Restore keys from backup on a new machine:

taosk -c /etc/taos \
  --restore \
  --machine-code /path/to/backup_file \
  --svr-key your_svr_key

The restore operation will bind the keys to the current machine's machine code.

Key Expiration Policy​

You can set key expiration time and policy through SQL (requires administrator privileges):

ALTER SYSTEM SET KEY_EXPIRATION 90 DAYS STRATEGY 'ALARM';

Policy options:

  • ALARM: Outputs alarm information in logs when keys expire

Configuration File Behavior Changes​

After enabling storage security, TDengine's configuration management changes as follows:

  1. Configuration only effective on first startup: After the system's initial startup, subsequent modifications to the taos.cfg file will not take effect
  2. Modify configuration through SQL: All configuration modifications must be executed through SQL commands, requiring administrator privileges

Configuration modification example:

ALTER DNODE 1 'debugFlag' '143';

Transparent Encryption Scope​

After enabling storage security, TDengine will transparently encrypt the following files:

  1. Configuration file encryption (requires CFG_KEY):

    • dnode.info, dnode.json
    • mnode.json, raft_config.json, raft_store.json
    • vnodes.json, vnode.json, etc.

  2. Metadata file encryption (requires META_KEY):

    • mnode SDB
    • snode checkpoint files

  3. Data file encryption (requires DATA_KEY):

    • TSDB data files
    • WAL write-ahead log files
    • STT files
    • TDB, BSE and other index files

All encrypted configuration files will contain a plaintext identifier header ("tdEncrypt") at the beginning to mark files as encrypted and avoid duplicate encryption.

Version Compatibility​

  • Upgrading from versions that do not support storage security to new versions can run normally
  • Encrypted databases from historical versions can be made compatible by specifying DATA_KEY
  • After enabling storage security, cannot rollback to historical versions that do not support storage security

Encrypt Stored User Passwords​

By default, user passwords are stored as MD5 hashes. Set encryptPassAlgorithm to sm4 to additionally encrypt stored password data; currently only sm4 is supported. Generate and configure the required keys before enabling this parameter. For parameter availability and modification rules, see taosd.