Skip to main content

Data Backup and Restoration

You can back up the data in your TDengine cluster and restore it in the event that data is lost or damaged.

Data Backup and Restoration Using taosdump​

taosdump is an open-source tool that supports backing up data from a running TDengine cluster and restoring the backed-up data to the same or another running TDengine cluster. taosdump can back up the database as a logical data unit or back up data records within a specified time period in the database. When using taosdump, you can specify the directory path for data backup. If no directory path is specified, taosdump will default to backing up the data in the current directory.

Back Up Data with taosdump​

Below is an example of using taosdump to perform data backup.

taosdump -h localhost -P 6030 -D dbname -o /file/path

After executing the above command, taosdump will connect to the TDengine cluster at localhost:6030, query all data in the database dbname, and back up the data to /file/path.

When using taosdump, if the specified storage path already contains data files, taosdump will prompt the user and exit immediately to avoid data overwriting. This means the same storage path can only be used for one backup. If you see related prompts, please operate carefully to avoid accidental data loss.

Restore Data with taosdump​

To restore data files from a specified local file path to a running TDengine cluster, you can execute the taosdump command by specifying command-line parameters and the data file path. Below is an example code for taosdump performing data restoration.

taosdump -i /file/path -h localhost -P 6030

After executing the above command, taosdump will connect to the TDengine cluster at localhost:6030 and restore the data files from /file/path to the TDengine cluster.

Data Backup and Restoration in TDengine Enterprise​

TDengine Enterprise implements incremental backup and recovery of data by using data subscription. The backup and recovery functions of TDengine Enterprise include the following concepts:

  1. Incremental data backup: Based on TDengine's data subscription function, all data changes of the backup object (including: addition, modification, deletion, metadata change, etc.) are recorded to generate a backup file.
  2. Data recovery: Use the backup file generated by incremental data backup to restore the backup object to a specified point in time.
  3. Backup object: The object that the user backs up can be a database or a supertable.
  4. Backup plan: The user creates a periodic backup task for the backup object. The backup plan starts at a specified time point and periodically executes the backup task at intervals of the backup cycle. Each backup task generates a backup point .
  5. Backup point: Each time a backup task is executed, a set of backup files is generated. They correspond to a time point, called a backup point . The first backup point is called the initial backup point .
  6. Restore task: The user selects a backup point in the backup plan and creates a restore task. The restore task starts from the initial backup point and plays back the data changes in the backup file one by one until the specified backup point ends.

Incremental Backup Example​

Incremental backup process

  1. The user creates a backup plan to execute the backup task every 1 day starting from 2024-08-27 00:00:00 .
  2. The first backup task was executed at 2024-08-27 00:00:00, generating an initial backup point .
  3. After that, the backup task is executed every 1 day, and multiple backup points are generated .
  4. Users can select a backup point and create a restore task .
  5. The restore task starts from the initial backup point, applies the backup points one by one, and restores to the specified backup point.

Back Up Data in TDengine Enterprise​

  1. In a web browser, open the taosExplorer interface for TDengine. This interface is located on port 6060 on the hostname or IP address running TDengine.
  2. In the main menu on the left, click Management and open the Backup tab.
  3. Under Backup Plan, click Create New Backup to define your backup plan.
    1. Database: Select the database that you want to backup.
    2. Super Table: (Optional) Select the supertable that you want to backup. If you do not select a supertable, all data in the database is backed up.
    3. Next execution time: Enter the date and time when you want to perform the initial backup for this backup plan. If you specify a date and time in the past, the initial backup is performed immediately.
    4. Backup Cycle: Specify how often you want to perform incremental backups. The value of this field must be less than the value of WAL_RETENTION_PERIOD for the specified database.
    5. Retry times: Enter how many times you want to retry a backup task that has failed, provided that the specific failure might be resolved by retrying.
    6. Retry interval: Enter the delay in seconds between retry attempts.
    7. Directory: Enter the full path of the directory in which you want to store backup files.
    8. Backup file max size: Enter the maximum size of a single backup file. If the total size of your backup exceeds this number, the backup is split into multiple files.
    9. Compression level: Select fastest for the fastest performance but lowest compression ratio, best for the highest compression ratio but slowest performance, or balanced for a combination of performance and compression.
  4. Users can enable S3 dumping to upload backup files to the S3 storage service. To enable S3 dumping, the following information needs to be provided:
    1. Endpoint: The address of the S3 endpoint.
    2. Access Key ID: The access key ID for authentication.
    3. Secret Access Key: The secret access key for authentication.
    4. Bucket: The name of the target bucket.
    5. Region: The region where the bucket is located.
    6. Object Prefix: A prefix for backup file objects, similar to a directory path on S3.
    7. Backup Retention Period: The retention duration for local backups. All files older than current time - backup_retention_period must be uploaded to S3.
    8. Backup Retention Count: The number of local backups to retain. Only the latest backup_retention_size backup files are kept locally.
  5. Click Confirm to create the backup plan.

You can view your backup plans and modify, clone, or delete them using the buttons in the Operation columns. Click Refresh to update the status of your plans. Note that you must stop a backup plan before you can delete it. You can also click View in the Backup File column to view the backup record points and files created by each plan.

Restore Data in TDengine Enterprise​

  1. Locate the backup plan containing data that you want to restore and click View in the Backup File column.
  2. Determine the backup record point to which you want to restore and click the Restore icon in the Operation column.
  3. Select the backup file timestamp and target database and click Confirm.

Restore to Another Cluster with the taosx CLI​

In addition to creating a restore task in taosExplorer, you can copy the backup files to another cluster and restore them with the taosx command-line interface.

Backup Directory​

New task-level backups are stored directly in the task ID directory:

<backup-root>/
└── <task_id>/
    ├── .taosx-backup-layout
    ├── <topic>-<timestamp>-<vgroup>-<index>.z
    ├── <topic>-<timestamp>-<vgroup>-<index>.mz
    └── ...

New task-level backup writers create the hidden .taosx-backup-layout marker in the task ID directory. It declares whether backup artifacts are stored directly in the task directory (task_dir) or in the historical -1 subdirectory (legacy_job_dir). When the marker is present, taosx treats it as authoritative and does not probe the alternative layout. Preserve the marker and do not edit or recreate it manually.

The layout marker is not an HA lock. During an upgrade, do not let an older taosx version that does not recognize the marker and a newer version write the same backup task concurrently. Stop the backup task and confirm that the old process has exited before switching versions. After a task_dir marker has been created, do not resume the same task and backup root with an older version that does not recognize the marker, even after the newer process has exited. Otherwise, the old and new files are written to different directories. To roll back, use a new task ID or a new backup root instead of appending to the existing backup chain.

Only jobs that were actually split from a task use a <task_id>/<job_id> subdirectory. A task-level backup does not require a job directory or a job ID in the restore command. The restore process automatically recognizes task-level directories created by earlier versions.

To restore a backup from an actual job, copy the job directory that contains the .z and .mz files and point local: directly to the copied directory. The following procedure uses a task-level backup as an example.

Procedure​

  1. Stop the backup task on Cluster A, or make sure that the backup point to restore has been created.

  2. Copy the complete task directory to Cluster B and preserve its directory hierarchy and original backup file names. For example:

    rsync -a /backup/10/ user@cluster-b:/tmp/10/

    Copy the complete task directory, including the hidden .taosx-backup-layout file. rsync -a preserves hidden files; do not copy with a shell glob such as /backup/10/*, because it omits hidden files. An incremental restore requires the complete file chain from the initial backup point through the target point. Copy .mz files as well when they are present in the backup directory.

  3. Recursively verify that the task directory contains backup files on Cluster B:

    find /tmp/10 -type f \( -name '.taosx-backup-layout' -o -name '*.z' -o -name '*.mz' \) -print

  4. Specify the copied task directory directly after local:. The following example restores the backup for task 10 to the db01 database on Cluster B:

    taosx run \
      -f 'local:/tmp/10?to=now&watch=false' \
      -t 'taos:///db01' \
      -vv

    For a historical task-level directory without a layout marker, you can specify the layout explicitly to skip legacy compatibility detection:

    # Backup files are stored directly in /tmp/10
    taosx run \
      -f 'local:/tmp/10?to=now&watch=false&layout=task_dir' \
      -t 'taos:///db01' \
      -vv

    # Backup files are stored in /tmp/10/-1
    taosx run \
      -f 'local:/tmp/10?to=now&watch=false&layout=legacy_job_dir' \
      -t 'taos:///db01' \
      -vv

    The layout parameter applies only to task-level backups. When a layout marker is present, an explicit layout must match it or the restore fails. A read-only restore does not create a layout marker.

  5. Check the taosx logs for the number of files and rows processed, and query the target cluster to verify the restored data. A zero exit code alone does not prove that data was restored.

If neither a layout marker nor an explicit layout is present, taosx performs legacy compatibility detection and logs the warning backup layout marker is missing; using legacy layout detection. Only in this case, if backup files exist both directly in the task directory and in its historical -1 subdirectory, does the restore report a directory-layout conflict. Verify both locations and consolidate the complete chain in a separate restore directory before retrying. Do not overwrite files with the same name when their contents differ.

Troubleshooting​

Port Access Exception​

A port access exception is indicated by the following error:

Error: tmq to td task exec error

Caused by:
    [0x000B] Unable to establish connection

If you encounter this error, check whether the data source FQDN is connected and whether port 6030 is listening and accessible.

Connection Issues​

A connection issue is indicated by the task failing to start and reporting the following error:

Error: tmq to td task exec error

Caused by:
    0: WebSocket internal error: IO error: failed to lookup address information: Temporary failure in name resolution
    1: IO error: failed to lookup address information: Temporary failure in name resolution
    2: failed to lookup address information: Temporary failure in name resolution

The following are some possible errors for WebSocket connections:

  • "Temporary failure in name resolution": DNS resolution error. Check whether the specified IP address or FQDN can be accessed normally.
  • "IO error: Connection refused (os error 111)": Port access failed. Check whether the port is configured correctly and is enabled and accessible.
  • "IO error: received corrupt message": Message parsing failed. This may be because SSL was enabled using the WSS method, but the source port is not supported.
  • "HTTP error: *": Confirm that you are connecting to the correct taosAdapter port and that your LSB/Nginx/Proxy has been configured correctly.
  • "WebSocket protocol error: Handshake not finished": WebSocket connection error. This is typically caused by an incorrectly configured port.

WAL Configuration​

A WAL configuration issue is indicated by the task failing to start and reporting the following error:

Error: tmq to td task exec error

Caused by:
    [0x038C] WAL retention period is zero

To resolve this error, modify the WAL retention period for the affected database:

ALTER DATABASE test WAL_RETENTION_PERIOD 3600;