MDEV-14992 BACKUP SERVER

[dr-m]

The following SQL statements will be introduced:

BACKUP SERVER TO '/path/to/directory';
BACKUP SERVER TO '/path/to/directory' 1 CONCURRENT;
BACKUP SERVER WITH 'command';
BACKUP SERVER WITH 1 CONCURRENT 'command';

In place of the 1, any positive number of threads may be specified. For the first variant, '/path/to' must exist and '/path/to/directory' must not exist; that is where the backup will be written to.

For the second variant, 'command' must be the name of a script or command that will be executed in a child process. The standard input of that command will be in a format that is compatible with GNU tar --format=oldgnu (and also BSD tar variants that are also part of Microsoft Windows and Apple macOS). The command is expected to optionally compress and encrypt the stream and redirect it to a file on a local or a remote server. The BACKUP SERVER WITH will append an additional argument, a positive base-ten number in ASCII, starting with 1, to identify the current thread. In this way, each concurrent stream can write a separate file.

The backup or the first stream will contain a file backup.cnf, which includes parameters needed for restoring the backup. Currently, these are innodb_log_recovery_start and innodb_log_recovery_target. If innodb_log_recovery_target>0, InnoDB will be in read-only mode, not allowing any writes to persistent files other than via the log application.

To restore a streaming backup made with BACKUP SERVER WITH, an empty directory needs to be created and all streams be extracted there using the standard tar utility of the operating system, optionally after undoing any encryption or compression that had been added by the backup command. Then, the backup is prepared or MariaDB server started up on the extracted directory, similar to as if the BACKUP SERVER TO statement had been used.

Note: The parameter innodb_log_recovery_start in backup.cnf is STRICTLY NECESSARY TO AVOID CORRUPTION! By default, InnoDB crash recovery starts from the latest available log checkpoint. However, for restoring a backup, recovery must start from the checkpoint that was the latest when the backup was started. Starting recovery from a possible later checkpoint will result in a corrupted database!

The following will be implemented separately:

MDEV-39061 mariadb-backup compatible wrapper script for BACKUP SERVER
MDEV-40163 Partial backup and restore
MDEV-39091 Back up ENGINE=RocksDB
MDEV-39092 Less blocking backup of ENGINE=Aria

The implementation introduces a basic driver Sql_cmd_backup, storage engine interfaces, and basic copying of the storage engines InnoDB, Aria, MyISAM, MERGE (MyISAM), Archive, CSV.

backup_target: A structured data type to represent a target directory. On Microsoft Windows, we must use directory paths because there is no variant of CopyFileEx() that would work on file handles.

backup_sink: Wraps a per-thread output stream as well as storage engine specific context.

handlerton::backup_start(), handlerton::backup_end(): Invoked at the start or end of a backup phase, in the thread that executes a BACKUP SERVER statement.

handlerton::backup_step(): A backup step that can be invoked from multiple threads concurrently, between the execution of the corresponding handlerton::backup_start() and handlerton::backup_end() of the same phase.

copy_entire_file(): A file copying service for POSIX systems.

copy_file(): A partial or sparse file-copying service for all systems.

backup_stream_append(): Equivalent to copy_file(), but appending to a stream. On Linux, this uses sendfile(2), which assumes that the source data will not be changed before the data has been consumed from the pipe.

backup_stream_append_async(): A variant of backup_stream_append() where the source file region is guaranteed to be immutable after the call returns. We must not use Linux sendfile(2) for copying data files that may be modified in place, because it could introduce a race condition between a page write that runs concurrently with a child process that is reading the data from the pipe.

InnoDB_backup::context: Backup context, attached to backup_sink so that context can continue to exist between the time a BACKUP SERVER releases all locks and another BACKUP SERVER starts executing, with innodb_backup pointing to the new backup, while the old backup is still being finished.

fil_space_t::write_or_backup: Keep track of in-flight page writes and pending backup operation. We must not allow them concurrently, because that could lead into torn pages in the backup.

fil_space_t::backup_end: The first page number that is not being backed up (by default 0, to indicate that no backup is in progress).

fil_space_t::BACKUP_BATCH_SIZE: The number of preceding pages that will be covered by fil_space_t::backup_end. This is the unit of "page range locking" during InnoDB backup.

log_sys.backup: Whether BACKUP SERVER is in progress. The purpose of this is to make BACKUP SERVER prevent the concurrent execution of SET GLOBAL innodb_log_archive=OFF or SET GLOBAL innodb_log_file_size when innodb_log_archive=OFF.

log_sys.archived_checkpoint: Keep track of the earliest available checkpoint, corresponding to log_sys.archived_lsn. This reflects SET GLOBAL innodb_log_recovery_start (which is settable now), for incremental backup.

buf_flush_list_space(): Check for concurrent backup before writing each page. This is inefficient, but this function may be invoked from multiple threads concurrently, and it cannot be changed easily, especially for fil_crypt_thread().

fil_system.have_all_spaces: Whether all tablespace metadata is guaranteed to be known. To speed up startup, InnoDB does not normally open all tablespace files.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[mariadb-SaahilAlam]

Please check this crash
# 2026-09-30T02:27:05 [3500784] | mariadbd: /data/Server/MDEV-14992_09/storage/innobase/log/log0log.cc:2140: void log_t::checkpoint_margin(): Assertion lsn <= get_lsn()' failed.`

RR trace is available on SDP :

/data/results/1790753566/004243

[dr-m]

I filed MDEV-41359 and pushed 3736e0c to address this. The assertion is correct; it had been added in 33ec544.

[mariadb-SaahilAlam]

Please check this crash on release build

stack_bottom = 0x7ed4d431f000 thread_stack 0x49000
/data/Server_bin/MDEV-14992_12_RelWithDebInfo/bin/mariadbd(my_print_stacktrace+0x32)[0x58f770fc7382]
mysys/stacktrace.c:216(my_print_stacktrace)[0x58f770b4c7af]
libc_sigaction.c:0(__restore_rt)[0x7ed4d6445330]
/data/Server_bin/MDEV-14992_12_RelWithDebInfo/bin/mariadbd(+0xf8a46f)[0x58f770e1646f]
include/fil0fil.h:1192(fil_node_t::set_backup_name())[0x58f770cf35b8]
sql/sql_plugin.cc:2584(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int*, void*), int, unsigned int, void*))[0x58f7708d3628]
sql/sql_backup.cc:1380(backup_execute(THD*, char const*, char const*, int))[0x58f770cf2ee4]
sql/sql_parse.cc:5949(mysql_execute_command(THD*, bool))[0x58f7708b7d36]
sql/sql_parse.cc:8024(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x58f7708bc583]
sql/sql_parse.cc:1906(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x58f7708be862]
sql/sql_parse.cc:1438(do_command(THD*, bool))[0x58f7708bff56]
sql/sql_connect.cc:1514(do_handle_one_connection(CONNECT*, bool))[0x58f7709ea15d]
sql/sql_connect.cc:1432(handle_one_connection)[0x58f7709ea455]
nptl/pthread_create.c:447(start_thread)[0x7ed4d649caa4]
x86_64/clone3.S:80(clone3)[0x7ed4d6529c6c]

MTR test:-

--source include/have_innodb.inc

SET @save_threads= @@GLOBAL.innodb_encryption_threads;
SET @save_key_age= @@GLOBAL.innodb_encryption_rotate_key_age;
SET GLOBAL innodb_encryption_threads= 16;

DELIMITER |;
CREATE PROCEDURE create_churn()
BEGIN
  WHILE IS_FREE_LOCK('stop_churn') DO
    CREATE OR REPLACE TABLE t1 (a INT) ENGINE=InnoDB ENCRYPTED=NO;
  END WHILE;
END|
CREATE PROCEDURE key_age_churn()
BEGIN
  WHILE IS_FREE_LOCK('stop_churn') DO
    SET GLOBAL innodb_encryption_rotate_key_age= 2;
    SET GLOBAL innodb_encryption_rotate_key_age= 1;
  END WHILE;
END|
DELIMITER ;|

--connect (con1,localhost,root,,)
--send CALL create_churn()
--connect (con2,localhost,root,,)
--send CALL key_age_churn()

--connection default
--let $wait_condition= SELECT COUNT(*) > 0 FROM information_schema.processlist WHERE info LIKE 'CREATE OR REPLACE TABLE t1%'
--source include/wait_condition.inc

--let $target_directory= $MYSQLTEST_VARDIR/tmp/backup_create_concurrent
--let $attempts= 200
--disable_query_log
while ($attempts)
{
  --error 0,ER_LOCK_DEADLOCK
  --eval BACKUP SERVER TO '$target_directory'
  --error 0,1
  --rmdir $target_directory
  dec $attempts;
}
--enable_query_log

SELECT GET_LOCK('stop_churn', 0);
--connection con1
--reap
--disconnect con1
--connection con2
--reap
--disconnect con2
--connection default
SELECT RELEASE_LOCK('stop_churn');

SET GLOBAL innodb_encryption_rotate_key_age= @save_key_age;
SET GLOBAL innodb_encryption_threads= @save_threads;
DROP PROCEDURE create_churn;
DROP PROCEDURE key_age_churn;
DROP TABLE t1;

[dr-m]

Filed as MDEV-41395 fil_space_t::create() may release fil_system.mutex.