Skip to content

Commit ab1e6fe

Browse files
committed
MDEV-11623 MariaDB 10.1 fails to start datadir created with
MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
1 parent a9d00db commit ab1e6fe

39 files changed

+1483
-1497
lines changed
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#!/usr/bin/perl
2+
# Convert tablespace flags to the format understood by MariaDB 10.1.0..10.1.20,
3+
# with the assumption that the flags were correct.
4+
5+
sub convert_to_mariadb_101
6+
{
7+
my ($file, $page_size) = @_;
8+
open(FILE, "+<", $file) or die "Unable to open $file\n";
9+
sysread(FILE, $_, $page_size)==$page_size||die "Unable to read $file\n";
10+
sysseek(FILE, 0, 0)||die "Unable to seek $file\n";
11+
12+
# FIL_PAGE_DATA + FSP_SPACE_FLAGS = 38 + 16 = 54 bytes from the start
13+
my($flags) = unpack "x[54]N";
14+
my $badflags = ($flags & 0x3f);
15+
my $compression_level=6;
16+
$badflags |= 1<<6|$compression_level<<7 if ($flags & 1 << 16);
17+
$badflags |= ($flags & 15 << 6) << 7; # PAGE_SSIZE
18+
19+
substr ($_, 54, 4) = pack("N", $badflags);
20+
# Replace the innodb_checksum_algorithm=none checksum
21+
substr ($_, 0, 4) = pack("N", 0xdeadbeef);
22+
substr ($_, $page_size - 8, 4) = pack("N", 0xdeadbeef);
23+
syswrite(FILE, $_, $page_size)==$page_size||die "Unable to write $file\n";
24+
close(FILE);
25+
}
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
#
2+
# MDEV-11623 MariaDB 10.1 fails to start datadir created with
3+
# MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K
4+
#
5+
call mtr.add_suppression("InnoDB: adjusting FSP_SPACE_FLAGS of tablespace");
6+
SET GLOBAL innodb_file_per_table=1;
7+
SET GLOBAL innodb_file_format=Barracuda;
8+
CREATE TABLE tr(a INT)ENGINE=InnoDB ROW_FORMAT=REDUNDANT;
9+
CREATE TABLE tc(a INT)ENGINE=InnoDB ROW_FORMAT=COMPACT;
10+
CREATE TABLE td(a INT)ENGINE=InnoDB ROW_FORMAT=DYNAMIC;
11+
CREATE TABLE tz(a INT)ENGINE=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=1;
12+
CREATE TABLE tdd(a INT) ENGINE=InnoDB, DATA DIRECTORY='MYSQL_TMP_DIR';
13+
CREATE TABLE tp(a INT) ENGINE=InnoDB page_compressed=1;
14+
CREATE TABLE ti(a INT) ENGINE=InnoDB;
15+
FLUSH TABLES ti FOR EXPORT;
16+
backup: ti
17+
UNLOCK TABLES;
18+
ALTER TABLE ti DISCARD TABLESPACE;
19+
restore: ti .ibd and .cfg files
20+
ALTER TABLE ti IMPORT TABLESPACE;
21+
BEGIN;
22+
INSERT INTO tr VALUES(1);
23+
INSERT INTO tc VALUES(1);
24+
INSERT INTO td VALUES(1);
25+
INSERT INTO tz VALUES(1);
26+
INSERT INTO tdd VALUES(1);
27+
INSERT INTO tp VALUES(1);
28+
INSERT INTO ti VALUES(1);
29+
# Kill the server
30+
CHECK TABLE tr,tc,td,tz,tdd,tp,ti;
31+
Table Op Msg_type Msg_text
32+
test.tr check status OK
33+
test.tc check status OK
34+
test.td check status OK
35+
test.tz check status OK
36+
test.tdd check status OK
37+
test.tp check status OK
38+
test.ti check status OK
39+
CHECK TABLE tr,tc,td,tz,tdd,tp,ti;
40+
Table Op Msg_type Msg_text
41+
test.tr check status OK
42+
test.tc check status OK
43+
test.td check status OK
44+
test.tz check status OK
45+
test.tdd check status OK
46+
test.tp check status OK
47+
test.ti check status OK
48+
DROP TABLE tr,tc,td,tz,tdd,tp,ti;

mysql-test/suite/innodb/r/doublewrite.result

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ set global innodb_buf_flush_list_now = 1;
3939
# Kill the server
4040
# Make the first page (page_no=0) of the user tablespace
4141
# full of zeroes.
42+
#
43+
# MDEV-11623: Use old FSP_SPACE_FLAGS in the doublewrite buffer.
4244
check table t1;
4345
Table Op Msg_type Msg_text
4446
test.t1 check status OK

mysql-test/suite/innodb/r/innodb-wl5522-debug-zip.result

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
1-
call mtr.add_suppression("InnoDB: Page for tablespace .* ");
1+
call mtr.add_suppression("InnoDB: Page for tablespace ");
2+
call mtr.add_suppression("InnoDB: Invalid FSP_SPACE_FLAGS=0x");
23
FLUSH TABLES;
34
SET GLOBAL innodb_file_per_table = 1;
45
SELECT @@innodb_file_per_table;
@@ -565,7 +566,7 @@ ERROR HY000: Tablespace has been discarded for table 't1'
565566
restore: t1 .ibd and .cfg files
566567
SET SESSION debug_dbug="+d,fsp_flags_is_valid_failure";
567568
ALTER TABLE test_wl5522.t1 IMPORT TABLESPACE;
568-
ERROR HY000: Internal error: Cannot reset LSNs in table '"test_wl5522"."t1"' : Unsupported
569+
ERROR HY000: Internal error: Cannot reset LSNs in table '"test_wl5522"."t1"' : Data structure corruption
569570
SET SESSION debug_dbug="-d,fsp_flags_is_valid_failure";
570571
DROP TABLE test_wl5522.t1;
571572
unlink: t1.ibd
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
--source include/have_innodb.inc
2+
--source include/not_embedded.inc
3+
4+
-- echo #
5+
-- echo # MDEV-11623 MariaDB 10.1 fails to start datadir created with
6+
-- echo # MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K
7+
-- echo #
8+
9+
# This is actually testing the opposite: starting the fixed 10.1 with
10+
# buggy 10.1 files (by manually converting the flags in the files).
11+
12+
call mtr.add_suppression("InnoDB: adjusting FSP_SPACE_FLAGS of tablespace");
13+
SET GLOBAL innodb_file_per_table=1;
14+
SET GLOBAL innodb_file_format=Barracuda;
15+
let INNODB_PAGE_SIZE=`select @@innodb_page_size`;
16+
let MYSQLD_DATADIR=`select @@datadir`;
17+
18+
CREATE TABLE tr(a INT)ENGINE=InnoDB ROW_FORMAT=REDUNDANT;
19+
CREATE TABLE tc(a INT)ENGINE=InnoDB ROW_FORMAT=COMPACT;
20+
CREATE TABLE td(a INT)ENGINE=InnoDB ROW_FORMAT=DYNAMIC;
21+
--disable_warnings
22+
CREATE TABLE tz(a INT)ENGINE=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=1;
23+
--enable_warnings
24+
25+
--replace_result $MYSQL_TMP_DIR MYSQL_TMP_DIR
26+
EVAL CREATE TABLE tdd(a INT) ENGINE=InnoDB, DATA DIRECTORY='$MYSQL_TMP_DIR';
27+
28+
CREATE TABLE tp(a INT) ENGINE=InnoDB page_compressed=1;
29+
CREATE TABLE ti(a INT) ENGINE=InnoDB;
30+
FLUSH TABLES ti FOR EXPORT;
31+
perl;
32+
do "$ENV{MTR_SUITE_DIR}/include/innodb-util.pl";
33+
ib_backup_tablespaces("test", "ti");
34+
EOF
35+
UNLOCK TABLES;
36+
ALTER TABLE ti DISCARD TABLESPACE;
37+
perl;
38+
do "$ENV{MTR_SUITE_DIR}/include/innodb-util.pl";
39+
ib_discard_tablespaces("test", "ti");
40+
ib_restore_tablespaces("test", "ti");
41+
do "$ENV{MTR_SUITE_DIR}/include/ibd_convert.pl";
42+
my $ps = $ENV{INNODB_PAGE_SIZE};
43+
my $dd = $ENV{MYSQLD_DATADIR};
44+
convert_to_mariadb_101("$dd/test/ti.ibd", $ps);
45+
EOF
46+
47+
ALTER TABLE ti IMPORT TABLESPACE;
48+
49+
BEGIN;
50+
INSERT INTO tr VALUES(1);
51+
INSERT INTO tc VALUES(1);
52+
INSERT INTO td VALUES(1);
53+
INSERT INTO tz VALUES(1);
54+
INSERT INTO tdd VALUES(1);
55+
INSERT INTO tp VALUES(1);
56+
INSERT INTO ti VALUES(1);
57+
58+
--source include/kill_mysqld.inc
59+
60+
perl;
61+
do "$ENV{MTR_SUITE_DIR}/include/ibd_convert.pl";
62+
my $ps = $ENV{INNODB_PAGE_SIZE};
63+
my $dd = $ENV{MYSQLD_DATADIR};
64+
65+
convert_to_mariadb_101("$dd/ibdata1", $ps);
66+
convert_to_mariadb_101("$dd/test/tr.ibd", $ps);
67+
convert_to_mariadb_101("$dd/test/tc.ibd", $ps);
68+
convert_to_mariadb_101("$dd/test/td.ibd", $ps);
69+
convert_to_mariadb_101("$dd/test/tz.ibd", 1024) if $ps<32768;
70+
convert_to_mariadb_101("$dd/test/tp.ibd", $ps);
71+
convert_to_mariadb_101("$dd/test/ti.ibd", $ps);
72+
convert_to_mariadb_101("$ENV{MYSQL_TMP_DIR}/test/tdd.ibd", $ps);
73+
EOF
74+
75+
--source include/start_mysqld.inc
76+
CHECK TABLE tr,tc,td,tz,tdd,tp,ti;
77+
--source include/shutdown_mysqld.inc
78+
79+
perl;
80+
do "$ENV{MTR_SUITE_DIR}/include/ibd_convert.pl";
81+
my $ps = $ENV{INNODB_PAGE_SIZE};
82+
my $dd = $ENV{MYSQLD_DATADIR};
83+
84+
convert_to_mariadb_101("$dd/ibdata1", $ps);
85+
convert_to_mariadb_101("$dd/test/tr.ibd", $ps);
86+
convert_to_mariadb_101("$dd/test/tc.ibd", $ps);
87+
convert_to_mariadb_101("$dd/test/td.ibd", $ps);
88+
convert_to_mariadb_101("$dd/test/tz.ibd", 1024) if $ps<32768;
89+
convert_to_mariadb_101("$dd/test/tp.ibd", $ps);
90+
convert_to_mariadb_101("$dd/test/ti.ibd", $ps);
91+
convert_to_mariadb_101("$ENV{MYSQL_TMP_DIR}/test/tdd.ibd", $ps);
92+
EOF
93+
94+
--let $restart_parameters=--innodb-read-only
95+
--source include/start_mysqld.inc
96+
CHECK TABLE tr,tc,td,tz,tdd,tp,ti;
97+
--source include/shutdown_mysqld.inc
98+
99+
--let $restart_parameters=
100+
--source include/start_mysqld.inc
101+
DROP TABLE tr,tc,td,tz,tdd,tp,ti;

mysql-test/suite/innodb/t/doublewrite.test

Lines changed: 38 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ call mtr.add_suppression("space header page consists of zero bytes.*test.t1");
1717
call mtr.add_suppression("checksum mismatch in tablespace.*test.t1");
1818
call mtr.add_suppression("Current page size .* != page size on page");
1919
call mtr.add_suppression("innodb-page-size mismatch in tablespace.*test.t1");
20-
call mtr.add_suppression("Database page corruption");
20+
call mtr.add_suppression("Trying to recover page.*from the doublewrite buffer");
2121
--enable_query_log
2222

2323
let INNODB_PAGE_SIZE=`select @@innodb_page_size`;
@@ -65,14 +65,48 @@ set global innodb_buf_flush_list_now = 1;
6565

6666
--echo # Make the first page (page_no=0) of the user tablespace
6767
--echo # full of zeroes.
68+
--echo #
69+
--echo # MDEV-11623: Use old FSP_SPACE_FLAGS in the doublewrite buffer.
70+
6871
perl;
6972
use IO::Handle;
7073
my $fname= "$ENV{'MYSQLD_DATADIR'}test/t1.ibd";
74+
my $page_size = $ENV{INNODB_PAGE_SIZE};
75+
my $page;
7176
open(FILE, "+<", $fname) or die;
72-
FILE->autoflush(1);
73-
binmode FILE;
74-
print FILE chr(0) x ($ENV{'INNODB_PAGE_SIZE'});
77+
sysread(FILE, $page, $page_size)==$page_size||die "Unable to read $file\n";
78+
sysseek(FILE, 0, 0)||die "Unable to seek $file\n";
79+
die unless syswrite(FILE, chr(0) x $page_size, $page_size) == $page_size;
7580
close FILE;
81+
82+
open(FILE, "+<", "$ENV{MYSQLD_DATADIR}ibdata1")||die "cannot open ibdata1\n";
83+
sysseek(FILE, 6 * $page_size - 190, 0)||die "Unable to seek ibdata1\n";
84+
sysread(FILE, $_, 12) == 12||die "Unable to read TRX_SYS\n";
85+
my($magic,$d1,$d2)=unpack "NNN";
86+
die "magic=$magic, $d1, $d2\n" unless $magic == 536853855 && $d2 >= $d1 + 64;
87+
sysseek(FILE, $d1 * $page_size, 0)||die "Unable to seek ibdata1\n";
88+
# Find the page in the doublewrite buffer
89+
for (my $d = $d1; $d < $d2 + 64; $d++)
90+
{
91+
sysread(FILE, $_, $page_size)==$page_size||die "Cannot read doublewrite\n";
92+
next unless $_ eq $page;
93+
sysseek(FILE, $d * $page_size, 0)||die "Unable to seek ibdata1\n";
94+
# Write buggy MariaDB 10.1.x FSP_SPACE_FLAGS to the doublewrite buffer
95+
my($flags) = unpack "x[54]N";
96+
my $badflags = ($flags & 0x3f);
97+
my $compression_level=6;
98+
$badflags |= 1<<6|$compression_level<<7 if ($flags & 1 << 16);
99+
$badflags |= ($flags & 15 << 6) << 7; # PAGE_SSIZE
100+
101+
substr ($_, 54, 4) = pack("N", $badflags);
102+
# Replace the innodb_checksum_algorithm=none checksum
103+
substr ($_, 0, 4) = pack("N", 0xdeadbeef);
104+
substr ($_, $page_size - 8, 4) = pack("N", 0xdeadbeef);
105+
syswrite(FILE, $_, $page_size)==$page_size||die "Unable to write $file\n";
106+
close(FILE);
107+
exit 0;
108+
}
109+
die "Did not find the page in the doublewrite buffer ($d1,$d2)\n";
76110
EOF
77111

78112
--source include/start_mysqld.inc

mysql-test/suite/innodb/t/innodb-wl5522-debug-zip.test

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,8 @@
1717
# allow test to run only when innodb-page-size=16
1818
--source include/have_innodb_16k.inc
1919

20-
call mtr.add_suppression("InnoDB: Page for tablespace .* ");
20+
call mtr.add_suppression("InnoDB: Page for tablespace ");
21+
call mtr.add_suppression("InnoDB: Invalid FSP_SPACE_FLAGS=0x");
2122
FLUSH TABLES;
2223

2324
let MYSQLD_DATADIR =`SELECT @@datadir`;

storage/innobase/buf/buf0dblwr.cc

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -589,6 +589,21 @@ buf_dblwr_process()
589589
continue;
590590
}
591591

592+
if (page_no == 0) {
593+
/* Check the FSP_SPACE_FLAGS. */
594+
ulint flags = fsp_header_get_flags(page);
595+
if (!fsp_flags_is_valid(flags)
596+
&& fsp_flags_convert_from_101(flags)
597+
== ULINT_UNDEFINED) {
598+
ib_logf(IB_LOG_LEVEL_WARN,
599+
"Ignoring a doublewrite copy of page "
600+
ULINTPF ":0 due to invalid flags 0x%x",
601+
space_id, int(flags));
602+
continue;
603+
}
604+
/* The flags on the page should be converted later. */
605+
}
606+
592607
/* Write the good page from the doublewrite buffer to
593608
the intended position. */
594609

storage/innobase/dict/dict0load.cc

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1052,8 +1052,6 @@ dict_check_tablespaces_and_store_max_id(
10521052

10531053
btr_pcur_store_position(&pcur, &mtr);
10541054

1055-
mtr_commit(&mtr);
1056-
10571055
/* For tables created with old versions of InnoDB,
10581056
SYS_TABLES.MIX_LEN may contain garbage. Such tables
10591057
would always be in ROW_FORMAT=REDUNDANT. Pretend that
@@ -1087,16 +1085,19 @@ dict_check_tablespaces_and_store_max_id(
10871085
if (space_id == 0) {
10881086
/* The system tablespace always exists. */
10891087
ut_ad(!discarded);
1090-
goto next_tablespace;
1088+
mem_free(name);
1089+
goto loop;
10911090
}
10921091

1092+
mtr_commit(&mtr);
1093+
10931094
switch (dict_check) {
10941095
case DICT_CHECK_ALL_LOADED:
10951096
/* All tablespaces should have been found in
10961097
fil_load_single_table_tablespaces(). */
10971098
if (fil_space_for_table_exists_in_mem(
1098-
space_id, name, TRUE, !(is_temp || discarded),
1099-
false, NULL, 0)
1099+
space_id, name, !(is_temp || discarded),
1100+
false, NULL, 0, flags)
11001101
&& !(is_temp || discarded)) {
11011102
/* If user changes the path of .ibd files in
11021103
*.isl files before doing crash recovery ,
@@ -1128,8 +1129,8 @@ dict_check_tablespaces_and_store_max_id(
11281129
/* Some tablespaces may have been opened in
11291130
trx_resurrect_table_locks(). */
11301131
if (fil_space_for_table_exists_in_mem(
1131-
space_id, name, FALSE, FALSE,
1132-
false, NULL, 0)) {
1132+
space_id, name, false,
1133+
false, NULL, 0, flags)) {
11331134
break;
11341135
}
11351136
/* fall through */
@@ -1191,7 +1192,6 @@ dict_check_tablespaces_and_store_max_id(
11911192
max_space_id = space_id;
11921193
}
11931194

1194-
next_tablespace:
11951195
mem_free(name);
11961196
mtr_start(&mtr);
11971197

@@ -2382,8 +2382,8 @@ dict_load_table(
23822382
table->ibd_file_missing = TRUE;
23832383

23842384
} else if (!fil_space_for_table_exists_in_mem(
2385-
table->space, name, FALSE, FALSE, true, heap,
2386-
table->id)) {
2385+
table->space, name, false, true, heap,
2386+
table->id, table->flags)) {
23872387

23882388
if (DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY)) {
23892389
/* Do not bother to retry opening temporary tables. */

0 commit comments

Comments
 (0)