The document describes a toolkit for recovering data from MySQL databases using InnoDB tables. It provides an overview of the tools in the toolkit, including stream_parser to parse raw data files into InnoDB pages, sys_parser to recover table structures from those pages, and c_parser and recover_dictionary tools to extract and load table records and reconstruct the InnoDB data dictionary. It demonstrates dropping a sample table and then using the toolkit to recover the table structure and records from the raw data files.

1. Toolkit for MySQL Data Recovery
https://twindb.com/
UNDROP FOR INNODB

2. 2
No Backups?
UnDROP for InnoDB

3. 3
Roadmap
•Update since last year
•How toolkit works
•Demo

4. 4
Update since last year
•Multi-threaded stream_parser
Raw image
Worker #1 Worker #2 Worker #3

5. 5
Update since last year
•SQL parser to define table structure

6. Table in File System
# ls -la /var/lib/mysql/sakila/actor.*
-rw-rw---- 1 mysql mysql actor.frm
-rw-rw---- 1 mysql mysql actor.ibd
# ls -la /var/lib/mysql/
-rw-rw----. 1 mysql mysql ibdata1
-rw-rw----. 1 mysql mysql ib_logfile0
-rw-rw----. 1 mysql mysql ib_logfile1

7. PRIMARY index - B+ Tree Structure
1 100
page 2 page 3
1 50
page 4 page 5
100 150
page 6 page 7
actor_id 1 2 … 49
first_name A C E
last_name K D F
last_updat
e
X Y … Z
actor_id 50 51 … 99
first_name G I K
last_name H J L
last_updat
e
X Y … Z
Page 3Page 2
Page 1
Page 4 Page 5

8. Record format
A row: (10, ‘Kevin’, ‘Spacey’, ‘2015-04-08 21:47:36’)
8
Actually stored as: (10, TRX_ID, PTR_ID, ‘Kevin’, ‘Spacey’, ‘2015-04-08 21:47:36’)
…. next
Extra 5 bytes:
0x00 00 00 0A
Fields
... ... Kevin 0x5525A1FF
A bit per NULL-able field
7 Spacey
Lengths

9. InnoDB Dictionary
mysql> SELECT * FROM SYS_TABLES WHERE NAME = 'sakila/actor'G
*************************** 1. row ***************************
NAME: sakila/actor
ID: 2642
N_COLS: 4
TYPE: 41
MIX_ID: 0
MIX_LEN: 80
CLUSTER_NAME:
SPACE: 2337
1 row in set (0.00 sec)
SYS_TABLES

10. InnoDB Dictionary
mysql> SELECT * FROM SYS_INDEXES WHERE TABLE_ID = '2642'G
*************************** 1. row ***************************
TABLE_ID: 2642
ID: 6133
NAME: PRIMARY
N_FIELDS: 1
TYPE: 3
SPACE: 2337
PAGE_NO: 3
*************************** 2. row ***************************
TABLE_ID: 2642
ID: 6134
NAME: idx_actor_last_name
N_FIELDS: 1
TYPE: 0
SPACE: 2337
PAGE_NO: 4
2 rows in set (0.00 sec)
SYS_INDEXES

11. 11
UnDrop for InnoDB

12. Introducing stream_parser
• Finds InnoDB pages in bytes stream
00000000 b7 da 39 28 00 00 00 0b ff ff ff ff ff ff ff ff |..9(............|
00000010 00 00 00 00 ff 98 e1 54 45 bf 00 00 00 00 00 00 |.......TE.......|
00000020 00 00 00 00 00 00 00 1c 25 cd 00 83 23 4f 04 4c |........%...#O.L|
00000030 00 00 00 02 00 01 00 73 00 00 00 00 00 00 00 00 |.......s........|
00000040 00 00 00 00 00 00 00 00 00 03 00 00 00 00 00 00 |................|
00000050 00 02 08 72 00 00 00 00 00 00 00 02 07 b2 08 01 |...r............|
00000060 00 00 03 00 8c 69 6e 66 69 6d 75 6d 00 09 05 00 |.....infimum....|
00000070 08 03 00 00 73 75 70 72 65 6d 75 6d 00 33 2f 2b |....supremum.3/+|
00000080 27 23 1d 16 10 08 00 00 10 13 00 ce 00 00 00 00 |'#..............|
00000090 00 00 00 0b 00 00 00 00 00 00 00 0b 00 00 00 00 |................|

13. stream_parser in action
# df -h /var/lib/mysql/
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 5.8G 146M 5.4G 3% /mnt/data
# ./stream_parser -f /dev/sdb -t 5.8G
Opening file: /dev/sdb
…
Size to process: 6227702784 (5.800 GiB)
…
Worker(0): 99.95% done. 2014-09-14 17:30:38 ETA(in 00:00:00).
Processing speed: 808.000 MiB/sec
All workers finished in 9 sec

14. stream_parser Result
# ls -la pages-sdb
total 24
drwxr-xr-x 4 root root 4096 Sep 14 17:30 .
drwxr-xr-x 7 root root 4096 Sep 14 17:30 ..
drwxr-xr-x 2 root root 12288 Sep 14 17:30 FIL_PAGE_INDEX
drwxr-xr-x 2 root root 4096 Sep 14 17:30 FIL_PAGE_TYPE_BLOB
# ls -la pages-sdb/FIL_PAGE_INDEX | head -100| tail -5
-rw-r--r-- 1 root root 81920 Sep 14 17:30 0000000000000727.page
-rw-r--r-- 1 root root 114688 Sep 14 17:30 0000000000000728.page
-rw-r--r-- 1 root root 16384 Sep 14 17:30 0000000000000729.page
-rw-r--r-- 1 root root 786432 Sep 14 17:30 0000000000000730.page
-rw-r--r-- 1 root root 180224 Sep 14 17:30 0000000000000731.page

15. Introducing c_parser
# ./c_parser -5f
pages-sdb/FIL_PAGE_INDEX/0000000000000022.page
-t actor.sql
> dumps/default/actor
2> dumps/default/actor.sql

16. c_parser Result
# head -5 dumps/default/actor
-- Page id: 3, Format: COMPACT, Records list: Valid, Expected records: (200 200)
000000000740 C0000001630110 actor 1 "PENELOPE" "GUINESS" "2006-02-15
04:34:33"
000000000740 C000000163011A actor 2 "NICK" "WAHLBERG" "2006-02-15 04:34:33"
000000000740 C0000001630124 actor 3 "ED" "CHASE" "2006-02-15 04:34:33"
000000000740 C000000163012E actor 4 "JENNIFER" "DAVIS" "2006-02-15 04:34:33"
# cat dumps/default/actor.sql
SET FOREIGN_KEY_CHECKS=0;
LOAD DATA LOCAL INFILE '/root/undrop-for-innodb/dumps/default/actor'
REPLACE INTO TABLE `actor` FIELDS TERMINATED BY 't' OPTIONALLY
ENCLOSED BY '"' LINES STARTING BY 'actort' (`actor_id`, `first_name`,
`last_name`, `last_update`);

17. Introducing sys_parser
# yum install mysql-community-devel.x86_64
# make sys_parser
/usr/bin/mysql_config
cc `mysql_config --cflags` `mysql_config --libs` -o sys_parser sys_parser.c
# ./sys_parser sakila/actor
CREATE TABLE `actor`(
`actor_id` SMALLINT UNSIGNED NOT NULL,
`first_name` VARCHAR(45) CHARACTER SET 'utf8' COLLATE
'utf8_general_ci' NOT NULL,
`last_name` VARCHAR(45) CHARACTER SET 'utf8' COLLATE
'utf8_general_ci' NOT NULL,
`last_update` TIMESTAMP NOT NULL,
PRIMARY KEY (`actor_id`)
) ENGINE=InnoDB;

18. Recover InnoDB Dictionary
# ./stream_parser -f /var/lib/mysql/ibdata1
Opening file: /var/lib/mysql/ibdata1
Size to process: 12582912 (12.000 MiB)
All workers finished in 0 sec
# ./recover_dictionary.sh
Generating dictionary tables dumps... OK
Creating test database ... OK
Creating dictionary tables in database test:
SYS_TABLES ... OK
SYS_COLUMNS ... OK
SYS_INDEXES ... OK
SYS_FIELDS ... OK
All OK
Loading dictionary tables data:
SYS_TABLES ... 46 recs OK
SYS_COLUMNS ... 304 recs OK
SYS_INDEXES ... 96 recs OK
SYS_FIELDS ... 118 recs OK
All OK

19. Getting index_id from Dictionary
mysql> SELECT SYS_TABLES.NAME, SYS_INDEXES.NAME,
SYS_INDEXES.ID
FROM SYS_TABLES
LEFT JOIN SYS_INDEXES
ON SYS_TABLES.ID = SYS_INDEXES.TABLE_ID
WHERE SYS_TABLES.NAME = 'sakila/actor';
+--------------+---------------------+------+
| NAME | NAME | ID |
+--------------+---------------------+------+
| sakila/actor | PRIMARY | 22 |
| sakila/actor | idx_actor_last_name | 23 |
+--------------+---------------------+------+
2 rows in set (0.00 sec)

20. 20
Demo
•Drop table sakila.actor
•File per table
•Scan disk image
•Recover structure from dictionary
•Recover table records

21. Let’s DROP TABLE `actor`?
mysql> drop table sakila.actor;
ERROR 1217 (23000): Cannot delete or update a
parent row: a foreign key constraint fails
mysql> SET FOREIGN_KEY_CHECKS=0;
Query OK, 0 rows affected (0.00 sec)
mysql> drop table sakila.actor;
Query OK, 0 rows affected (0.01 sec)

22. Table’s removed from Dictionary
mysql> SELECT * FROM
information_schema.INNODB_SYS_TABLES
WHERE NAME = 'sakila/actor';
Empty set (0.00 sec)

23. actor.frm is removed
# ls -la /var/lib/mysql/sakila/actor.frm
ls: cannot access /var/lib/mysql/sakila/
actor.frm: No such file or directory

24. So is actor.ibd
# ls -la /var/lib/mysql/sakila/actor.ibd
ls: cannot access /var/lib/mysql/sakila/
actor.ibd: No such file or directory

25. InnoDB Dictionary
•We need it for:
a) index_id
b) Table structure

26. Where is my table?
# grep -r PENELO pages-sdb/FIL_PAGE_INDEX
Binary file pages-sdb/FIL_PAGE_INDEX/0000000000000022.page matches
Binary file pages-sdb/FIL_PAGE_INDEX/0000000000000706.page matches
Binary file pages-sdb/FIL_PAGE_INDEX/0000000000000747.page matches
# hexdump -C pages-sdb/FIL_PAGE_INDEX/0000000000000022.page | head -15
00000000 ea a2 dd d0 00 00 00 03 ff ff ff ff ff ff ff ff |................|
00000010 00 00 00 00 00 1b 82 20 45 bf 00 00 00 00 00 00 |....... E.......|
00000020 00 00 00 00 00 06 00 33 1d cb 80 ca 00 00 00 00 |.......3........|
00000030 1d ad 00 02 00 c7 00 c8 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 16 00 00 00 06 00 00 |................|
00000050 00 02 00 f2 00 00 00 06 00 00 00 02 00 32 01 00 |.............2..|
00000060 02 00 1c 69 6e 66 69 6d 75 6d 00 05 00 0b 00 00 |...infimum......|
00000070 73 75 70 72 65 6d 75 6d 07 08 00 00 10 00 29 00 |supremum......).|
00000080 01 00 00 00 00 07 40 c0 00 00 01 63 01 10 50 45 |......@....c..PE|
00000090 4e 45 4c 4f 50 45 47 55 49 4e 45 53 53 43 f2 f5 |NELOPEGUINESSC..|
000000a0 a9 08 04 00 00 18 00 26 00 02 00 00 00 00 07 40 |.......&.......@|
000000b0 c0 00 00 01 63 01 1a 4e 49 43 4b 57 41 48 4c 42 |....c..NICKWAHLB|
000000c0 45 52 47 43 f2 f5 a9 05 02 00 00 20 00 21 00 03 |ERGC....... .!..|
000000d0 00 00 00 00 07 40 c0 00 00 01 63 01 24 45 44 43 |.....@....c.$EDC|
000000e0 48 41 53 45 43 f2 f5 a9 05 08 04 00 28 00 27 00 |HASEC.......(.'.|

27. Checklist before Launch
✓InnoDB pages (credit to stream_parser)
✓Table Structure (credit to sys_parser)
✓index_id (InnoDB dictionary)

28. Load Table Back
# mysql sakila < actor.sql
# mysql sakila < dumps/default/actor.sql
# mysql sakila -e "SELECT COUNT(*) FROM actor"
+----------+
| COUNT(*) |
+----------+
| 200 |
+----------+

29. 29
https://twindb.com

30. Follow Us
https://twindb.com