O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
InnoDB Alchemy
Ryan Robson
#cpgethightech #cpanelconf
A Quick Refresher
• InnoDB is the default engine as of MySQL 5.5.5+
• Non-portable file structu...
#cpgethightech #cpanelconf
DataDictionary
Table
id
flags
columns
indexes
appr. rows
Columns
names
lengths
data types
Index...
#cpgethightech #cpanelconf
Removing this is essentially the same as
removing access to all of your InnoDB tables.
That's t...
#cpgethightech #cpanelconf
Data
Dictionary
•Exists
•Stored Correctly
File
System
•Exists
•Accessible
Error: Trying to open...
#cpgethightech #cpanelconf
Data
Dictionary
•Exists
•Stored Correctly
File
System
•Exists
•Accessible
Unrecognized Ta
ble
E...
#cpgethightech #cpanelconf
Data
Dictionary
•Exists
•Stored Correctly
File
System
•Exists
•Accessible
Functional Table
#cpgethightech #cpanelconf
Deciphering the Error Logs
Error
Type
s
Data Dictionary
Checksum Mismatch
Unable to
open/read/w...
#cpgethightech #cpanelconf
Data Dictionary Errors
Why do they occur? What are your options?
Deleted or moved ibdata1 file
...
#cpgethightech #cpanelconf
Alchemy Solutions
WARNING
Before attempting any alternative recovery solutions, always make sur...
#cpgethightech #cpanelconf
Alchemy Solutions
Hex Editing
Name Byte Length Offset Description
FIL_PAGE_SPACE_OR_CHKSUM 4 0 ...
#cpgethightech #cpanelconf
Alchemy Solutions
Data dictionary example
scenario:Space ID conflict shows in the error log ( /...
#cpgethightech #cpanelconf
Alchemy Solutions
Data dictionary example
scenario:Writing the new space ID to the appropriate ...
#cpgethightech #cpanelconf
Page Corruption Errors
Why do they occur? What are your options?
Power failure / Physical failu...
#cpgethightech #cpanelconf
mysql> SELECT COUNT(1) FROM customer_corrupt;
ERROR 2006 (HY000): MySQL server has gone away
No...
#cpgethightech #cpanelconf
Alchemy Solutions
Page corruption example
scenario:
140911 13:21:36 InnoDB: Page checksum 41165...
#cpgethightech #cpanelconf
Alchemy Solutions
Hex Editing
Name Byte Length Offset Description
FIL_PAGE_SPACE_OR_CHKSUM 4 0 ...
#cpgethightech #cpanelconf
Alchemy Solutions
Hex Editing
storage/innobase/include/fil0fil.h
FIL Trailer (8)
Name Byte Leng...
#cpgethightech #cpanelconf
Alchemy Solutions
Hex Editing
InnoDB: Page checksum 4116551042, prior-to-4.0.14-form checksum 3...
#cpgethightech #cpanelconf
The Future
• MariaDB defaulting with XtraDB
• Aria being developed to eventually be a combinati...
#cpgethightech #cpanelconf
Online Courses
A 3-unit advanced course on InnoDB is currently being developed for our cPanel U...
Questions
#cpanelconf
Contact
#cpanelconf
Email: ryan.robson@cpanel.net
Twitter: cPanelRyan
LinkedIn: https://www.linkedin.com/in/robworks
cPanelCon 2015: InnoDB Alchemy
Próximos SlideShares
Carregando em…5
×

cPanelCon 2015: InnoDB Alchemy

Presented at cPanel Conference 2015 in Denver, CO.

  • Entre para ver os comentários

  • Seja a primeira pessoa a gostar disto

cPanelCon 2015: InnoDB Alchemy

  1. 1. InnoDB Alchemy Ryan Robson
  2. 2. #cpgethightech #cpanelconf A Quick Refresher • InnoDB is the default engine as of MySQL 5.5.5+ • Non-portable file structure • Crash recovery functionality using redo logs • Uses the data dictionary to maintain table definitions, structures, and indexes. • Physical file structure made up of tablespaces, pages, extents, segments, and inodes. • Principles of ACID compliance
  3. 3. #cpgethightech #cpanelconf DataDictionary Table id flags columns indexes appr. rows Columns names lengths data types Indexes id fields unique? location Constraints What does the data dictionary contain? Tables • IDs • Structural Metadata Columns • Names • Attributes Indexes • IDs • Types Where is the data dictionary stored? ibdata 1 Stores the system tablespace, which contains the data dictionary Composed internally of "system tables" that store metadata on table structures. Stored by default in the data directory, which defaults to /var/lib/mysql
  4. 4. #cpgethightech #cpanelconf Removing this is essentially the same as removing access to all of your InnoDB tables. That's the only reason that this can appear to "fix" a crashed MySQL environment. ibdata1 Can be removed for re-creation fairly safely, and can be necessary in some recovery/repair situations. ib_logfileX In the Data Directory
  5. 5. #cpgethightech #cpanelconf Data Dictionary •Exists •Stored Correctly File System •Exists •Accessible Error: Trying to open a table, but could not open the tablespace file './database/table.ibd'! Unrecognized Ta ble
  6. 6. #cpgethightech #cpanelconf Data Dictionary •Exists •Stored Correctly File System •Exists •Accessible Unrecognized Ta ble Error: Can't open table from data dictionary though .frm file exists
  7. 7. #cpgethightech #cpanelconf Data Dictionary •Exists •Stored Correctly File System •Exists •Accessible Functional Table
  8. 8. #cpgethightech #cpanelconf Deciphering the Error Logs Error Type s Data Dictionary Checksum Mismatch Unable to open/read/write/access file Assertion failure Signal 6/Signal 11 Page checksum 863794368, prior-to-4.0.14-form checksum stored checksum 1169627998, prior-to-4.0.14-form stored checksum 2838245694 Examples Can't open table from data dictionary though frm file exists Error: trying to open a table, but could not open the tablespace file './horde/horde_cache.ibd'! Assertion failure in thread 3067665264 in file buf0buf.cc line 2641 20:13:06 UTC - mysqld got signal 6 ;
  9. 9. #cpgethightech #cpanelconf Data Dictionary Errors Why do they occur? What are your options? Deleted or moved ibdata1 file Migrated from another environment Copies are made, but extensions are left intact Files are backed up directly, then restored Restore from a backup Recover an ibdata1 file Try to establish a copy of the data from a dump or table copy Recover the CREATE TABLE statement. re-create the table.
  10. 10. #cpgethightech #cpanelconf Alchemy Solutions WARNING Before attempting any alternative recovery solutions, always make sure to back up your MySQL data in whatever way possible: Perform MySQL dumps (when possible): mysqldump –AER > /root/dump.sql Make a copy of the MySQL data directory (or at least the ibdata* and ib_log* files): cp –Rp /var/lib/mysql /var/lib/mysql.bak
  11. 11. #cpgethightech #cpanelconf Alchemy Solutions Hex Editing Name Byte Length Offset Description FIL_PAGE_SPACE_OR_CHKSUM 4 0 Checksum (Space ID in older MySQL versions) FIL_PAGE_OFFSET 4 4 Page Number FIL_PAGE_PREV 4 8 Previous Page (in key order) FIL_PAGE_NEXT 4 12 Next Page (in key order) FIL_PAGE_LSN 8 16 LSN of page’s latest log record FIL_PAGE_TYPE 2 24 Page Type FIL_PAGE_FILE_FLUSH_LSN 8 26 Flushed-up-to LSN (only in space ID 0, page 0) FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID 4 34 Space ID InnoDB data is frequently stored in predictable, usable patterns at the byte level, in a page-by-page format, each page representing a 16384-byte block of data. Using this, you can directly modify a number of values in files stored by InnoDB to affect their behavior. storage/innobase/include/fil0fil.h
  12. 12. #cpgethightech #cpanelconf Alchemy Solutions Data dictionary example scenario:Space ID conflict shows in the error log ( /var/lib/mysql/`hostname`.err ): Error: tablespace id in file ‘dictionary.ibd' is 12, but in the InnoDB data dictionary it is 6. -bash-4.1# xxd -ps -s 34 -l 4 dictionary.ibd 0000000c -bash-4.1# xxd -ps -s 38 -l 4 dictionary.ibd 0000000c -bash-4.1# echo $((0x0c)) 12 Hex value stored in two primary locations in the .ibd file:
  13. 13. #cpgethightech #cpanelconf Alchemy Solutions Data dictionary example scenario:Writing the new space ID to the appropriate byte locations in the tablespace file (.ibd file): -bash-4.1# cd /var/lib/mysql/roundcube -bash-4.1# printf ‘x00x00x00x06’ | dd of=dictionary.ibd bs=1 seek=34 count=4 conv=notrunc -bash-4.1# printf ‘x00x00x00x06’ | dd of=dictionary.ibd bs=1 seek=38 count=4 conv=notrunc The stored space ID within this tablespace file is now “6”, in the primary two locations. Once the table is successfully loaded back into MySQL, be sure to make a dump, drop the table, and re-import that dump to make sure everything has properly and “organically” been added back into the data dictionary and MySQL metadata.
  14. 14. #cpgethightech #cpanelconf Page Corruption Errors Why do they occur? What are your options? Power failure / Physical failure Hardware failure Driver / Kernel bug File system write errors No built-in mechanism to “repair” a corrupted page MySQL may be offline Logged information can be ambiguous or unspecific Backups may not be available
  15. 15. #cpgethightech #cpanelconf mysql> SELECT COUNT(1) FROM customer_corrupt; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... Connection id: 1 Current database: testdb +----------+ | COUNT(1) | +----------+ | 599 | +----------+ 1 row in set (0.00 sec) You might see something like..
  16. 16. #cpgethightech #cpanelconf Alchemy Solutions Page corruption example scenario: 140911 13:21:36 InnoDB: Page checksum 4116551042, prior-to-4.0.14-form checksum 351785139 InnoDB: stored checksum 4003879582, prior-to-4.0.14-form stored checksum 2163901611 InnoDB: Page lsn 3886914220 2890537894, low 4 bytes of lsn at page end 12867924 InnoDB: Page number (if stored to page already) 2948225628, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 3390548810 … 140911 13:21:36 InnoDB: Error: space id and page n:o stored in the page InnoDB: read in are 3390548810:2948225628, should be 236:8! InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 8.40 4116551042 351785139 4003879582 2163901611 2948225628 3390548810 3390548810:2948225628 236:8 Space ID (and potentially page ID) hex edit method can potentially be applied here as well, however the checksum will still need to be adjusted.
  17. 17. #cpgethightech #cpanelconf Alchemy Solutions Hex Editing Name Byte Length Offset Description FIL_PAGE_SPACE_OR_CHKSUM 4 0 Checksum (Space ID in older MySQL versions) FIL_PAGE_OFFSET 4 4 Page Number FIL_PAGE_PREV 4 8 Previous Page (in key order) … … … … So we know how to correct that space ID, and we can probably translate that over to changing the page number (referred to as “page offset” in the source) as well. How about changing the checksum? The thing about doing any kind of manual changes to the physical InnoDB files is that this method will ALWAYS produce a checksum mismatch, because any byte changes in the file will of course affect the calculated checksum. So not only can updating the checksum alone be one way to try getting back some quick access to your table in a serious recovery scenario, but it is in fact a required final step to doing any byte manipulation with InnoDB. storage/innobase/include/fil0fil.h
  18. 18. #cpgethightech #cpanelconf Alchemy Solutions Hex Editing storage/innobase/include/fil0fil.h FIL Trailer (8) Name Byte Length Offset Description FIL_PAGE_END_LSN 8 16376 Low 4 bytes: Checksum, Last 4 bytes: FIL_PAGE_LSN In order to do this properly, we’ll need to be aware of one other location in which the checksum for each page is stored – the trailer. This is an 8-byte set of values that are stored at the end of each 16384-byte page, meaning that It begins at byte offset 16376 of a page, and ends with the byte at offset 16383. The checksum stored here is a little different, as it is considered the old-style (pre-4.0) checksum. This is stored here simply as a secondary means of verification, along with the primary, “new-style” checksum at the beginning of the page.
  19. 19. #cpgethightech #cpanelconf Alchemy Solutions Hex Editing InnoDB: Page checksum 4116551042, prior-to-4.0.14-form checksum 351785139 InnoDB: stored checksum 4003879582, prior-to-4.0.14-form stored checksum 2163901611 # printf '%Xn' 4116551042; printf '%Xn' 351785139 F55D9582  Primary calculated checksum 14F7D0B3  “Old-style” calculated checksum # expr 16384 * 8  Example Page 8 131072  Starting byte offset for Page 8 Writing the primary calculated checksum over the stored value of page 6: # printf ‘xF5x5Dx95x82’ | dd of=table.ibd bs=1 seek=98304 count=4 conv=notrunc
  20. 20. #cpgethightech #cpanelconf The Future • MariaDB defaulting with XtraDB • Aria being developed to eventually be a combination of MyISAM and InnoDB’s key features • InnoDB features being developed to allow more flexibility and fewer limitations on the way that the file system can be utilized, or that tablespace files can be imported. • Data dictionary being revamped.
  21. 21. #cpgethightech #cpanelconf Online Courses A 3-unit advanced course on InnoDB is currently being developed for our cPanel University learning environment. The first one, which is based on last year’s InnoDB Anatomy presentation, is available online now for preview access by using the group code: cPanelConf2015
  22. 22. Questions #cpanelconf
  23. 23. Contact #cpanelconf Email: ryan.robson@cpanel.net Twitter: cPanelRyan LinkedIn: https://www.linkedin.com/in/robworks

×