Wordcamp Ottawa WP Big Mig

ivan@Kruchkoff.com https://github.com/ivankruchkoff/wp-bigmig
Enterprise Migrations
How to move 1 MILLION+ posts from ANY CMS to
WordPress

Ivan@Kruchkoff.com https://github.com/ivankruchkoff/wp-bigmig
About Me
• Engineer at 10up and yes, we're hiring
• Work and travel
• ivan@kruchkoff.com
• @ivan on WP Slack

10up Engineering Best
Practices
• https://10up.github.io/Engineering-Best-Practices/
migrations/
• Includes things like:
• wp_defer_term_counting( true )
• deﬁne( 'WP_POST_REVISIONS', 0 )
• Using php 7.1
• deﬁne( 'WP_IMPORTING', true )

WordPress Single to
Multisite
• https://github.com/10up/MU-Migration/
• Get as many single sites as you need into a multi
site.
• Handle deltas by remigrating.
• Gotchas: Theme Mods and custom tables

The Problem Statement
• Acme corp is frustrated using BoringCMS:
• Editorial ﬁnd the interface horrible to use
• The performance is incredibly slow
• Countless hours spent keeping the CMS running
• They've evaluated other options and have decided
on WordPress.
• Your team is responsible for getting them there.

Our Aim
• Migrate a live site containing over a million posts
• Migrate all assets (e.g. images)
• Run two CMSs simultaneously for a period of time
• Make delta migrations

Our Approach
• while ( WordPress not live in Prod() ) {
• export new or updated posts from BoringCMS()
• export new or updated assets to S3()
• update and improve posts parser()
• import posts()
• }

The 3 CMS stages
• BoringCMS live in prod
• BoringCMS is still live, WordPress is ready in prod
• WordPress live in Prod

BoringCMS in prod
• No WordPress
• WordPress development in progress.
• Content snapshot in WordPress
• WordPress development complete
• Latest content in WordPress

BoringCMS is live, WP is
ready in prod
• Editorial in BoringCMS
• Latest content in WordPress
• Editorial in WordPress

WP Live in Prod
• Proﬁt
• :partyparrot:

Export new or updated posts
from BoringCMS()
• Fetch all posts
• Fetch posts updated since
• Fetch posts with min/max id

Export new or updated
assets to S3()
• Assets is just a fancy way of saying images + more.
• Fetch all assets
• Fetch all assets created / modiﬁed since
• Use s3-parallel-put for performance

Update and Improve Posts
Parser
• Start with a few content exports, ideally as JSON
• Find your unique identifier (ID / File Path)
• Parse the wp_posts fields first:
• post_author, post_date/post_modified, post_content ,
post_title, post_name, post_parent, guid, post_type
• Log all the things
• Any time an error occurs, pause handling this file, log a
message, move to /investigate, continue to next file.

Import Posts
• First few runs, ignore the data.
• Create four folders:
• pending - any file we haven't yet read / processed
• processed - files that were successfully imported into WP.
• ignored - files that aren't being handled by WP
• investigate - files that should be handled by WP, but threw an
error.
• Log every step

Logging
• https://github.com/jbroadway/analog
• Processing $filename
• Error $descriptive_error_message in $filename
• Parsed $post_type in $filename
• Processed $filename as $post_id

Code Structure
• WP Cli command
• File parser, this takes a text ﬁle and makes it a PHP
data type (json_decode( ﬁle_get_contents( $fn) )
• Content Parser, marshalls data from old format into
something that can be used to create a post.
• Post, contains params for wp_insert_post()
post_meta array and taxonomy terms

Example Dataset
• Thanks to the team at Web Hose
• https://webhose.io/datasets

Look at 1 post
• news_0003607.json
• How do I get the post title? (thread -> title)
• How do I get the post slug? ( thread -> uuid? )
• How do I get the post content? ( text )
• How do I get taxonomy? (entities)

Migrate Overview
• Enable plugin
• Add your parser* to ExportFilesIterator ->
maybe_insert_content()
• Setup your dir structure (e.g. your content is in ./
exports)
• mkdir -p migrate/pending/123 && mv exports migrate/
pending/123
• wp bigmig -f migrate

Your JSON parser
/** 
* Parse a json file, and if it's valid call @maybe_insert_content 
* @param $filename 
* 
* @return bool 
*/ 
function parse_json_file( &$filename ) { 
$content_string = file_get_contents( $filename ); 
$content = json_decode( $content_string, true); 
if ( ! is_array( $content ) ) { 
Logger::log( "Could not parse JSON file into array in {$filename}",
Analog::ERROR ); 
FileMover::move_to_investigate( $filename ); 
return false; 
} 
$this->maybe_insert_content( $content, $filename ); 
}

Create
WebHoseArticleContentParser
<?php 
namespace WP_CLIBigMigParsers; 
 
class WebHoseArticleContentParser { 
 
protected $filename; 
protected $parsed_content = array(); 
protected $content; 
 
/** 
* Parse a news article json file, returns an array for creating a story WP_Post. 
*/ 
public function parse() { 
$this->set_title(); 
$this->set_body(); 
 
return $this->get_parsed_content(); 
} 
 
public function __construct( &$content, &$filename ) { 
$this->content = $content; 
$this->filename = $filename; 
} 
 
public function get_parsed_content() { 
return $this->parsed_content; 
} 
 
function set_title() { 
$this->parsed_content[ 'title' ] = $this->content['title']; 
} 
 
 
}

Create a
WebHoseArticlePost
<?php 
namespace WP_CLIBigMigPosts; 
 
use WP_CLIBigMigLogger; 
 
class WebHoseArticlePost extends Post { 
 
public function prepare() { 
$this->type = 'WebHoseArticle'; 
$this->set_insert_param_from_content( 'post_content', 'body' ); 
$this->set_insert_param_from_content( 'post_title', 'title' ); 
 
$this->insert_params[ 'meta_input' ] = $this->meta_input; 
} 
}

Look at your logs
• localhost - 2017-07-22 19:35:34 - 1 - Now processing /mnt/tmpfs/
migrate/pending/articles/654/news_0000111.json, processed 1 ﬁles.
• localhost - 2017-07-22 19:35:34 - 1 - Moved /mnt/tmpfs/migrate/
pending/articles/654/news_0000111.json to /mnt/tmpfs/migrate/
processed/654/news_0000111.json status: processed
• localhost - 2017-07-22 19:35:34 - 1 - Finished processing content
export /mnt/tmpfs/migrate/pending/articles/654/news_0000111.json
as WP Post 30.
• localhost - 2017-07-22 19:35:34 - 1 - Now processing /mnt/tmpfs/
migrate/pending/articles/654/news_0000110.json, processed 2 ﬁles.

Skip the boring bits
• Content will be broken.
• Tax mapping
• Edge cases.
• That custom format they only used in 1998 for 3
months.
• That 1 article type that is XML

Speeding Up the Process
• What are your bottlenecks? Latency everywhere
• Reading from a local disk / networked disk / ram
• Reading / writing from a local db / networked db.
• Replication to read nodes.
• Single Process
• Term counting etc. - look at 10up Best Practices :)

Speedups
• Provision your machine with lots of ram
• defaults['memory'] = 8192 for VVV
• Make a ramdisk
• sudo mount -o size=1G -t tmpfs none /mnt/tmpfs
• Parallelise your processes
• dirsplit -m -e1 -s 10M migrate
• Gnu Parallel, Bash, Gearman, pcntl_fork()

A Demo?
• https://github.com/ivankruchkoff/wp-bigmig

Questions?
• ivan@kruchkoff.com

Wordcamp Ottawa WP Big Mig

Recomendados

Recomendados

Mais conteúdo relacionado

Último

Último (20)

Destaque

Destaque (20)

Wordcamp Ottawa WP Big Mig