O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

Migrating data to Drupal 8

Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Próximos SlideShares
GeoServer Developers Workshop
GeoServer Developers Workshop
Carregando em…3
×

Confira estes a seguir

1 de 35 Anúncio

Migrating data to Drupal 8

Baixar para ler offline

Recorrido por el framework de migraciones de Drupal 8.

Empezaremos revisando las opciones que proporciona Drupal (core y contrib) para realizar migraciones automáticas desde Drupal 5, 6 o 7, o Wordpress.

Continuaremos exponiendo algunos escenarios complejos en los que se requiere implementar migraciones a medida para filtrar y procesar los datos de origen (de Drupal o fuentes diversas) antes de volcarlos en destino.

Por último, veremos algunas estrategias que hemos usado para ejecutar la migración de grandes volúmenes de datos.

https://2017.drupalcamp.es/sessions/migrando-datos-drupal-8-jonhattan

Recorrido por el framework de migraciones de Drupal 8.

Empezaremos revisando las opciones que proporciona Drupal (core y contrib) para realizar migraciones automáticas desde Drupal 5, 6 o 7, o Wordpress.

Continuaremos exponiendo algunos escenarios complejos en los que se requiere implementar migraciones a medida para filtrar y procesar los datos de origen (de Drupal o fuentes diversas) antes de volcarlos en destino.

Por último, veremos algunas estrategias que hemos usado para ejecutar la migración de grandes volúmenes de datos.

https://2017.drupalcamp.es/sessions/migrando-datos-drupal-8-jonhattan

Anúncio
Anúncio

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a Migrating data to Drupal 8 (20)

Anúncio

Mais recentes (20)

Migrating data to Drupal 8

  1. 1. MIGRATING DATA TO DRUPAL 8 Jonathan Araña Cruz #DrupalCampES2017.drupalcamp.es
  2. 2. about:me Jonathan Araña Cruz / jonhattan ● +10 years Drupal ● +10 years sysadmin ● Drush maintainer. Other contribs ● Infrastructure as code ● Indentation, syntax highlight addict
  3. 3. Topics ● Migrate & Drupal 8 ● Automagic migrations ○ Limitations ● Custom migrations ○ Anatomy of migrations ○ Development workflow ○ Solving hard problems ● Migrating large volumes of data ● Tips & Tricks
  4. 4. Well known solution in contrib (D6, D7) Drastically changed although philosophically the same Migrate contents and configuration settings New upgrade path for major version upgrades (D6,7->D8) Migrate & Drupal 8
  5. 5. Automagic migrations ● Drupal-to-Drupal ● Wordpress-to-Drupal ● Other sources
  6. 6. Drupal 5, 6 or 7 to Drupal 8 ● Install a fresh Drupal 8 ● Download and enable Drupal 8 version of your modules ● Enable core modules: Migrate Drupal UI, Migrate Drupal, Migrate ● Visit /upgrade (Drupal 5 requires Drupal 5 migration (migrate_drupal_d5) module --only supports terms, users and nodes) Automagic migrations »
  7. 7. Via web Automagic migrations » Drupal 5, 6 or 7 to Drupal 8
  8. 8. With Drush drush dl migrate_upgrade drush en migrate, migrate_drupal, migrate_upgrade drush migrate-upgrade --legacy-db-url=mysql://user:pass@12.34.56.78/d6db --legacy-root=http://myd6site.com ● Provided by Migrate Upgrade (migrate_upgrade) Automagic migrations » Drupal 5, 6 or 7 to Drupal 8
  9. 9. Limitations ● Preserves auto-incremental ids ● Fresh install. One shot. ○ Overwrites existing config ○ No rollbacks ○ No incremental migrations ● 1-1 Migration (attempt) ○ Source structure is literally replicated ● Many modules not covered (imce , ubercart/commerce, ...) Automagic migrations » Drupal 5, 6 or 7 to Drupal 8
  10. 10. Wordpress to Drupal 8 Two modules available: ● Wordpress Migrate (wordpress_migrate) ○ Migrates WordPress blog exports (WXR format) ○ Posts, pages, comments, attachments, tags and categories ● WP Migrate (wp_migrate) ○ SQL: comments, posts, terms, users, vocabularies ○ Half baked Automagic migrations »
  11. 11. From other sources to Drupal 8 ● SQL, json, xml, csv, excel, … ● No automation ● Helper module Migrate UI (migrate_ui) - outdated with Drupal 8.3.x Automagic migrations »
  12. 12. Custom migrations
  13. 13. Why? ● Limitations of automagic migrations: ○ One-shot ○ 1-1 migration ○ Unsupported modules ● Incremental migrations, updates, rollbacks ● Opportunities: ○ New site structure ○ Transformation of contents ○ Migrate from diverse sources ○ ... Custom migrations »
  14. 14. How to ● Migrations are config ● Quickstart: pick & tweak automagic migrations ● Migrate Plus (migrate_plus) provides config entities, hooks, events... ● Put your migrations in a module: ○ mymodule/config/install/migrate_plus.migration.articles.yml Custom migrations » drush migrate-upgrade --legacy-db-url=mysql://user:pass@12.34.56.78/d6db --legacy-root=http://myd6site.com --configure-only
  15. 15. Anatomy of migrations (I) ● A migration config consists of: ○ ID, label,... ○ Migration source ○ Migration destination ○ Fields mapping / process ■ Describes, property-by-property, how the destination is to be constructed from the source data Custom migrations »
  16. 16. Anatomy of migrations (II) id: enredate_teachers__user label: 'enredate.org teachers - user entity' migration_group: enredate migration_dependencies: { } source: plugin: csv path: 'private://migrate/sources/crm-teachers.csv' header_row_count: 1 delimiter: ';' column_names: - {Nombre: 'Nombre'} - {Email: 'Email'} - {CodWebEnredate: CodWebEnredate} - {UserNameEnredate: UserNameEnredate} - {PassEnredate: PassEnredate} keys: ['CodWebEnredate'] Custom migrations »
  17. 17. Anatomy of migrations (II) destination: plugin: 'entity:user' md5_passwords: true process: name: UserNameEnredate pass: PassEnredate mail: Email status: plugin: default_value default_value: 1 roles: roles Custom migrations »
  18. 18. Source plugins (I) ● Core modules provide source plugins for Drupal 6 and 7 elements ○ d6_node, d7_node, user, file, menu, block, ... ● Contrib: ○ Migrate Plus (migrate_plus) - url source and parsers for xml, json, soap ○ Migrate Source CSV (migrate_source_csv) ○ Migrate Google Sheets (migrate_google_sheets) ○ Commerce Migrate (commerce_migrate) - ubercart and commerce sources Custom migrations » Anatomy of migrations »
  19. 19. Source plugins (II) ● @MigrateSource annotation ● Plugin/migrate/source namespace ● Some core interfaces and base classes: ○ MigrateSourceInterface ○ Node < FieldableEntity < DrupalSqlBase < SqlBase < SourcePluginBase Custom migrations » Anatomy of migrations »
  20. 20. Destination plugins (I) Custom migrations » Anatomy of migrations » ● Core modules provide destination plugins for Drupal 8 ○ entity:user, entity:node, url_alias… ● Contrib modules provide their destinations as needed ○ field_group, entity_reference_revisions(paragraphs), externalauth, redirect...
  21. 21. ● @MigrateDestination annotation ● Plugin/migrate/destination namespace ● Some core interfaces and base classes: ○ MigrateDestinationInterface ○ EntityUser < Entity < DestinationBase Destination plugins (II) Custom migrations » Anatomy of migrations »
  22. 22. Process plugins ● Core provides a bunch of plugins: ○ get, callback, concat, default_value, iterator, machine_name, migration_lookup, skip_on_empty, skip_row_if_not_set, static_map, substr,... ● Contrib modules provide their process plugins as needed: geofield, migrate_process_map... Custom migrations » Anatomy of migrations »
  23. 23. Examples (I) get default_value static_map langcode: plugin: static_map map: _esp: es _cat: ca _eus: eu _gal: gl source: language mail: Email mail: plugin: get source: Email init_mail: ‘@mail’ status: plugin: default_value default_value: 1 Custom migrations » Anatomy of migrations » Process plugins
  24. 24. Examples (II) callback migration_lookup skip_on_empty field_related/target_id: - plugin: skip_on_empty source: relacionados - plugin: migration migration: - abo__news - abo__videos name: plugin: callback callable: trim source: UserNameEnredate field_related/target_id: plugin: migration migration: - abo__news - abo__videos source: relacionados Custom migrations » Anatomy of migrations » Process plugins
  25. 25. Pipeline Custom migrations » Anatomy of migrations » Process plugins process: foo: - plugin: concat delimiter: '_' source: - source1 - source2 - plugin: callback callable: striptags - plugin: callback callable: trim - plugin: skip_on_empty - plugin: machine_name
  26. 26. Development workflow ● Code - Execute (import, update) - Debug - Rollback. Cleanup ● You want a runner to execute migrations: ○ Migrate Tools (migrate_tools) - Drush commands & UI ○ Migrate Run (migrate_run) - lightweight fork of Migrate Tools ○ Migrate manifest (migrate_manifest) Custom migrations »
  27. 27. Debugging ● Migrate Devel (migrate_devel) - extends Migrate Tools ● Drush Entity (drush_entity) - Inspect entities ● Migrate messages and maps ● var_dump() Breakpoints: ○ DrupalmigrateMigrateExecutable::processRow ○ DrupalmigratePluginmigratedestinationEntityContent::import ● Dummy fields Custom migrations » Lifecycle dummy: plugin: callback callable: drush_print_r source: source_field
  28. 28. Solving hard problems ● Auxiliary and compound migrations ● Massaging the body ○ Cleanup ○ Convert tokens ○ Get rid of <img> ● Preparation of files and videos ● Redirections in HAProxy Custom migrations »
  29. 29. Migrating largevolumes of data
  30. 30. What we mean by «large» ● 150k files ● 20k videos ● 60k nodes ● 8k terms ● 30k users ● 1M comments Large migrations can take hours to run. Days to review. ~9 hours in our best effort Migrating large volumes of data »
  31. 31. Migrating on steroids ● PHP 7.x & Redis ● MySQL datadir and tmpdir on ramdisk ● MySQL indexes ● Run large migrations in chunks ● Parallelized migrations ● Lightweight database. Drush Shrinkdb (drush_shrinkdb) ● Previous download Migrating large volumes of data »
  32. 32. Tips & Tricks First one: “before [...] you need to stop and install your brain” @hernanibf “My site is slow” Drupalcamp Spain ‘12
  33. 33. Tips & Tricks ● Ponder. Not everything deserves being migrated automatically ● Prioritize. Not all data have the same value ● Automate & measure. Identify regressions and opportunities to improve ● Use KPIs to measure the quality of the migration ● Test the most important urls
  34. 34. Tips & Tricks ● Order matters ● Ensure consistent source database state ● Use a MySQL read only user to connect the sources ● Don’t do it in production. Prevent performance impact or data loss/corruption ● Test run in a production-like environment. Avoid surprises the M-day

×