San Francisco Magento Meeting 2014-03-12

1. On all Fronts
Testing
Magento
San Francisco Magento Meetup Group - March 11, 2014

2. Fabrizio
BrancaLead System Developer at

3. E-Commerce:
Magento
CMS:
TYPO3
Portals:
ZF, FLOW,…
Mobile Searchperience:
SOLR
120 people
in 7 offices
world-wide
High
Performance
/Scale
Global
Enterprise
Projects

4. Unit
Service
UI
Acceptance
http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
What the
customer thinks
the code should do
What the
developer thinks
the code should do
Load
Test

5. Continuous
Integration
Deployment
Development
Delivery
Improvement

6. Quality Assurance
Continuous
Everything sounds better
with “continuous”

7. “Continuous”
==“Doingstuffoverandoveragain”
Builds
Deployment Tests Infrastructure
Setup
Devbox
Setup
== “Automation”

8. timespent
task size
does it
manually
does it
manually
gets annoyed
writes script to
automate
runs script
wins
loses
Geeks vs. Non-Geeks
Makes fun of geek’s
complicated method

9. — Ovidiu Platon
We are not shipping your machine!
if it works on your machine!
I don’t care

10. Picture
The BIG

11. Vagrant
Boxes

12. Awesome
stuff
Every single
line potentially
can break
production!
Bratwurst
(Germans)
Donut
(Americans)
Pizza
(Italians)

13. Version Control
GIT, SVN,…
PHP lint

14. Version Control Code Reviews
Gerrit, Crucible,
Phabricator,…

15. Code Reviews Build
Files
DB
Installer
Settings
Triggered by
commit, by time
or manually
Jenkins
minify js/css
.tar.gz
“virtual package”
(revision)

16. Version Control Code Reviews Build
Files
DB
Installer
Settings

17. Build
Files
B
ler
Settings
Static Code
Analysis
CodeSniffer
php-pmd
php-cpd
php-depend

18. Static Code
Analysis
Install on
“Integration”
First system where
code from all
developers/teams
meets each other
unstable system.
Might be broken.
Will be rebuild
without warning

19. Install on
“Integration”
Unit Tests
PHPUnit
Code
coverage

20. Unit Tests Integration Tests
Test features that rely on other
parts (e.g. framework, external
services,…)
External components
(SSL, Redis, Varnish,…)
Basically this is also
PHPUnit, but
addressing other stuff

21. Integration Tests Acceptance Tests

22. Acceptance Tests
Install on
“Staging”
Behat
(Cucumber)
Same setup as production,
but different machines
Client QA /
approval

23. Install on
“Staging”
*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning
Stress
Tests
Jmeter
Spin up EC2
instances in the
cloud to produce
traffic*

24. Stress
Tests
Install on
“Production”

25. Install on
“Production”
Enable Features

26. Version
Control
Code
Reviews
Build Static Code
Analysis
Install on
“Integration”
Unit Tests Integration
Tests
Acceptance
Tests
Install on
“Staging”
Install on
“Production”
Enable
Features

27. with EcomDev_PHPUnit
Unit Tests

28. https://github.com/EcomDev/
EcomDev_PHPUnit
use the
“dev”
branch!

29. Config Model Block Helper Controller
fat model,
skinny controller…

30. Classes
PHPUnit_Framework_TestCaseEcomDev_PHPUnit_Test_Case
EcomDev_PHPUnit_Test_Case_Config EcomDev_PHPUnit_Test_Case_Controller

31. Assertions

32. Fixtures

33. Mocks

34. Data Provider

35. Magento Integration
Testsuite
management
Framework
integration
Create test
environment
(db+config)

36. Manual
http://www.ecomdev.org/2011/05/24/ecomdev_phpunit-manual-version-0-2-0-is-available-for-
download.html
Video tutorial
http://www.ecomdev.org/2012/01/31/test-driven-magento-development-seminar-video-and-
code.html
Documentation
More Information

37. With Selenium
Acceptance Testing

38. Acceptance Tests

39. What’s out there?
Selenium for PHP
PHPUnit_Selenium
(Sebastian Bergmann)
WebDriver-PHP
(Chibimagic)
Php-webdriver-bindings
(Lukasz Kolczynski)
php-webdriver
PHPUnit_Selenium
(Giorgio Sironi)

40. php-webdriver
Selenium 2
08/2011 Facebook/php-webdriver (Justin Bishop)
Element-34/php-webdriver (Adam Goucher)
instaclick/php-webdriver (Anthon Pang)

41. php-webdriver
“A very thin wrapper
of WebDriver”

42. So what?
A thin layer good!
But not enough…
no test logic
no higher level
components
dealing with
Selenium specific tasks
not compatible to
legacy test cases
No abstraction /
structure

43. Architecture
HTTP
AUT
No direct access
(DB, classes,…)
REST*
(over HTTP)
*http://code.google.com/p/selenium/wiki/JsonWireProtocol

45. Project test cases
PHPUnit
Project Components
Components Library
Menta
php-webdriver
reusable
project
specific

46. Menta
Selenium 2 Testing Framework for PHP
Who is
“Menta”?
What is
“Menta”?

47. Features
MentaThe framework.
Not the cat…
Components Session Management
?Bonus Features :)Event / ObserverConfiguration Management
PHPUnit integration

48. Why?
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
Data Providers
✔
✔
✔
✔
✔
✔
✔
✔
✔
Dependent Tests
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
Test Infrastructure

49. Menta for PHPUnit
Abstract test class
PHPUnit_Framework_TestCaseMenta_PHPUnit_Testcase_Selenium2ProjectY_Abstract_TestCase
Very thin layer on
top of PHPUnit
ProjectY_TestCase_Registration ProjectY_TestCase_Checkout
the “magic” happens
in the components

50. HTML ReportScreenshots
(on demand
and on failure)
test description
from phpdoc
additional
messages
error details

51. HTML Report“Duration
Heatmap”
shows long
running tests

52. Menta for PHPUnit
Text Result PrinterPHPUnit 3.7.20 by Sebastian Bergmann.
Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml
> SUITE:
> SUITE:
> SUITE: Acceptance_Tests_Account_Register
> TEST: Acceptance_Tests_Account_Register::register
SUCCESS. (Duration: 8.18 sec)
> TEST: Acceptance_Tests_Account_Register::login
SUCCESS. (Duration: 6.5 sec)
> TEST: Acceptance_Tests_Account_Register::testRegistrationMail
FAILURE: Searching for mail with subject 'Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae' timed out (Duration: 22
< Duration: 36 sec
< Duration: 36 sec
> SUITE:
> SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest
> TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCart
FAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat(' ', @class, ' '), ' attentionText ')]" does not contain text "Bulb Vase" Fail
< Duration: 9.87 sec
> SUITE: Acceptance_Tests_Checkout_CheckoutTest
> TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryView
SUCCESS. (Duration: 1.79 sec)
> TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart
FAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec)
SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec)

53. Components
Common Assert Wait Screenshot Selenium1Facade
Imap WebService W3C Validator Curl …
Some might not even
require a Selenium session

54. Components!
PageObjects?
PageObjectsComponents
{ }}⊇{ PageObjects are
components

55. PageObjects?
represents
services offered by
a page or feature
is the only thing with
a deep knowledge of
the HTML
clickCheckoutButton() isInCart($product) getTotals()
✔
✔
✔

56. Component
Libraries
Menta
Magento
(your library)
Project-
specific
Magento
(community-provided)
General
(your library)

57. Gallery
Screenshot

58. Perceptual diffs
Build X
Build X-1
Perceptual
Diff

59. github.com / AOEmedia / Menta
github.com / AOEmedia / Menta_SampleProject
It’s composer based and
you’ can get started
in a few minutes!

61. With JMeter
Load Testing

62. JMeter
jmeter.apache.org
Current version:
2.11

63. JMeter
HTTP
Simple Setup

64. JMeter
Slave
JMeter
Slave
JMeter
Slave
JMeter
Master
Master/Slave Setup

65. JMeter
Slave
JMeter
Slave
JMeter
Slave
JMeter
Master
EC2 Setup
JMeter
Slave
JMeter
Slave
…

66. Creating
JMeter tests
JMeter
Test
Manually
Replaying Apache
Logs
Recording
Browser
interaction using
a proxy“Convert” existing
acceptance tests
CSV Files
(Selenium)
HTT

67. HTT
Behat Selenium Firefox Proxy
JMeter
Test
Converting
Behat to JMeter

68. Using a Proxy
JMeter

69. Using a Proxy
Browser configuration
Manual
configuration
Browser
Extension

70. Using a Proxy
Browser configuration

71. java -jar selenium-server-standalone-2.40.0.jar
-trustAllSSLCertificates
-Dwebdriver.firefox.profile=jmeter
Using a Proxy
Selenium configuration

72. http://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.pdf
https://wiki.apache.org/jmeter/TestRecording210
http://jmeter.apache.org/usermanual/component_reference.html#HTTP%28S%29_Test_Script
_Recorder
Using a
Proxy

73. https://jmeter.apache.org/usermanual/test_plan.html
• ThreadGroups
• Controllers
• Samplers
• Logic Controllers
• Test Fragments
• Timers
• Assertions
• Configuration Elements
• Pre-Processor Elements
• Post-Processor Elements
Elements of a
Test Plan

74. http://www.slideshare.net/shubelal/abj-meterbrief-upladed/7
Elements of a
Test Plan

75. • Tree structure
• Inheritance (Cookie Manager, Header Manager,…)
• Timers
• Variables
• Assertions
• Random values
• Scripting
• Extracting values
• Following redirects
• (check examples in bin/examples)
Basic Concepts

76. Workflows
Search / Filters
Static Pages
404
What to testConsider
conversion rate /
funnel!

77. JMeter
Plugins
jmeter-plugins.org

79. Distributed
load testing
General Master/Slave setup
https://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf
AOE presentations
http://www.slideshare.net/axeljung01/jmeter-and-amazon-cloud-workshop-t3dd2012
http://www.slideshare.net/aoemedia/performance-measurement-and-tuning
On EC2
https://github.com/oliverlloyd/jmeter-ec2
http://www.http503.com/2012/run-jmeter-on-amazon-ec2-cloud/

82. Server Monitoring

83. Thank you!
Any questions?
http://www.aoe.com
http://www.fabrizio-branca.de
@fbrnc
Follow me on twitter!
My blog