Cynthia Wu gave a presentation on software testing frameworks. She discussed what frameworks are, the benefits of using pre-fabricated frameworks, and how to set expectations for testing automation. Wu outlined guidelines for evaluating frameworks using three criteria: technical, management, and informative. She provided examples of how two popular frameworks, Cucumber and Protractor, measure up based on these criteria. Wu emphasized that no framework is perfect and organizations may need to create customized solutions. Her overall message was that testing automation requires ongoing learning and organizations should not expect immediate results or be afraid of failures along the way.
2. Agenda
● What is a framework? Why go Pre-Fab?
● How to set expectations (short/long-term)
● How to approach “testing” automation
frameworks
○ TMI Guidelines
● Aim for the win, but don’t be afraid to “lose”
2
3. “
3
a software framework is an
abstraction in which software
providing generic functionality...a
standard way to build and deploy
...universal, reusable...platform to
facilitate development of software
applications, products and solutions.
Thanks Wikipedia
5. 5
Why go with Pre-Fab?
● Saves time
● Test Case Management
● Test Execution Management
● Reporting
● CI Integration
● Extensibility
….so why does it hurt?
6. What they still lack
● The magic
● Connectivity to other tools/systems
● Extensibility
● Monitoring
6
8. 8
SW != LH
● Can I create a test?
○ Can it run in CI?
● Reporting on tests
● Can I create more
tests?
● Can I extend my
validations?
● Who is going to create
the tests?
● Who is going to
consume the reports?
● What are you trying to
prove?
● What kind of resourcing
are you/company willing
to invest in?
12. Technical
● Is it still being supported?
○ How good is the documentation?
○ What communities are there?
● How much control do users have in test execution?
● What languages(s) can we use with it?
○ Extensibility?
○ How reusable can we make our functions (magic)?
● What does it integrate with?
○ Maven/Gradle
12
13. Management
● Is it version friendly?
○ Can multiple people work on it actively and nicely?
○ Multiversion app - Multiversion framework
● Easy to organize?
○ Do you need special IDE plugins?
● How much time should be allocated to improvements?
○ Adaptivity to changes in testing requirements
○ Adaptivity to changes in business requirements
13
14. Informative
● How well can I inform consumers of the framework itself?
○ Logging
○ Documentation
● How useful are its reports?
○ How easily can I enhance those reports?
● How easy is it to communicate with the community?
○ Contribute
○ Get feedback
14
16. ● Actively supported
● Lots of execution options
● Almost all the languages
○ Emphasis on steps
being reusable
● Maven/Gradle support
Cucumber - Technical
● Need to know how to code
○ Have to build all the
integration and
support
● Some language support $$
● Easy to build brittle
framework
16
17. ● Somewhat “easy” to
version control
● Can be actively worked by
multiple people
● Recommended using an
IDE with plugins
Cucumber - Management
● Can be nightmare to
manage without oversight
● Can be difficult to adapt to
business/testing changes
○ Framework driven by
Gherkin rather than
test needs
■ Complicates
testing state
17
18. ● Monitor with whatever logging you desire
● Reports are lovely
○ Jenkins plugin is nice
● Custom reports relatively easy to create
Cucumber - Informative
18
19. Protractor - Technical
● Actively supported
● Execution options
configured in config file
● Only NodeJS
● Almost a framework
○ Needs to piggyback
off something
■ Mocha
■ Jasmine
● Focus on UI testing
○ Messy to extend to
anything outside of UI
19
20. Protractor - Management
● Can be version friendly
● Recommended to use an
IDE that has Javascript
support
● Can be difficult to work
concurrently
● Need time to overcome
browser testing fragility
20
21. Protractor - Informative
● Has its own report
extension
● Dependent on the report of
the framework that its
piggybacking
21
25. What did I learn this morning?
No Silver Bullet, Golden
Ticket…
Be sure to set
expectations with
yourself, your team,
and your stakeholders
How to evaluate
pre-fab frameworks
with TMI
There will be
something custom for
you
Don’t be afraid to fail -
try something new!
You will learn
something
There is no end to
automation - pace
yourself
25
26.
27. How to Contact Me
wu.x.cynthia@gmail.com
www.dcast.io
If you’re in DC check out the D-CAST
Meetup for local testing events!
