1. 1Samsung Open Source Group
Stefan Schmidt
Samsung Open Source Group
s.schmidt@samsung.com
QA: Where We Are And Where We
Should Go
EFL Developer Day US 2015

2. 2Samsung Open Source Group
Agenda
●
Current QA Summary
●
Missing QA Parts
●
Steps Forward
●
Q&A Topics

3. 3Samsung Open Source Group
Current QA Summary

4. 4Samsung Open Source Group
Current: Compile Time Testing
●
Developers machine (One arch, make,
hopefully make check and distcheck)
●
Machines of other developers
●
Jenkins covers different architectures,
different compilers, different
dependencies and versions

5. 5Samsung Open Source Group
Current: Unit Testing
●
Test cases collected in test suite and
run with make check
●
Should be run before committing
●
Also run by Jenkins
●
While unit testing is the main idea it also
had catched segfaults due to basic run
time testing of new code

6. 6Samsung Open Source Group
Current: Miscellaneous
●
Static Analysis
●
Different EFL profiles
●
Different configuration options
(Wayland, XCB)
●
Different make targets (all, check,
distcheck, doc)

7. 7Samsung Open Source Group
Missing QA Parts

8. 8Samsung Open Source Group
Missing: Runtime Testing
●
We rely on the developer testing his code
before committing
●
We rely on the community of developers
and users to run the latest code and
report bugs
●
No automation

9. 9Samsung Open Source Group
Missing: Graphical Unit Tests
●
No usage of graphical test cases
●
Tooling is there: Excatness
●
Almost no developer uses it, regularly
●
Automation is highly needed

10. 10Samsung Open Source Group
Missing: Code Coverage
●
Only 39.2% of our code base covered by u
nit tests
●
Newly added API's should be
mandantory to come with unit tests
●
My goal would be that we increase coverag
e 5-10% per release
●
How to motivate unit test writing?

11. 11Samsung Open Source Group
Missing: Pre-push CI
●
CI infrastructure only available post-push to master
●
The various tooling only runs on code committed
to the master branch of our repos
●
Testing the code before this stage could be
beneficial
●
Testing developer branches on demand and
hooking it up with Phab to run it on incoming
submissions

12. 12Samsung Open Source Group
Missing: Code Complexity
●
More valid combinations for test matrix
– New jobs for xcb and wayland have
just been added
●
What other configuration options should
be officially supported?

13. 13Samsung Open Source Group
Steps Forward

14. 14Samsung Open Source Group
Steps Forward: Unit Testing
●
Proposal: Make test cases for public API
additions (EAPI) mandatory from today
●
Aim for 5% coverage increase in every
upcoming release
●
Aim for 60 - 70% coverage at the end of
2015
●
Not only NULL checks, but valid test cases

15. 15Samsung Open Source Group
Steps Forward: Graphical Testing
●
Identify the problems with exactness on
Jenkins
●
Find a way to fix or work around them
●
Run exactness during the nightlies
●
So far screenshots for elementary_tests
and some edje examples.
●
What else should be tested?

16. 16Samsung Open Source Group
Steps Forward: API's and Breaks
●
Run ABI/API checker every night
●
Catch breaks
●
Catch new additions
– Summary mail to list to highlight additions and maybe
raise discussion if needed. (Shortly after the addition a
nd not weeks later)
– Summary of new additions for release NEWS
– Notifications for manual bindings to catchup
– Check if new APIs have documentation

17. 17Samsung Open Source Group
Steps Forward: Pre-push CI
●
Two entry points we should cover:
– Developers: Allow dev branches to be run
through Jenkins
– Contributors: Run incoming phab submissions
through Jenkins and mark them as
passed/failed
●
Both could have the same Jenkins jobs at the
backend

18. 18Samsung Open Source Group
Q&A Topics

19. 19Samsung Open Source Group
Q&A Topics
●
Mandatory unit tests for new public APIs?
●
5% coverage increase per release a sane goal?
Who would help with this work?
●
How to motivate the writing of unit tests?
●
Anyone willing to drive the exactness integration
work until the end?
●
Is there interest for build hooks for phab
submissions?

20. 20Samsung Open Source Group
Thank you.

21. 21Samsung Open Source Group
We are hiring.
jobs@osg.samsung.com

22. 22Samsung Open Source Group
Attributions
● Brunoschalch:
https://www.flickr.com/photos/bsgphotos/7594903774/in/ph
otostream/
● Katja Schulz:
https://www.flickr.com/photos/treegrow/14404553548
● Fernando Flores: https://www.flickr.com/photos/ferjflores/86
97860914
● David Brossard: https://www.flickr.com/photos/string_bass_d
ave/8152835443
● Anne Reeves:
https://www.flickr.com/photos/charlock/10677411025
● Andrew: https://www.flickr.com/photos/polandeze/7413117
58