Bitbar hosted a local meetup in Helsinki where we discussed mobile performance metrics, performance testing/monitoring and using production data to optimize testing.

1. Bitbar
The Mobile DevOps
Company

2. Speaker
Lasse Häll
Solutions Consultant

3. Bitbar Technologies
• Bitbar Technologies is the company behind Testdroid
• Operations: Finland (Helsinki & Oulu), Unites States (San Jose,
CA) & Poland (Wroclaw)
• Founded in 2009 – 52 Employees – Customers in 30+ Countries
• Private Company – VC Backed – Qualcomm, DFJ, Creathor,
Inventure
• Provides Technology and Services For High Performance Mobile
SW Development and Testing Solutions Based On Open Standards
Customers include:

4. Mobile Testing Pain Points
• HW Fragmentation – Hundreds of
Permutations
• Device Acquisition & Management
• Inefficiency with Distributed Teams, App
Platforms, Tooling Preferences
• Slow Manual Testing
• Detailed Reporting

5. Why Cloud?
• Home brew setup vs pre-existing infrastructure
• Device maintenance is a pain
• We specialize in this area, so you don’t have to
– Streamlined procurement
– Scale won’t be an issue
• There’s also the boss asking for reports and data

6. Mobile Game Testing
• Has to be done no questions about it
• Particularly difficult vs. other functional mobile app testing
– No native UI controls
• Resource intensive applications
– Great variation on different devices
– Crashes and slowness
– Network connection dependent
– Real mobile devices needed, simulators and emulators don’t
represent the real thing

7. Mobile Game Testing in Bitbar Cloud
• We don’t really mind which technique is used, our environment supports any.
• Appium + image recognition
– Can be laborious for early stage games where graphics keep changing a lot
– Allows testing the released app
– Take screenshots and find matches during tests
– Can be slow as needs multiple tools (Appium + OpenCV)
– Black box testing – can be done by anyone
• Alternative sources of data - game logs guide testing
– Special build for testing
– Gray box testing – done by dev teams
• Self playing games
– Special build for testing
– Gray box testing – done by dev teams

8. Demo!
• Appium + image recognition
– https://github.com/bitbar/testdroid-samples/tree/master/image-
recognition

9. Gamebench Results
• Frame rate: 30fps for slower games is ok. Games with rapid reflexes or simulated physics or
collisions (like Angry Birds) target 60fps
– If your game drops below 24fps (against a 30fps median) or 50fps (against a 60fps median),
GameBench will give you a low score for frame rate stability. Consistently smooth games
usually get a stability score of 80 percent or higher.
• CPU Usage: 15 percent is a healthy limit for average normalised CPU usage. Unnormalised
CPU usage threshold could be at 30 percent for a dual-core device.
– Sustained excessive CPU usage increases battery drain and temperature -> the device
throttles performance.
• GPU Usage: many games push GPUs to their limits, this increases the likelihood of janks (i.e.
micro-stutter) and frame rate drops. Aim for a peak of 80 or 90 percent GPU usage. Using over 70
percent of GPU on current high-end devices very often max out the GPU on older or cheaper
devices.
• Memory usage: RAM hogging is bad. Leaking memory is unacceptable. Casual games should
aim for under 200MB on iOS and 500MB on Android. If you go above, blacklist older and low end
devices to avoid bad reviews.