★ Session Summary ★ Android provides CTS - Compatibility Test Suite, written in java and run to verify the Android system. However Android has limited low-level tests, typically needed to port, benchmark and optimize components, in isolation. There is a need to create, enhance, share and deploy such user-space tests. In this discussion, Ashok will cover recent work on multimedia codec tests for AArch64 done by ARM and Linaro, provide a quick overview on current Android user space tests, being run as part of Linaro's monthly release. This will be followed by an open discussion --------------------------------------------------- ★ Resources ★ Zerista: http://lcu14.zerista.com/event/member/137789 Google Event: https://plus.google.com/u/0/events/c32of9jvrqg07t5blth0c3kbrgs Video: https://www.youtube.com/watch?v=xNr6xsvnNVA&list=UUIVqQKxCyQLJS6xvSmfndLA Etherpad: http://pad.linaro.org/p/lcu14-502 --------------------------------------------------- ★ Event Details ★ Linaro Connect USA - #LCU14 September 15-19th, 2014 Hyatt Regency San Francisco Airport --------------------------------------------------- http://www.linaro.org http://connect.linaro.org

1. 1
Android™ User-Space Tests Multimedia codec tests, status and open discussion
Ashok Bhat
System & Software, ARM
LCU14 – 502
September 2014

2. 2
Use-case - Android Media Codec - AArch64 Port - Validation
Android user-space tests and benchmarks - Discussion
Do we need more?
How to create?
How to share?
How to deploy and maintain?
Agenda

3. 3
Android Core Media Support
Type
Format / Codec
Encoder
Decoder
Audio/
Speech
AAC LC
•
•
HE-AACv1 (AAC+)
•
•
HE-AACv2 (enhanced AAC+)
•
AAC ELD (enhanced low delay AAC)
•
•
AMR-NB
•
•
AMR-WB
•
•
FLAC
•
•
MP3
•
Vorbis
•
PCM/WAVE
•
Video
H.263
•
•
H.264 AVC
•
•
MPEG-4 SP
•
VP8
•
•

4. 4
Application and Native Media Framework Interaction
Media Codecs
libstagefright.so
Application Framework
android.media.*
Binder IPC Proxies
(frameworks/av/libmedia)
Media Player Binder
MediaPlayerService.cpp
Media Player Service
(frameworks/av/media/libmediaplayerservice)
OMX Core
MediaSource.cpp
Stagefright Engine (frameworks/av/libstragefright)
Native Multimedia Framework
(frameworks/av)
OpenMax Integration Layer
OMX Plug-in
Codec Implementation
OpenMAX IL Component
SOURCE : https://source.android.com/devices/media.html

5. 5
Android Media Codecs

6. 6
Codecs external to Android
Active upstream projects
Projects reside in external directory
Audio – AAC, FLAC, Vorbis
Video – VP8, VP9
Typically have validation suite outside Android
Codecs present only in Android
No upstream project
Projects reside in frameworks/av/media/libstagefright/codecs
Speech – AMR-NB, AMR-WB
Audio – MP3
Video – H.263, H.264, MPEG4
Rely on CTS for validation
Android Media Codecs

7. 7
Decoder and Encoder need to conform to a standard
For example, MP3 decoder conforms to ISO/IEC 13818-3 and ISO/IEC 11172-3
For a decoder, standard provides a set of conformance clips and a reference decoder
For an encoder, standard may provide a set of input clips and a test criteria
Decoder test mechanism
Decode conformance test clips
Compare output with the output generated by the reference decoder
Output should be either bit-exact or within a specified range when compared to reference output
Encoder test mechanism
Encode input for different combination of encoding parameters
Check if encoded output conforms to the standard
Typical codec validation

8. 8
Compatibility Test Suite
Part of Android compatibility program
Runs on a desktop machine and executes test cases directly on attached devices or an emulator
Does not claim to be comprehensive
Media codec validation
Part of android.media.cts package
Present in cts/tests/tests/media/src/android/media/cts directory
Tests are Java programs that run on top of the Android software stack
Android CTS

9. 9
Need a method to validate AArch64 port
CTS is a good starting point. But...
Not meant to be comprehensive
Runs on top of Android stack – tricky to use for benchmarking component
Need something in addition to CTS that would
Allow validation/benchmarking/optimization at the component level.
Allow validation of conformance test vectors – preferably file I/O based.
Allow tests to be part of larger testing framework like LAVA.
Media Codec - AArch64 Port - Validation

10. 10
New component level tests developed
By ARM and Linaro
For Video, Audio and Speech components
For encoder and decoder
Mostly file based tests – tests use file input and generate file output
Tests can run independent of Android – Does not require Android Software Stack
Allows porting/optimization/testing of components outside Android
Available in AOSP Gerrit and Linaro Android user-space git repo
Patches pushed to AOSP Gerrit – Not completely merged yet
Part of Linaro Android user-space git repo
Run as part of LAVA for each Linaro Android release
Media Codec - AArch64 Port – Validation (Contd...)

11. 11
Media Codec Test Patches in AOSP Gerrit

12. 12
Tests in Linaro Android user-space git
Repo for Android user-space tests and programs
Link
https://android.git.linaro.org/gitweb/platform/external/linaro-android-userspace-test.git
Contents

13. 13
Android supports extensive list of media codecs
A significant set of codecs exist only in Android and do not have an upstream project
Codec validation requires exhaustive conformance tests as specified by standard
CTS is a good starting point but not the right place for comprehensive component level validation
Codec component level tests were developed by ARM and Linaro
Test patches are available in AOSP Gerrit
Test code, media files and test script are also available in Linaro Android user-space git repo
Tests are run as part of each Linaro Android release
Summary

14. 14
Do we need more such component/subsystem level tests?
How to create them?
How do we share them?
Will Android Linaro git repo meet your requirement? Do you need anything different?
How to maintain them?
Is running part of Linaro Android release sufficient?
Anything else that can be done differently?
Discussion

15. 15
Thank you!