SG-GTUG

1. Multimedia Application
Development on Android
KRK MOHAN
Senior Engineer - Muvee Technologies
www.muvee.com
krk.mohan@gmail.com
1

2. Agenda
• Multimedia and Android
• Video & Audio: MediaPlayer, MediaRecorder,
JetPlayer, SoundPool, AudioTrack
• Images: OpenGL-ES graphics, Bitmaps,
ImageView
2

3. Scope
• Introductory level
• Focus on what can and cannot be done with
Android FroYo’s multimedia capabilities
• References to APIDemos projects and
developer site articles provided
3

4. Multimedia Apps
• Media Consumption - Slide Shows, Media
Players,YouTube, facebook etc
• Media Production - Image Editors, Camera
enhancements
• Huge scope for improvement - great
opportunity for developers
4

5. You will identify a great product when you see it
Deep, Indulgent, Complete, Elegant (DICE)
Guy Kawasaki
“The Macintosh Way”
5

6. android.media
• Contains classes to play audio/video content
from raw resource files, file streams and JET
content
• Contains classes to record audio/video
content from cameras or mic
http://developer.android.com/intl/de/guide/topics/media/
index.html
6

7. File formats and codecs
• File formats for recording media on device
• 3GPP, MP4, RAW_AMR
• Codecs used for recording media on device
• H263, H264, MPEG4 (SP), AMR_NB
7

8. The big 2
MediaPlayer and MediaRecorder classes encapsulate:
File parsers for media file formats such as 3GP, MP4
&
Codecs to encode or decode media content
Two media frameworks present in FroYo
OpenCORE and StageFright
8

9. MediaPlayer - state diagram
From Android Developers -
MediaPlayer entry
9

10. Audio
• MediaPlayer & MediaRecorder
• SoundPool - multiple audio sources
• AudioTrack & AudioRecord- raw audio
manipulation
• JetPlayer - interactive music
10

11. Audio - MediaPlayer
Playing an audio resource
MediaPlayer mp = MediaPlayer.create(context, R.raw.sound_file_1);
mp.start();
Playing a file from file system
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE);
mp.prepare();
mp.start();
11

12. Audio - SoundPool
• Multiple sounds can be played from raw or
compressed source. Sounds are identified
by sound ids.
• More efficient than multiple MediaPlayers
• Each sound can be assigned a priority
• Adjustable playback frequency
• Support for repeat mode
12

13. Audio - AudioTrack
• Used to play raw audio samples (PCM)
stored in memory buffers or files
• Static mode for playing sounds that fit into
memory and need to be played often with
low overhead
• Streaming mode for sounds that are too big
to fit into memory
13

14. Audio - AudioTrack
AudioTrack audioTrack = new
AudioTrack(AudioManager.STREAM_MUSIC,
44100,
AudioFormat.CHANNEL_CONFIGURATION_MONO,
AudioFormat.ENCODING_PCM_16BIT,
musicLength,
AudioTrack.MODE_STREAM);
audioTrack.play();
audioTrack.write(music, 0, musicLength); //blocking
Check out AudioRecord class
Some examples at:
http://emeadev.blogspot.com/2009/09/raw-audio-manipulation-in-android.html
14

15. Audio - JETPlayer
• Interactive music playback by altering
tempo, volume level etc of a series of MIDI
clips
• Used extensively in game development
• JET Creator is used to author JET files
http://developer.android.com/intl/de/guide/topics/media/jet/
jetcreator_manual.html
15

16. Video - MediaPlayer
mMediaPlayer = new MediaPlayer();
mMediaPlayer.setDataSource(localPath);
mMediaPlayer.setDisplay(holder);
mMediaPlayer.setOnCompletionListener(this);
mMediaPlayer.setOnPreparedListener(this);
mMediaPlayer.prepareAsync();
public void onPrepared(MediaPlayer m) {
mMediaPlayer.start();
}
public void onCompletion(MediaPlayer m) {
mMediaPlayer.release();
}

17. Video - VideoView
mVideoView = (VideoView)
findViewById(R.id.surface_view);
mVideoView.setVideoPath(localPath);
mVideoView.setMediaController
(new MediaController(this));
mVideoView.requestFocus();
17

18. iMovie on Android!?
• Classes needed to process individual video
frames
• API level 1 contained MediaPlayer.getFrame()
• MediaPlayer.snoop() is used in music
visualization live wall paper, but is marked
@hide
18

19. way forward...
• OMXCodec.h or a similar interface needs to
be exposed via NDK
• This would facilitate comprehensive video
processing and editing
19

20. Graphics - OpenGL-ES
• 3D graphics API to process and render
images/textures
• On Android
GL10 loosely corresponds to OpenGL-ES 1.0/1.1
GL20 loosely corresponds to OpenGL-ES 2.0
• Performance depends on GPU
• No major issues with code portability across
GPUs
20

21. GLSurfaceView
From Android Developers:
• Manages a surface, which is a special piece of memory that can
be composited into the Android view system.
• Manages an EGL display, which enables OpenGL to render into
a surface.
• Accepts a user-provided Renderer object that does the actual
rendering.
• Renders on a dedicated thread to decouple rendering
performance from the UI thread.
• Supports both on-demand and continuous rendering.
• Optionally wraps, traces, and/or error-checks the renderer's
OpenGL calls.
21

22. OpenGL-ES Textures
• Create a Bitmap out of the image to be
processed
• GLUtils class provides helper functions such
as:
• texImage2D(int target, int level, int internalformat,
Bitmap bitmap, int border)
• target = GL10.GL_TEXTURE_2D, level = 0,
internalformat = GL10.GL_RGB,
GL_UNSIGNED_SHORT_5_6_5
• Compressed textures
22

23. GLSurfaceView.Renderer
class MyRenderer implements GLSurfaceView.Renderer {
public void onDrawFrame(GL10 gl) {
// called repeatedly to animate the loaded texture
}
public void onSurfaceChanged(GL10 gl, int width, int height) {
}
public void onSurfaceCreated(GL10 gl, EGLConfig config) {
}
}
23

24. Together...
• To begin rendering
mGLSurfaceView = new GLSurfaceView(this);
mGLSurfaceView.setRenderer(new MyRenderer());
setContentView(mGLSurfaceView);
• To pause rendering
mGLSurfaceView.onPause();
•To resume rendering
mGLSurfaceView.onResume();
For full example, checkout CubeMapActivity example in
ApiDemos
24

25. Misc OpenGL-ES
• setDebugFlags(DEBUG_CHECK_GL_ERROR |
DEBUG_LOG_GL_CALLS);
• Use
GLSurfaceView.setRenderMode(RENDERMODE_WHEN_
DIRTY) & GLSurfaceView.requestRender()for passive
applications
• Use EGLChooser to select an appropriate EGL
configuration
• Minimize texture sizes and bit depth for greater
performance
• Use NDK wherever feasible to speed up the application
25

26. Bitmaps & image
processing
• BitmapFactory.decode(filepath) returns a
Bitmap of the image
• Use getPixels to access the image’s pixels
• getPixels(int[] pixels, int offset, int stride, int x, int y, int
width, int height)
• Manipulate the pixels and use
Bitmap.compress() to encode the processed
pixels into a JPEG image
26

27. ImageView
• Display PNG, JPG images
• Apply effects like color tinting
• View can be animated
ImageView image = (ImageView) findViewById(R.id.image);
image.setBackgroundResource(R.drawable.my_image);
http://developer.android.com/intl/de/guide/topics/graphics/2d-graphics.html

28. Appendix - Animations
<animation-list xmlns:android="http://schemas.android.com/apk/res/android"
android:oneshot="true">
<item android:drawable="@drawable/rocket_thrust1" android:duration="200" />
<item android:drawable="@drawable/rocket_thrust2" android:duration="200" />
<item android:drawable="@drawable/rocket_thrust3" android:duration="200" />
</animation-list>
ImageView rocketImage = (ImageView) findViewById(R.id.rocket_image);
rocketImage.setBackgroundResource(R.drawable.rocket_thrust);
rocketAnimation = (AnimationDrawable) rocketImage.getBackground();
rocketAnimation.start();
28

29. Thank You!
29