This presentation slide is aimed to represent my knowledge gain over 1 year experience in AOSP build system and Dalvik VM porting. It serves some fundamental aspect of Android's internals. Dalvik VM porting attempts and methods are also included (Spoil: They are all fail).
PS: This slide is not complete, and is under updated.
2. Outline
Introduction to Android
Introduction to Dalvik VM
Introduction to Bionic LibC
Introduction to QEMU
Merits of Porting GLibC Dalvik VM
Porting Attempted Approaches
Available Alternative Ports
How to Setup AOSP Build System
3. Introduction Android
Open Source OS for smart devices
Developed by Google
Based on Linux Kernel
Smartphones, Tablets, Gadgets
With several custom modules
Ranked 1st among other Smart Device OSes
81% market share in Q3, 2013 – IDC Report
9. Introduction to Dalvik VM
K. Yaghmour, Embedded Android 1st edition, Chapter 2, Figure 2-1
10. Introduction to Dalvik VM
The core of Android Applications
All fancy Android applications are run by it
Register-based Process Virtual Machine
Think of running a Java application
Intermediate Language = Dalvik Bytecode
Executable File = Dalvik Executable (DEX)
A converted Java class done by “dx” tool
Reduce redundancy in variables
11. Introduction to Dalvik VM
Variable Length ISA
1 Word = 2 bytes = 16 bits
Fetched 1 word a time
OP code is 8-bit length = Max of 256 instructions
Use only 246 instructions
Attached in the 1st word
Default Endianess is Little
Default OP code handles up to 65536 Virtual Registers (VRs)
Ideally infinite
VRs are 32-bit width
64-bit variable = concatenate 2 consecutive VRs together
Has Pseudo Instruction concept
Must be optimized by “dexopt” before run
4 Types of constant pool: String, Type, Field, Method
2 Types of prelinked offset: VTab offset, Field offset
Source Android – Bytecode for the Dalvik VM (cited Nov 26, 2013)
Source Android – Dalvik VM Instruction Formats (cited Nov 26, 2013)
12. Introduction to Dalvik VM
2 Modes of Machine Code Interpretation (Mterp)
Fast – Based on manually written assembly code for each OP code
Portable – Based on compiled generic C++ files for each OP code
Enhancement
Java Native Interface
Just-In-Time compiler
Allow Dalvik VM to call native functions
Prepare bytecode segments into machine code ahead during run-time
Zygote
Mother process which
Create a Dalvik VM
And preloads frequently used shared libraries
New Dalvik VM starts by forking from this template Dalvik VM
Reduce loading time
Save memory space (Copy-On-Write)
Forays in software development – Stack based vs Register based Virtual Machine Architecture, and the Dalvik VM (cited Nov
26, 2013)
Linux For You – Virtual Machines For Abstraction: The Dalvik VM (cited Nov 26, 2013)
Wikipedia – Just-in-time compilation (cited Nov 26, 2013)
13. Stack-based Virtual Machine
Don’t name register in instuctions
Simple design
Instruction width is short
Redundancy in using same variables
Everything is queued up in its stack
No name to registers = Copy on a stack repetitively
More instructions used to complete a task
But, total program size is smaller than Registerbased
Y. Shi, D. Gregg, A. Beatty, Virtual Machine Showdown: Stack versus Registers, VEE’05
Forays in software development – Stack based vs Register based Virtual Machine Architecture, and the Dalvik VM (cited Nov
26, 2013)
Linux For You – Virtual Machines For Abstraction: The Dalvik VM (cited Nov 26, 2013)
14. Register-based Virtual Machine
Physical CPU analogy
Explicitly naming register
Instruction width is longer
Can do complex register manipulation
Fast
No need to be in order like in Stack-based
Fewer VM instructions
An effect from larger instruction width can be neglected
A bit memory consuming
Larger program size
Normally require a lot of space for virtual registers
Y. Shi, D. Gregg, A. Beatty, Virtual Machine Showdown: Stack versus
Registers, VEE’05
15. From Source Code to Execution
Java Source Code
javac
Java Class
dx
DEX file
dexopt
Optimized DEX
dalvikvm
Forays in software development – Stack based vs Register based Virtual Machine Architecture, and the Dalvik VM (cited Nov
26, 2013)
16. DEX Structure
HEADER
The Header (size of 0x70 bytes)
STRING_IDS
String Identifier List
TYPE_IDS
Type Identifier List
PROTO_IDS
Method Prototype Identifier List
FIELD_IDS
Field Identifier List
METHOD_IDS
Method Identifier List
CLASS_DEFS
Class Definition List
DATA
Data area for support contents above
LINK_DATA
Data used in statically linked files
Source Android – Dalvik Executable Format (cited Nov 26, 2013)
17. How to port Dalvik VM to new Architecture
Google mentions only for porting to other Android with different
architecture
In short = Build for Target
Document in AOSP/dalvik/docs/porting-guide.html
Steps
Create new Machine Interpreter (Mterp) in both C and its machine
code for each instruction code
Create new Mterp config for your architecture
Update Mterp by AOSP/dalvik/vm/mterp/rebuild.sh
This assembles your written instruction codes
InterpAsm-Your_Arch.S + InterpC-Your_Arch.cpp
Modify Target build configuration in AOSP/dalvik/vm/Dvm.mk
Set $(TARGET_ARCH) by lunch or export
TARGET_ARCH=Your_Arch
Build new libdvm.so by m libdvm
18. libART (Android RunTime Library)
Android 4.4’s experimental virtual machine library
Disabled by default
Turn on by checking this option in Developer options
Changing to libART from libDVM
Requires all apps to be recompiled = Restart + Super long first boot
Use Ahead-Of-Time (AOT) scheme instead of JIT
Precompile Dalvik Bytecode into machine language during installation
Take longer time to install for the first time
Applications run faster
Apps are in native form, so they are very ready to be executed
Saving more energy
Apps end faster = More PEs idle time
Occupy more storage space
Decompress to machine code
Android Headlines – Google is Looking to Replace Dalvik with ART (cited Nov
26, 2013)
19. Introduction to Bionic LibC
Derivation of BSD standard C
library
Developed by Google
Small 2.11
GLibC
1,208,224
bytes
uClibc 0.9.30
243,938 bytes
Upto 114% faster than GLibC
License
dietlibc
No SysV IPC support
Android has Binder IPC
musl
No wchar support
Has several LGPL kernel specific
Linux
BSD
GPL
MIT
features
GLibC,
uClibc
Not compatible with GLibC
JNI vs. Native C bound with GLibC
Bionic
ARM / x86 / MIPS
Partially POSIX compatible
Optimized for low clock processors
Currently support for
Fast
Completely rewritten by Google
Bionic 2.1
424,235 bytes
Based on Futexes
NetBSD (libc) + FreeBSD (libm)
Fast Pthread implementation
No C++ exception support
Not full C++ STL
Not fully POSIX Thread compliant
Wikipedia – Bionic (software) (cited Nov 25, 2013)
ICCAS2012 – Benchmarking Java Application using JNI and Native C Application on
Android
Coding Relic – The Six Million Dollar LibC (cited Nov 25, 2013)
20. Introduction to QEMU
Quick EMUlator
Free and Open Source Hardware Virtualizer
Can emulate
Whole system / board – Versatile Express-A9
Only CPU – x86, sparc, ARM
qemu-arm-softmmu -M vexpress-a9
qemu-i386
Has Kernel-based Virtual Machine (KVM) support
Accelerate emulation with same CPU architecture as its
Host
Wikipedia – QEMU (cited Nov 25, 2013)
21. Merits of Porting GLibC Dalvik VM
Getting full capability of GLibC
C++ Exception handling
Wide charactor handling
POSIX Thread compliant
Can be run on any ordinary Linux
GlibC is shipped with Linux by almost distros
Consistence with other existing Linux programs
Can talk to each other under same SysV IPC
-----------------------------------------------------------------What could come after this
Comparison between Bionic and GLibC Dalvik VM
Porting GLibC Dalvik VM for TCT processor
GLibC porting to TCT is under going
22. Porting Attempted Approaches
Copy-Paste Bionic Dalvik VM
1.
Failed
Cannot find libraries
Copy-Paste Bionic Dalvik VM + Bionic LibC
2.
Failed
Something low-level failed
Copy-Paste Bionic Dalvik VM + Bionic LibC + Ashmem
3.
Failed
Something low-level failed (same output as previous attempt)
GLibC - Dalvik VM porting
4.
Not done
Make modification to build/core Makefiles
Only HOST configurations
23. Testing Environment
ARM
Use QEMU to emulate VExpress-A9 with bare Linux kernel with Busybox
shell
Use QEMU to emulate Vexpress-A9 with Ubuntu 13.04
http://balau82.wordpress.com/2010/03/27/busybox-for-arm-on-qemu/
http://releases.linaro.org/13.04/ubuntu/vexpress/
x86
Use QEMU to emulate QEMU’s x86 default board with Debian 7 Squeeze
Run on Host machine directly
24. 1st Attempt
Copy-Paste Bionic Dalvik VM
Does not work
No Bionic LibC for it
Error message is
bash: [Directory to Dalvik VM]/dalvikvm: No such file or directory
This error message is ALSO OCCUR when run a GLibC linked
program on an Android Emulator
25. 2nd Attempt
Copy-Paste Bionic Dalvik VM
Copy Bionic LibC to /system/lib
/system/bin/linker is hardcoded to link with /system/lib
Create dalvik-cache directory for Optimized DEX
/data/dalvik-cache
Error message is
It is done by setting “rpath” flag
Dalvik VM init failed (checked log file)
There is no Log file left
No porting of liblog and Logger
26. 3rd Attempt
Setting is same as 2nd attempt
This attempt was tested only on
Emulated VExpress-A9 with bare Linux kernel with
Busybox shell
Linux kernel is shipped with Ashmem
Ashmem = Annonymous Shared Memory
Now shipped with Linux mainline residing in Staging directory
It is said that Ashmem is needed for Dalvik VM porting
Error message is
Sometimes, people call it Android Shared Memory
Dalvik VM init failed (checked log file)
Failure cause is still unknown
Dalvik Virtual Machine on TS-7800 SBC (cited Nov 26, 2013)
27. 4th Attempt
Host version of Dalvik VM is linked with GLibC
Fake an attribute of Host: HOST_ARCH
GLibC linked Dalvik VM for that architecture
Try with ARM architecture
Modify several Makefiles to make sure AOSP builds for
only Host side
Many Makefiles contains builds for Host and Target in one file
m dalvik-host conscrypt-hostdex
Minimum Dalvik VM build for Host
Create new Makefiles and Includes for new host
architectures
Still not complete
Configuration problems
28. Modifying Guide
Build acp (Android’s cp tool)
$ m acp
Modify build/core/config.mk Force use acp for x86 Host
From
$ ACP := $(BUILD_OUT_EXECUTABLES)/acp$(BUILD_EXECUTABLE_SUFFIX)
To
$ ACP := $(HOST_OUT_ROOT)/linux-x86/bin/acp$(BUILD_EXECUTABLE_SUFFIX)
Modify build/core/envsetup.mk to accept Host Architecture Overwrite
From
$ BUILD_ARCH := $(HOST_ARCH)
To
$ BUILD_ARCH := $(HOST_ARCH)
$ ifneq ($(HOST_ARCH_OWR),)
$ HOST_ARCH := $(HOST_ARCH_OWR)
$ endif
29. Guide
Build acp (Android’s cp tool)
$ m acp
Modify build/core/config.mk Force use acp for x86 Host
From
$ ACP := $(BUILD_OUT_EXECUTABLES)/acp$(BUILD_EXECUTABLE_SUFFIX)
To
$ ACP := $(HOST_OUT_ROOT)/linux-x86/bin/acp$(BUILD_EXECUTABLE_SUFFIX)
Modify build/core/envsetup.mk to accept Host Architecture Overwrite
From
$ BUILD_ARCH := $(HOST_ARCH)
To
$ BUILD_ARCH := $(HOST_ARCH)
$ ifneq ($(HOST_ARCH_OWR),)
$ HOST_ARCH := $(HOST_ARCH_OWR)
$ endif
30. Guide
Modify build/tools/acp/Android.mk Disable acp build
From
$ include $(BUILD_HOST_EXECUTABLE)
To
$ ifeq ($(HOST_ARCH),x86)
$ include $(BUILD_HOST_EXECUTABLE)
$ endif
Create new build/core/combo/HOST_linux-arm.mk
Manually merge between HOST_linux-x86.mk and TARGET_linux-arm.mk
Create new build/core/combo/include/arch/host_linuxarm/AndroidConfig.h
Manually merge between linux-x86, target_linux-x86, and target_linux-arm
31. Available Alternative Ports
idvk – Independent Dalvik
Myriad Alien Dalvik
Run Android apps on non-Android devices
IcedRobot
Not actually porting
Just extract what Dalvik VM needs to be built from AOSP
Use OpenJDK instead of Sun JDK which was under license
problem between Google and Oracle
libhybris.so
Overwrite Bionic symbol to solve incompatibility with GLibC
Written by Mer developer
Used by Sailfish OS and Ubuntu Touch
32. How to Setup AOSP
1.
2.
This guide assumes an environment of x86 Ubuntu-64bit 12.04++
Download and Install Oracle JDK6 (see later slide)
Download and Install Other Required Software
1.
Required Packages
$ sudo apt-get install git gnupg flex bison gperf build-essential zip curl libc6-dev libncurses5-dev:i386
x11proto-core-dev libx11-dev:i386 libreadline6-dev:i386 libgl1-mesa-glx:i386 libgl1-mesa-dev g++-multilib
mingw32 tofrodos python-markdown libxml2-utils xsltproc zlib1g-dev:i386
$ sudo ln -s /usr/lib/i386-linux-gnu-mesa-libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so
2.
Repo
$ mkdir ~/bin
$ PATH=~/bin:$PATH
$ curl http://commondatastorage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
$ chmod a+x ~/bin/repo
Source Android – Downloading and Building (cited Nov 26, 2013)
33. How to Setup AOSP (cont.)
Set up repo repository
3.
$ mkdir AOSP
$ cd AOSP
$ repo init -u https://android.googlesource.com/platform/manifest
Sync with server
3.
$ repo sync
Add GPG key
4.
$ gpg --import
1.
Copy the following key
-----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.4.2.2 (GNU/Linux) mQGiBEnnWD4RBACt9/h4v9xnnGDou13y3dvOx6/t43LPPIxeJ8eX9WB+8LLuROSV
lFhpHawsVAcFlmi7f7jdSRF+OvtZL9ShPKdLfwBJMNkU66/TZmPewS4m782ndtw7 8tR1cXb197Ob8kOfQB3A9yk2XZ4ei4ZC3i6wVdqHLRxABdncwu5hOF9KXwCgkxMD
u4PVgChaAJzTYJ1EG+UYBIUEAJmfearb0qRAN7dEoff0FeXsEaUA6U90sEoVks0Z wNj96SA8BL+a1OoEUUfpMhiHyLuQSftxisJxTh+2QclzDviDyaTrkANjdYY7p2cq
/HMdOY7LJlHaqtXmZxXjjtw5Uc2QG8UY8aziU3IE9nTjSwCXeJnuyvoizl9/I1S5 jU5SA/9WwIps4SC84ielIXiGWEqq6i6/sk4I9q1YemZF2XVVKnmI1F4iCMtNKsR4
MGSa1gA8s4iQbsKNWPgp7M3a51JCVCu6l/8zTpA+uUGapw4tWCp4o0dpIvDPBEa9 b/aF/ygcR8mh5hgUfpF9IpXdknOsbKCvM9lSSfRciETykZc4wrRCVGhlIEFuZHJv
aWQgT3BlbiBTb3VyY2UgUHJvamVjdCA8aW5pdGlhbC1jb250cmlidXRpb25AYW5k cm9pZC5jb20+iGAEExECACAFAknnWD4CGwMGCwkIBwMCBBUCCAMEFgIDAQIeAQIX
gAAKCRDorT+BmrEOeNr+AJ42Xy6tEW7r3KzrJxnRX8mij9z8tgCdFfQYiHpYngkI 2t09Ed+9Bm4gmEO5Ag0ESedYRBAIAKVW1JcMBWvV/0Bo9WiByJ9WJ5swMN36/vAl
QN4mWRhfzDOk/Rosdb0csAO/l8Kz0gKQPOfObtyYjvI8JMC3rmi+LIvSUT9806Up hisyEmmHv6U8gUb/xHLIanXGxwhYzjgeuAXVCsv+EvoPIHbY4L/KvP5x+oCJIDbk
C2b1TvVk9PryzmE4BPIQL/NtgR1oLWm/uWR9zRUFtBnE411aMAN3qnAHBBMZzKMX LWBGWE0znfRrnczI5p49i2YZJAjyX1P2WzmScK49CV82dzLo71MnrF6fj+Udtb5+
OgTg7Cow+8PRaTkJEW5Y2JIZpnRUq0CYxAmHYX79EMKHDSThf/8AAwUIAJPWsB/M pK+KMs/s3r6nJrnYLTfdZhtmQXimpoDMJg1zxmL8UfNUKiQZ6esoAWtDgpqt7Y7s
KZ8laHRARonte394hidZzM5nb6hQvpPjt2OlPRsyqVxw4c/KsjADtAuKW9/d8phb N8bTyOJo856qg4oOEzKG9eeF7oaZTYBy33BTL0408sEBxiMior6b8LrZrAhkqDjA
vUXRwm/fFKgpsOysxC6xi553CxBUCH2omNV6Ka1LNMwzSp9ILz8jEGqmUtkBszwo G1S8fXgE0Lq3cdDM/GJ4QXP/p6LiwNF99faDMTV3+2SAOGvytOX6KjKVzKOSsfJQ
hN0DlsIw8hqJc0WISQQYEQIACQUCSedYRAIbDAAKCRDorT+BmrEOeCUOAJ9qmR0l EXzeoxcdoafxqf6gZlJZlACgkWF7wi2YLW3Oa+jv2QSTlrx4KLM= =Wi5D -----END PGP PUBLIC
KEY BLOCK-----
2.
Then press Ctrl+D
Source Android – Downloading and Building (cited Nov 26, 2013)
34. How to Setup AOSP (cont. 2)
5.
Setup AOSP environment
$ . build/envsetup.sh
6.
Select Target build
$ lunch
** Example of building full options for generic ARM target
$ . build/envsetup.sh
$ lunch full-eng
$ make -j8
-jN flag specifies N jobs to run simultaneously = Many cores processors benefit
from it
Source Android – Downloading and Building (cited Nov 26, 2013)
35. Installing Oracle JDK6
Download JDK6 from Oracle
Execute the downloaded bin file = File extract
Make a directory for it
Copy the extracted files to that directory
sudo mkdir /usr/lib/jvm
sudo mv jdk-1.6.0_xx /usr/lib/jvm/jdk-1.6.0_xx
Add Java alternatives
sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk-1.6.0_xx/bin/java 1
sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk-1.6.0_xx/bin/javac 1
sudo update-alternatives --install /usr/bin/javaws javaws /usr/lib/jvm/jdk-1.6.0_xx/bin/javaws 1
Update Java alternatives
sudo update-alternatives --config java
sudo update-alternatives --config javac
sudo update-alternatives --config javaws
Enable Java plugin for Mozilla Firefox and Google Chrome
64-bit: sudo ln -s /usr/lib/jvm-1.6.0_xx/jre/lib/amd64/libnpjp2.so /usr/lib/mozilla/plugins
32-bit: sudo ln -s /usr/lib/jvm-1.6.0_xx/jre/lib/i386/libnpjp2.so /usr/lib/mozilla/plugins
dev sniper – Ubuntu 12.04 – install sun jdk 6-7 (cited Nov 26, 2013)