Mobile devices are becoming more and more powerful. They come with all sorts of wonderful hardware like cpu/gpus, tons of ram and blazing fast download times. Smart phones have become commoditized in a sense. What's the next evolution of mobile? Now these devices are coming with a really solid set of sensors and apis that allow developers to determine a user's context. How does that work? Developers fuse the sensor output to infer context and infer events from the data. This talk will discuss ways to do it, challenges and drawbacks.

1. Making Sense
The Road to mobile Awareness
• Jared Sheehan
• Twitter: @jayroo5245
• meetup.com/DCAndroid
• slideshare.net/Jayroo5245

2. • What is Contextual
Awareness?
• Use Cases
• Sensor Fusion
• Hard way
• Medium hard way
• Easy way
• Questions
Agenda

3. “Context is any information that can be used to
characterize the situation of an entity. An entity is
a person, place, or object that is considered
relevant to the interaction between a user and an
application, including the user and application
themselves. ”
• Anind Dey
• Director of Human-Computer
interaction at Carnegie Mellon
University

4. • Mobile sensing of a user’s context
• Sensor based algorithms
• Some Sensor Types on the Android
Platform
• Accelerometer
• Gyroscope/Orientation/Rotation
Vector
• Barometric Pressure
• Magnetic Field
• Gravity
• Relative Humidity
• Ambient Room Temperature
• Device Temperature
Mobile Contextual Awareness

5. Sensor Types in Android

6. • Detecting when a user:
• Changes the orientation of a
their device
• walking, running or biking
• Driving a vehicle
• Handling their device
• Driving AND Handling their
device
• Disclaimer – Don’t do it
• Drives by a restaurant or coffee
shop when it is open
• Driving detection
• Google Places
• Time
Use Cases

7. Sensor fusion is combining of
sensory data or data derived
from disparate sources such
that the resulting information
has less uncertainty than
would be possible when
these sources were used
individually.
Sensor Fusion – What is it?

8. If a developer takes
individual sensor output
and combines it with
additional output from
other sensors (or other
hints) then you get a
better understanding of
what is going on with the
mobile device.
Sensor Fusion – What is it?

10. Determine the attitude of a
mobile device.
Attitude - orientation of a
device relative to Earth's
horizon
Sensor Fusion – Example

12. The common way to get the
attitude of an Android
device is to use the
SensorManager.getOrientati
on() method to get the three
orientation angles. These
two angles are based on the
accelerometer and
magnetometer output.
Sensor Fusion – Example

13. In simple terms, the
accelerometer provides the
gravity vector (the vector
pointing towards the center
of the earth) and the
magnetometer works as a
compass. The Information
from both sensors suffice to
calculate the device’s
orientation.
Sensor Fusion – Example

14. Isn’t that enough?
Sensor Fusion – Example
No
Problem is that both sensor
outputs are inaccurate,
especially the output from the
magnetic field sensor which
includes a lot of noise.
How do we fix it?

15. Gyro drift and noisy orientation
are common issues with this
approach, to solve it, the
gyroscope output is applied
only for orientation changes in
short time intervals. While the
magnetometer/accelerometer
data is used as support
information over long periods
of time.
Sensor Fusion – Example

16. This is equivalent to low-pass
filtering of the accelerometer
and magnetic field sensor
signals and high-pass filtering
of the gyroscope signals. The
overall sensor fusion and
filtering looks like this:
Sensor Fusion – Example

17. Sensor Fusion – Example

18. So what exactly does high-pass
and low-pass filtering of the
sensor data mean? The sensors
provide their data at (more or
less) regular time intervals.
Their values can be shown as
signals in a graph with the time
as the x-axis, similar to an
audio signal.
Sensor Fusion – Example

19. The low-pass filtering of the
noisy
accelerometer/magnetometer
signal (accMagOrientationin
the above figure) are
orientation angles averaged
over time within a constant
time window.
Sensor Fusion – Example

20. Initialize sensor
containers:
Sensor Fusion – Example

21. Register you listeners:
Sensor Fusion – Example

22. Store sensor events:
Sensor Fusion – Example

23. At some time interval you
process the sensor arrays and
then events can be inferred
from a single or multiple
passes.
Sensor Fusion – Example

24. Example of Rotation Vector
processing:
https://developer.android.com
/reference/android/hardware/
SensorEvent.html#values
Sensor Fusion – Example

25. • https://github.com/Jayroo5245/mak
ingsense
• https://github.com/Jayroo5245
Demo time!

26. • This is a simple-ish formula to
obtain one feature
• Very large task
• Lots of math, calculations,
sensor state maintenance
• Not something a standard
Android developer is used to
working with
Sensor Fusion – Challenges

27. • How do you support 100% of
devices?
• Very difficult
• Android Fragmentation
• Not all sensors return values
at the same frequency
Sensor Fusion – Challenges

28. • Process prioritization issues
• OEMs build devices to
their specs, not ours
• Missing sensors on some
devices.
• Android/Java platform
limitations
• Go Native - NDK
Sensor Fusion – Challenges

29. Example Platform limitation:
The Android Platform was not
designed to process sensor
data as fast as it is generated.
Using an Executor had the best
results but you will not get
consistent 16, 32 or 64 hertz.
Sensor Fusion – Challenges

31. • External Libraries
• Lost - Drop in Replacement for
Google’s Fused Location API
• www.zendrive.com
• www.driversiti.com
• www.pathsense.com
• www.locationkit.io
Alternatives to the hard way:

32. • External Libraries
• Licensing – IE Cost
• Probably don’t do exactly
what you want
• Inference
Change/Deprecation
• lack support
• Battery Drain
Drawbacks to External Libs:

33. Battery Issues:

35. Let Google Do It for you – Awareness API

36. • Current Local Time
Context #1– Time

37. • Latitude
• Longitude
Context #2 – Location

38. • Place, including place Type
Context #3 – Place

39. • Activity Recognition
• Detected user activity
(walking, running, biking)
Context #4 – Activity

40. • Nearby beacons (including
namespace, type, and
content)
Context #5 – Beacons

41. • Are the Headphones plugged?
Context #6 – Headphones

42. • Current Weather Conditions
Context #7 – Weather

43. • Apps can combine these
context signals to make
inferences about the
user's current situation,
and use this information
to provide customized
experiences.
• Exp: Suggest a playlist
while jogging in the
rain.
What is it?

44. • Easy implementation
• One API
• Signals are processed
for the app
• No need to build
complicated
algorithms
• Optimized Battery
Awareness Benefits

45. • Fence API
• System Notifications
• Snapshot API
• Real time request
Great now what?

46. • Push Mechanism - React to
specific situations
• Provides notifications
when a specific
combination of actions
occur
• Exp: Tell me when a user is
biking, its lunchtime and
near a bike friendly
restaurant
Fence API

47. • Pull mechanism
• Provides notifications
when a specific
combination of actions
occur
• Exp: Tell me when a user is
biking, its lunchtime and
near a bike friendly
restaurant
Snapshot API

48. • Hard way
• Build your own
• Easier Way
• External Lib
• Easiest Way
• Awareness API
Three options

49. Thank you for coming!
The Road to mobile Awareness
• Jared Sheehan
• Twitter: @jayroo5245
• meetup.com/DCAndroid
• slideshare.net/Jayroo5245

50. Sources
• https://en.wikipedia.org/wiki/Sensor_fusion
• https://developer.android.com/guide/topics/sensors/sensors_overview.html
• http://plaw.info/2012/03/android-sensor-fusion-tutorial/comment-page-1/
• http://www.androidpolice.com/2016/05/19/the-new-awareness-api-will-let-
apps-better-understand-your-environment/
• https://developers.google.com/awareness/overview
• https://www.interaction-design.org/literature/book/the-encyclopedia-of-
human-computer-interaction-2nd-ed/context-aware-computing-context-
awareness-context-aware-user-interfaces-and-implicit-interaction