Speaker: Dominik Helleberg, inovex MobileTech Conference 2015 September 2015 Mehr Vorträge: https://www.inovex.de/de/content-pool/vortraege/

1. Mobile Analytics
mit Elasticsearch und Kibana
Dominik Helleberg

2. Speaker
Dominik Helleberg
Mobile Development
Android / Embedded
Tools
http://dominik-helleberg.de/+

3. Mobile Analytics

4. Warum?
Server
OS
Hardware
Runtime
Software
Client
(Browser)
Backend
Server
OS
Hardware
Runtime
Software
Client
(Browser)
Client
(Browser)

5. Warum?
Server
OS
Hardware
Runtime
Software
Server
OS
Hardware
Runtime
Software

6. Warum?
Client
OS
Hardware
Runtime
Software
Backend
Client
OS
Hardware
Runtime
Software
Client
OS
Hardware
Runtime
Software

7. Was?
Crash Reporting
Logging
Analytics

8. Crash Reporting
E/AndroidRuntime( 1188): Caused by: java.lang.RuntimeException: CRASH
E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.c.b(Unknown Source)
E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.SelectRiver.onStart(Unknown Source)
E/AndroidRuntime( 1188): at android.app.Instrumentation.callActivityOnStart(Instrumentati
E/AndroidRuntime( 1188): at android.app.Activity.performStart(Activity.java:5018)
E/AndroidRuntime( 1188): at android.app.ActivityThread.performLaunchActivity(ActivityThre
E/AndroidRuntime( 1188): ... 11 more

9. Crash Reporting

10. Crash Reporting
• Pro-aktive Benachrichtigung bei Crashes
• Nachvollziehbare Stacktraces
• Device / App und Umgebungs-Infos
• Optional: Nachricht vom Benutzer
• Statistische Analysen

11. Crash Reporting
Nachvollziehbare Stacktraces
E/AndroidRuntime( 1188): Caused by: java.lang.RuntimeException: CRASH
E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.c.b(Unknown Source)
E/AndroidRuntime( 1188): at org.cirrus.mobi.pegel.SelectRiver.onStart(Unknown Source)
E/AndroidRuntime( 1188): at android.app.Instrumentation.callActivityOnStart(Instrumentati
E/AndroidRuntime( 1188): at android.app.Activity.performStart(Activity.java:5018)
E/AndroidRuntime( 1188): at android.app.ActivityThread.performLaunchActivity(ActivityThre
E/AndroidRuntime( 1188): ... 11 more
Exception Type: SIGSEGV
Exception Codes: SEGV_ACCERR at 0x13babeb8
Crashed Thread: 8
Thread 8 Crashed:
0 libobjc.A.dylib 0x0000000196f5bbd0 0x196f40000 113616
1 CoreFoundation 0x0000000185369458 0x185364000 21592
2 QuartzCore 0x000000018983cad8 0x189830000 51928
3 QuartzCore 0x0000000189842aec 0x189830000 76524
4 libobjc.A.dylib 0x0000000196f61724 0x196f40000 136996
5 CoreFoundation 0x000000018536d074 0x185364000 36980
6 Foundation 0x00000001862acffc 0x186298000 86012
7 CFNetwork 0x0000000184ec9aac 0x184dac000 1170092….

12. Crash Reporting
Crash
reporter
DISTRIBUTION
PLATTFORM
dSYM
mapping
Mobile
Crash
reporter
App
Crash
Analytics
Logging
reporter
Analytics
reporter

13. Crash Reporting
• App-Stores
• 3rd Party
• Custom inhouse

14. Crash Reporting
App Stores

15. Crash Reporting
App Stores

16. Crash Reporting
App Stores

17. Crash Reporting
App Stores
• No unified interface
• Internal betas
• Internal apps
• Limited to App-Store functionality

18. Crash Reporting
3rd Party Solutions

19. Crash Reporting
3rd Party Solutions
• Internal apps (if there’s internet access)
• Pricing
• Privacy

20. Crash Reporting
inhouse
Crash
reporter
DISTRIBUTION
PLATTFORM
dSYM
mapping
Mobile
Crash
reporter
App
Crash
Analytics

21. Crash Reporting
inhouse
Crash
reporter
dSYM
mapping
Crash Analytics
ReceiverStorageAnalytics
+
Alarming

22. Crash Reporting
inhouse

23. Crash Reporting
inhouse

24. Crash Reporting
inhouse
• Limited Analytics (Trends, queries, search
etc.)
• User Interface
• Fixed data-structures
• Custom stack

25. Crash Reporting – with ELK
inhouse 2.0
ELK?
https://pixabay.com/p-­145930/

26. Crash Reporting – with ELK
inhouse 2.0
Elasticsearch L
ogstash
K
ibana

27. Crash Reporting – with ELK
inhouse 2.0
L
ogstash

28. Crash Reporting – with ELK
inhouse 2.0
L
ogstash
“Logstash is a data pipeline that
helps you process logs and other
event data from a variety of systems.
With 165 plugins and counting,
Logstash can connect to a variety of
sources and stream data at scale to a
central analytics system.”
https://www.elastic.co/products/logstash

29. Crash Reporting – with ELK
inhouse 2.0
L
ogstash
Logstash
Output
Plugins
Filter
Plugins
Input
Plugins

30. Crash Reporting – with ELK
inhouse 2.0
L
ogstash
Logstash
Output
Plugins
Filter
Plugins
Input
Plugins
(~40)
elasticsearch
eventlog
file
http
jdbc
kafka
jmx
log4j
tcp
twitter
syslog
rabbitMQ
redis
udp
zeromq

31. Crash Reporting – with ELK
inhouse 2.0
L
ogstash
Logstash
Output
Plugins
Filter
Plugins
(~40)
Input
Plugins
aggregate
anonymize
collate
csv
checksum
drop
dns
geoip
grok
extractnumbers
mutate
metrics
ruby
uuid
throttle
urldecode
translate

32. Crash Reporting – with ELK
inhouse 2.0
L
ogstash
Logstash
Output
Plugins
(~50)
Filter
Plugins
Input
Plugins
elasticsearch
csv
cloudwatch
datadog
email
file
g_c_s
hipchat
http
irc
jira
kafka
mongodb
nagios
pipe
s3
redmine
redis
sqs
tcp
rabbitmq
syslog
stdout
zabbix

33. Crash Reporting – with ELK
inhouse 2.0
L
ogstash
Logstash
Output
elastic
search
Filter
de-­‐obfuscate
(custom)
Input
HTTP
(json)
dSYM
mapping

34. Crash Reporting – with ELK
inhouse 2.0
E
lasticsearch
• Document / Data Store (JSON)
• RESTful API
• Uses Lucene for indexing
• Optimized for search queries
• Easy to distribute and scale
• Auto-­Schema
• Easy to start with

35. Crash Reporting – with ELK
inhouse 2.0
E
lasticsearch
Terms:
• Index
• Type
• Document
• Field
• Mapping

36. Crash Reporting – with ELK
inhouse 2.0
E
lasticsearch
Index:
logstash-­‐%{platform}-­‐%{PACKAGE_NAME}-­‐%{+YYYY.MM.dd}
Type
“crash-­‐report”
Document
{ "APP_VERSION_NAME": "1.3.2",
"AVAILABLE_MEM_SIZE": 10004287488,
"USER_APP_START_DATE": "2015-­‐08-­‐27T13:37:59.000+02:00",
"CUSTOM_DATA": {},
"BRAND": "samsung“....
}

37. Crash Reporting – with ELK
inhouse 2.0
K
ibana
Kibana is an open source analytics and visualization
platform designed to work with Elasticsearch.
You use Kibana to search, view, and interact with data
stored in Elasticsearch indices.
You can easily perform advanced data analysis and
visualize your data in a variety of charts, tables, and
maps.
Kibana makes it easy to understand large volumes of
data. Its simple, browser-­based interface enables you to
quickly create and share dynamic dashboards that
display changes to Elasticsearch queries in real time.
https://www.elastic.co/guide/en/kibana/current/introduction.html

38. Crash Reporting – with ELK
inhouse 2.0
K
ibana

39. Crash Reporting – with ELK
inhouse 2.0
Crash
Reporter
ACRA
(Android)
QuincyKit
(iOS)
dSYM
mapping
Crash Analytics
Logstashelastic
search
Kibana

40. Crash Reporting – with ELK
inhouse 2.0

41. Crash Reporting – with ELK
inhouse 2.0
Crash
Reporter
ACRA
(Android)
QuincyKit
(iOS)
dSYM
mapping
Crash Analytics
Logstashelastic
search
Kibana
Watcher
Kibana
Plugins
(soon!)

42. Logging with ELK
Logging
Library
Remote Logging
Logstashelastic
search
Kibana

43. Analytics
A bit more complicated then crash reporting and
logging…

44. Analytics

45. Analytics

46. Analytics

47. Analytics

48. Analytics

49. Analytics

50. Analytics with ELK
Analytics Library
(userID,
Session
Tracking,
PageViews)
Analytics
Logstashelastic
search
Kibana
Device
Infos
Carrier
GeoIP
UserProfiles
Kibana-­‐
Plugins
(not yet!)

51. ELK -> Getting started
Hosting:
elastic.co
google-­compute
qbox
Define index strategy
Check your Schema
Changes are hard
.raw vs. analyzed fields

52. Summary
ELK Stack seems like a good fit for mobile analytics.
Kibana PlugIns will help with missing Features.
Logstash PlugIns and Client Side Library needed in
order to improve Analytics capabilities.
It needs a bit of time and work but it looks promising.

53. Danke!
Danke!