Testing IoT considerations in agile digital world

1. Transforming Agile Digital Quality
Testing in the IoT Era
Amir Rozenberg, Director of Product

2. BUT: I still have ~20 GitHub projects!
• Product manager by day, coder at night
• Responsible for the core platform at Perfecto
• Publications: DZone, CIO Magazine,
MobileTestingBlog.com etc.
• I toy with
• Security & performance testing
• Globalization
• Elasticity
• Music and video streaming quality
• …
Amir_rozenberg

3. 10/5/2017 3© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Agenda
IoT Driving Digital Transformation
Quality Requirements for Healthcare Apps
Achieving Success with IoT Apps
Q&A

4. It’s Here
It’s Massive
It’s Impactful

5. How does IoT fit into your Agile Strategy?

6. Different IoT Segments
Source: Charalampos Doukas

7. Digital IoT Healthcare Improves Patient Care

9. IoT Healthcare Example: Application Architecture
(Pro/Reactive)
Sensors
Connected
Apps
Cloud
Operations
Healthcare
Ecosystem
• Are the measurements
correct?
• Is the action correct?
• Is the data sent correct?
• Given data, does the app
behave as expected?
• Feature/function
• Compliance
• Backend processes
• Sharing
• Compliance

10. Testing IoT
Sensors

11. Sensors Requirements
• Proactive, Invasive
• Independent (not relying on
backend availability)
• Class 3 FDA approval
• Certification against every
device, OS, version,
Bluetooth version
• Cannot leave premises
• Retail/Medical
• Reactive- informational
• Semi-Independent

12. How to test the sensors? : Firmware approach
Test Considerations
• LONG development cycles
• Extensive compliance requirements and process
• Follows embedded SW practices
Examples of Test Practices
• Testing mostly HW/FW oriented
• Connectivity (BLE): On/Off
• No API- No way to automate
• Test bench onprem
• Scale challenges (especially proactive devices)
• Inability to ‘shrink time’

13. How to test the sensors? :
Simulators (fake device, same SW)
Examples of Test Practices
• Simulated sensor data
• Test application logic
• Test firmware update process
• Trend responsiveness of action
Test Challenges
• Vendor typically owns the simulator
• No/Few APIs, no automation
• No connectivity testing
• Test bench onprem

14. Testing real sensors is hard,
functions will move to
smartphones

15. Testing IoT
Connected Apps

16. Healthcare Applications
 Mission critical: saving lives and improving patient care
 Treatment research
 Streamlining healthcare processes

17. Data-driven testing
• Mainstream data
• Extreme data
• Data at different
frequency
• Long term data
• Mix of conditions
Auth
Data
intake
AUX
functions
Dev-Test-Staging-Prod
Stub Backend
Testing the App (Given sensor data)

18. Testing the App (Given sensor data)
BLENO Box
20-30 m
Value: 140 BPM
Object
(Json file)
Auth
Data
intake
AUX
functions
Dev-Test-Staging-Prod
Stub Backend

19. The Bleno Box
BLENO Box
Docker Image
Bleno 3rd Party Code (JS)
Spring Front-end
nodejs
Bluetooth Library
Linux
• Script sends ‘Create’ message
• Contains a Json file
• File is translated to JS code, based on
Bleno classes
• JS is executed with nodejs

20. IoT Data-Driven Testing (DDT): JSON File Example
// This file exemplifies the syntax used for creating a BLE Peripheral
// It defines the following Services: Health Thermometer, Blood Pressure, Heart Rate, Battery
name : "HMO", // The Peripheral name ("Health Maintenance Organization")
services : [ {// This service is defined according to https://www.bluetooth.com/specifications/…org.bluetooth.service.health_thermometer.xml
name : "Health Thermometer", // The service name may be used by a script for updating it
uuid : 1809, // Service assigned number as defined in the link above
characteristics : [ { // See https://www.bluetooth.com/specifications/…org.bluetooth.characteristic.temperature_measurement.xml
name : "Temperature Measurement", // The service name may be used by a script for updating it
uuid : 2A1C, // The Characteristic assigned number as defined in the link above
value : [
{float : 36.2}, // Temperature in centigrade (Floating point)
0x07 // Temperature type (See https://www.bluetooth.com/specifications/…org.bluetooth.characteristic.temperature_type.xml)
],
properties : [read, notify], // Characteristic is readable and notifiable
interval : 30000 // Notification interval is 30 seconds
},{
name : "Temperature Type", // See https://www.bluetooth.com/specifications/gatt/viewer?attributeXmlFile=org.bluetooth.characteristic.temperature_type.xml
uuid : 2A1D,
value : [0x06], // Mouth
…

21. Test Bed
Requirements

22. Your Success Criteria: Lab
Maximize Test
Coverage
Maximize
% of
Automation
Compliance
Scalable
Reporting

23. Your Success Criteria: Reporting
Maximize Test
Coverage
Maximize
% of
Automation
Compliance
Scalable
Reporting

24. Your Success Criteria: Network Traffic
Compliance
Maximize Test
Coverage
Maximize
% of
Automation
Compliance
Scalable
Reporting
HAR File
• What is impacting the responsiveness of my app?
• What servers am I interacting with?
• What data is exposed in my service API calls?

25. Your Success Criteria: Compliance
Maximize Test
Coverage
Maximize
% of
Automation
Compliance
Scalable
Reporting
Uncompromised Security
• Network
Operations team
• Support Desk
• Infrastructure
monitoring
24x7
14 Global data
centers
• High Availability
architecture
• Auto recovery
• Data recoveryWorld
Class

26. Let’s Summarize
Complexity,
compliance
Time, $, Resources
Divide and Conquer
(Test what you can)
Define, prioritize platforms, test
cases and data (Factors for IoT)
Set the lab: Assets, global,
scalable & compliant
Build service virtualization
Automate, Automate, Automate
Audit- ready reporting

28. @Amir_Rozenberg