Internet of Things (IoT), is physical objects identifying themselves and communicating using internet as medium. Gartner forecasts that 25 billion connected things will be in use by 2020. In this exploding IoT, things, users and cloud services connect using internet to enable new use cases and new business models across multiple markets and applications. Once products composed of mechanical and electrical parts, have now evolved to become complex systems that combine context aware things, software and connectivity that can interoperate. Ranging from automobiles, light bulbs to smoke detectors – even the watch you are wearing may already be embedded with connected software. The rapid adoption and popularity of interconnected devices is creating new opportunities and challenges. QA team especially are faced with challenges from both business and technical perspective. This paper highlight few of the challenges and provides potential solutions that testing team can consider.

1. Author: Rakesh Reddy Lokireddy
ZenQ (www.zenq.com)
3/25/2016
Challenges for QA team in the
world of Internet of Things

2. P a g e 1 | 9
Table of Contents
1. Executive Summary....................................................................................................................2
2. What is the IoT? ...........................................................................................................................3
3. Three C’s of IoT............................................................................................................................3
4. Impact of IoT on Industry and Technology ...........................................................................3
5. Challenges for QA teams...........................................................................................................6
5.1. Replication of production environment.........................................................................6
5.2. Test approach and skill set required to support end to end testing efforts .........6
5.3. Understanding how a product is used by end user across industries ...................7
5.4. Identifying and performing vulnerability testing with large number of entry
points7
5.5. Just testing is not enough; get into the wild..................................................................8
6. Conclusion:......................................................................................................................................9
7. References: .....................................................................................................................................9

3. P a g e 2 | 9
1. Executive Summary
Internet of Things (IoT), is physical objects identifying themselves and communicating
using internet as medium. Gartner forecasts that 25 billion connected things will be in
use by 2020. In this exploding IoT, things, users and cloud services connect using
internet to enable new use cases and new business models across multiple markets
and applications.
Once products composed of mechanical and electrical parts, have now evolved to
become complex systems that combine context aware things, software and connectivity
that can interoperate. Ranging from automobiles, light bulbs to smoke detectors – even
the watch you are wearing may already be embedded with connected software. The
rapid adoption and popularity of interconnected devices is creating new opportunities
and challenges.
QA team especially are faced with challenges from both business and technical
perspective. This paper highlight few of the challenges and provides potential solutions
that testing team can consider.

4. P a g e 3 | 9
2. What is the IoT?
The IoT (Internet of Things), links smart objects to the Internet. It can enable an
exchange of data never available before, and bring users information in a more secure
way(1)
(1)
http://www.cisco.com/c/en/us/solutions/internet-of-things/overview.html
IoT with its smart objects enables businesses to gather valuable insights in customer
behaviour. This in-turn helps businesses to focus on actionable events that can increase
service quality, efficiency, getting products to market faster and also to continuously
innovate to meet customer needs
3. Three C’s of IoT
At the heart of IoT are 3 C’s -
 Context aware Things (Sensors, RFID, Hardware)
 Communication medium (Protocols, Data brokers / cloud platforms)
 Computing (Data aggregation, Machine Learning, Data Science)
Content aware Things are devices that include Sensors, RFID’s, embedded software and
hardware required to communicate with external environment.
Communication medium is the networking component that assists the Things to
communicate with another thing or the external environment
Computing enables businesses to gather data and learn about user behavior and also in
making intelligent decisions
4. Impact of IoT on Industry and Technology
Organizations across the industries can use IoT to derive considerable cost savings,
learn user behavior better and increase revenues by enhancing process efficiency,
boosting productivity. Below table highlights opportunities and benefits of IoT across
various industries,
INDUSTRY KEY CHANGE POTENTIAL BENEFITS
AUTOMOTIVE AND
TRANSPORTATION
Real-time driving behavior,
traffic and vehicle
diagnostics.
Improved customer
experience, reduced
pollution, increased safety
and additional revenue
streams
HEALTHCARE Remote monitoring of staff
and patients ability to
Improved employee
productivity, resource
usage and outcomes that

5. P a g e 4 | 9
locate and identify status
of equipment
result in efficiency gains
and cost savings
MANUFACTURING Quick response to
fluctuations in demand;
maximized operational
efficiency, safety and
reliability, using smart
sensors and digital control
systems.
Enhance agility and
flexibility, reduced energy
consumption and carbon
footprint
RETAIL Stock-out prevention
through connected and
intelligent supply chains.
Ability to predict consumer
behavior and trends, using
data from video
surveillance cameras,
social media, internet and
mobile device usage
SUPPLY CHAIN Real-time tracking of parts
and raw materials, which
helps organization
preempt problems,
address demand
fluctuations and efficiently
manage all stages of
manufacturing
Reduced working capital
requirements, improved
efficiencies and avoidance
of disruptions in
manufacturing
INFRASTRUCTURE Smart lighting, water,
power, fire, cooling, alarms
and structural health
systems
Environmental benefits
and significant cost savings
with better utilization of
resource and preventive
maintenance of critical
systems
OIL AND GAS Smart components Reduced operating costs
and fuel consumption
INSURANCE Innovative services such as
pay-as-you-go insurance
Significant cost savings for
both insurers and
consumers
UTILITIES Smart grids and meters More responsive and
reliable services;
significant cost savings for
both utilities and
consumers resulting from
demand-based and
dynamic pricing features
The 3 core components of IoT are connected and managed by a stack of technology
solutions available in the market. Below diagram highlights an example IoT stack,

6. P a g e 5 | 9
IoT Stack
Sensors /Hardware
/ Firmware
Device
Management &
Intelligence
•Device Hub /
Gateway
•Device
Management
Application PaaS
(aPaaS)
•API Design / build
•API runtime
management
User Interface
•Websites
•Utilities (appliances,
touch consoles etc)
•Mobile Apps

7. P a g e 6 | 9
5. Challenges for QA teams
There are a plethora of applications / platforms under development to support IoT
ecosystem. Organization need testing of these applications and for traditional QA
teams, this poses a variety of challenges. From both technical and business
perspectives, following details challenge and potential solutions,
5.1. Replication of production environment
The challenge: QA environment need to be a replication of production
What is at stake: Consider health and fitness application like Fitbit, MapMyRide, The
Neura where millions of devices are tracking and sending information to servers.
Replication of the same in test / stage environment may not be practical plus time &
cost consuming
Potential solutions: Building Protocol simulators and data recorder can come in handy
when there is a need for huge variety of device end points and interfaces to validate
and can also be great enabler in compatibility testing of application across device sets
and communication layers
5.2. Test approach and skill set required to support end to end
testing efforts
The challenge: An approach with focus only on functional testing aspects and a team of
experts in functional testing is not enough
What is at Stake: With several million sensors and different types of devices providing
the experience in conjunction with intelligent software, mere functionality validation is
not enough in an environment as complex as IoT Test environment that simulates huge
volumes, velocity and variety of data generated to verify validation of service and data
layers.
Potential Solution: Test approach should consider end to end testing perspective before
splitting them into various layer with following,
 Consider splitting testing into three layers
o Device interaction (Layer in which hardware and the software integrates)
 Compliance checks
 Interoperability
 Identify and authentication management
 Disaster & Recovery testing
 Data protection
 Security aspects
o User interaction (Layer considered as touch point for end user)

8. P a g e 7 | 9
 Usability and user experience
 Compatibility
 Front end validation
 Network and device level tests
 Integration testing
o Business logic (Layer where the data is analyzed and actionable data is
relayed to a device / end user)
 Functional testing
 Integration testing
 API testing
 Upskill / form a team of tester with mix of skills, combining QA, quality
engineering and hardware validation (compliances)
 QA for hardware software convergence need collaborating team
5.3. Understanding how a product is used by end user across
industries
The challenge: Solution developed in this space can be customized to use across
Industries. Consider a solution developed to Bid, Buy, Transport and Deliver
commodities like Oil to end user. In the event of Oil spill over from the cargo ship, an
appropriate event should be triggered to all the impacting agencies, in this case,
Environment agency for clean-up, Insurance company for claim proceeds, Financial
trader to inform their client, logistics & warehouses.
What is at Stake: Not having cross domain knowledge could potentially result in system
failures
Potential Solution:
- Build a team of testers with domain knowledge
- Upskill testers with specific domain knowledge
- Brainstorm application behaviour with Subject Matter Experts, Business Analysts,
Developers and Tester
- Consider creating test cases that test end to end user cases
5.4. Identifying and performing vulnerability testing with large
number of entry points
The challenge: Thousands of smart, connected devices like connected cars, robotic
manufacturing, smarter medical equipment, smart grid, and countless industrial control
systems deliver new experiences to people, lowering costs. Unfortunately, this growth

9. P a g e 8 | 9
in connected devices brings increased security risks. Threats quickly evolve to target this
rich and vulnerable landscape.
What is at Stake: Serious risks include physical harm to people, prolonged downtime,
and damage to equipment such as pipelines, blast furnaces, and power generation
facilities.
Potential Solution: IoT Security can be covered with four cornerstones,
- Protecting communications
o Protecting communication requires encryption and authentication for
devices to know whether or not they can trust a remote system. Newer
technologies like elliptic curve cryptography work ten times better than
predecessors.
- Protecting Devices
o Protecting devices against attack requires both code signing, to be sure all
code is authorized to run, and run-time protection, and to be sure
malicious attacks don’t overwrite code after it is loaded. Code signing
cryptographically ensures code hasn’t been tampered after being “signed”
as safe for the device, and it can be done at “application” and “firmware”
levels. All critical devices, whether a sensor, a hub, or anything else,
should be configured to only run signed code and never run unsigned
code.
- Managing Devices
o Unfortunately, vulnerabilities will eventually be discovered in valuable
devices that will then need to be patched long after they shipped. Even
obfuscated code for critical systems will be reverse engineered,
vulnerabilities discovered, and updates required. Nobody wants to send
employees to physically visit each device for updates, especially if that
involves a fleet of trucks. For such reasons, over-the air (OTA)
manageability must be built into devices before they ship
- Understanding Your System
o Of course, no matter how well you lock everything down, and no matter
how well you manage your systems, some threats can defeat all of those
countermeasures to establish a toehold in your systems. For such
reasons, it’s crucial to have an IoT Security Analytics capability that helps
you best understand your network by helping you flag anomalies that
might be suspicious or dangerous, malicious or not
5.5. Just testing is not enough; get into the wild
The challenge: Application developed in IoT span across Industry and requires testers to
test the system as an end user. For example, consider an application like The Neura,

10. P a g e 9 | 9
which sends a predefined alter to your travel buddy when you are 10 mins away from
pick up place. In this scenario, large emphasis is required on use case scenario as
opposed to field validation
What is at Stake: Risks like customer retention, loss of business to loss of life
Potential Solution:
- Test approach should consider Use case scenarios’ as top priority
- Tester can no long sit in a lab and perform test validation, they have to get into
the wild to learn behavior of the application as required
- Test labs should be created with adequate tools to similar real time scenarios
6. Conclusion
Testing teams should consider these challenges as opportunities to build protocol
simulators, test data generators, equip testing teams with domain knowledge and
define test approach that can scale per the business needs.
7. References
- www.gartner.com
- Research material from Forrester research, IBM and McKinsey&Co
- www.Semantic.com