AI is the new ELECTRICITY - said Andrew Ng. There are two sides of the coin. There are a lot of nay-sayers for AI. At the end of the day, it will be Augmented Intelligence, Adaptive Intelligence, Automated Intelligence that will propel human intelligence forward - more than anything else. It will be a great time ahead. Whether it would be an "Eye(AI) Wash" as skeptics say or an "I wish" from them for starting late on the journey, only time will tell. It is a matter of when and how long, instead of an If. #ArtificialIntelligence #IntelligentTesting #QCoE #NextGenTesting #QualityFocusedDelivery #DigitalInnovation #ITIndustry #NewAgeIT #InnovativeTesting#AIFication #Automation #DigitalEconomy #Singularity #Transcendence #Futurism
2. • Progress of AI and Robotics
• What’s the need for Artificial Intelligence?
• What will happen at singularity?
• Some AI Concepts
High level Intro to AI
• Is it an Intelligent Activity?
• Are we testing at the heights of
Augmented General Intelligence?
• AI in Testing - Is it augmented or Artificial
/ Is anything artificial about it?
• How will AI evolve Testing?
• Some Examples of AI Testing
AI in Testing
Agenda
3. - ANDREW NG
Founder of Coursera, Stanford Adjunct Professor
Ex. Chief AI Scientist of BAIDU
5. Source : https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/
A Good Explanation of Progress of AI
E a r l y A I
Basic Turing Test Style
Use of Memory and Knowledge
Post John McCarthy’s Conceptualization
Basic Robotics and Degrees of Freedom
D e e p L e a r n i n g
Rapid Infrastructure Growth
Advanced Algorithms
Big Data Explosion
Quantum Computing
M a c h i n e L e a r n i n g
Algorithms Centric
Statistics Driven
Supervised and Unsupervised Learning
6. Perhaps the greatest
Computer Scientist ever
predicting on Machine
Intelligence
We have clearly passed
the TURING TEST
We are seeing Leaps and
Bounds in advances of
Technology!
Let’s hear!
8. https://youtu.be/zatL4uFRpC0
Fast Learning – Download and Fly an helicopter
Can AI Take us to this stage?
How about Fast Testing – Hey – Can I test this brand new “thing” in 2 minutes?
9. From a Leader in AI – AI or an Algorithm Writing itself
10. Top Human World Champions Royally defeated by AI!
2011
2016
1996/97
IBM’s
Deep Blue
IBM
Watson
Google
Deepmind
AlphaGo won 60–0 rounds on two public Go websites
including 3 wins against World Go champion Ke Jie.
11. But... AI is not without controversies though!
Facebook Researchers shut down an AI
engine at the Facebook AI Research Lab
(FAIR), discovering that the AI created
its own unique language undecipherable
by humans - Simultaneous glimpse of
both the awesome and horrifying
potential of AI
Elon Musk - “AI isotentially more
Dangerous than Nukes”
sets up a $1 billion (£770M) OpenAI.org to try
and promote safe development of AI
Vladimir Putin -
“Whoever masters AI will rule the world!”
ISAAC ASIMOV’s Laws of Robotics
Law 1: A tool must not be unsafe to use.
Law 2: A tool must perform its function
efficiently unless this would harm the user. The
safety of the user is paramount.
Law 3: A tool must remain intact during its use
unless its destruction is required for its use or for
safety.
15. AI Startups are taking it to next Level – in all areas
Source: https://www.cbinsights.com/research-ai-100
Bots Automobile
Computer
Vision
Core /
Functional AI
Commerce IIoT/IOT
Healthcare Fintech Robotics
Analytics
Cyber
Security
Sales &
Marketing
17. “ Let’s get to TESTING
• How is AI helping
Testing?
• How can we test
better with AI?
• How can we test AI
systems Better?
17
18. “Be a yardstick of
QUALITY. Some
people aren't used
to an environment
where EXCELLENCE
is expected”
18
Steve Jobs
19. Business Agility - Some Statistics
19
Google - refactors code by
50% each month*
Netflix - 5 Billion+ API
Calls per Day (and
increasing daily)
~75% of Corporates to
have bi-modal IT
~63% all projects are not
aligned to Business
Strategy
~79% organizations using
CI/CD/DevOps practices in
one form or the other
52% of Fortune 500
companies have
disappeared from the list
& Average S&P500 span
reduces from 61 Years to
17 Years in 60 years
In 2020, 100 million
consumers will shop via
augmented reality
By 2020, 30% web
browsing will be done
without a screen
by 2022 - $1 Trillion a year
to be saved through IoT
Source: Gartner, Inc. Top Strategic Predictions for 2017 and Beyond: Surviving the Storm Winds of Digital Disruption, 14-Oct-2016
* - Google runs on ~2 Billion LOC Source: CA Workshop on Modern Software Factory
Source: CB Insights
* AR Market $143 Billion by 2020 - HW/SW/Apps/Consulting & SI
20. Is the Testing Industry ready for testing the
following innovations?
21
21. Tip of the Iceberg seen in 2016
2016 A Year in Review – Software Failures
22Source: Tricentis Software Fails 2016 Report - https://www.tricentis.com/wp-content/uploads/2017/01/20161231SoftwareFails2016.pdf
Over 4.4 Billion
people got affected by
a Software Fail (Up
from 4.3 Billion in
2015) > 50% Global
Population
$1,062,106,142,949
- Assets Affected (Up
from $4.2 Billion in
2015)
315 years, 6 months,
2 weeks, 6 days, 16
hours, & 26 minutes -
Accumulated time-
lost due to Bugs
2.66 Billion Mobile
Phones impacted
with Malware
12% Year on Year
Increase in impactful
Software Bugs
British Airways lost
$20 Billion (3%) in
Market Cap within a
few days after a failed
software upgrade
More Than 21
Million Automobile
recalls as a result of
Glitches / Bugs
$5.7 Billion Impact
in Failed Government
Software Projects due
to Bugs
2.2 Billion people live on less than $2 a day
22. One School of Thought on Testing – By Tricentis
Source: TRICENTIS webinar on Future of Testing
WhereAIcanhelp
Legacy
Firms
Bi-model Firms
Technology Leaders
25. Some Algorithms making Machine Think!
Source: https://futurism.com/predicting-2017-the-rise-of-synthetic-intelligence/ - Some of the artificial intelligence (AI) algorithms currently helping machines think. Credit: CIO Journal/Narrative Science
26. Approaches used for AI, Machine Learning and Deep Learning
Reinforcement Learning
• Passive Reinforcement
Regression Algorithms
• Linear Regression
• Gaussian Process
Supervised Learning
• Neural Networks
Unsupervised Learning
• Independent Component
Analysis
• Principle Component
Analysis
Natural Language
Understanding
• Morphological, , semantic,
syntactic , Discourse
analysis
Natural Language
Generation
• Deep planning
• Syntactic generation
Clustering Algorithms
• K-Means Clustering
• KPCA – Kernel Analysis
Statistical Algorithms
• Support Vector Machines
• K-Nearest Neighbor
• Native Bayes Classifier
• Maximum Entropy Classifier
Pattern Recognition
• Statistical , Syntactic
approach
• Template Matching
• Neural Networks
Other Techniques
• Spanning Trees and Graphs
• Neural Network – Multi-
Level Perceptron's
Other Techniques
• Labeling
• Hidden Markov Model
• Maximum Entropy MM
Other Techniques
• Conditional Random Fields
• Parsing Algorithms
28. What are the feasibilities
with AI Driven Testing?
30
Automated Defect
Detection
Automated
Exploratory
Testing
Test Coverage
Heat map
Self Healing
Automation
Predictive
Modeling
Self Adjusting
Regression
Pattern
Recognition
Risk & Coverage
Optimization
Diagnostic,
Prescriptive and
Predictive Analysis
Deep Learning
Root-Cause
Analysis
Sentiment Analysis
29. 31
AI Models Algorithms
Application Under
Test
Designer Developer Business UserTesterBots / Agents
AI Engine
Testing Outcomes
Test Cases
Production
Logs
Requirements
Defect Logs Source Code
Traceability
Matrix
Root Cause
Analysis
Test Data
Specifications
Functional
Logic
Sample AI Model for Testing
Historical & Real-time Data
31. Example: Candy Crush Saga’s AI Strategy
https://www.youtube.com/watch?v=wHlD99vDy0s
• Use of AI engine for continuous Feedback Loop
• Use of BOTS to perform Testing
• Continuous Feedback Loop
• Deep Artificial Neural Network
• Use of Monte Carlo Tree Simulation
• Use of Advanced Automation by BOTS
• Hybrid Test team (150-200+ Testers) with unique skills
• Use of Data Scientists for Domain Knowledge, Fun (using
historic info and user behavior, Game Balancing)
• Regular Crash Testing, Performance Testing, Regression
Testing
• Regular Upgrade of AI Bot for Testing
v
Since John McCarthy invented AI in 1956 – Progressed by Marvin Minsky Etc.
Ray Kurzweil – Highlights that Singularity will happen by 2045
https://youtu.be/zatL4uFRpC0
Fast Learning – Download and Fly an helicopter – Can AI be so advanced that we can scan an image and learn faster, deeper and efficiently?
Let’s hear it from one of the leaders in the AI Space – AI or an Algorithm Writing Code – Watch out Developers and Testers!
Garry Kasparov
Lee Seoul
Ken Jennings and Randy Burr
Google DeepMind's AlphaGo won 60–0 rounds on two public Go websites including 3 wins against world Go champion Ke Jie.
AI induced Algorithms have been winning a tough game of Texas Hold’em poker where majority of the information is hidden – against world’s leading Poker Players as well.
AI is definitely not without controversies – It could potentially start the WW3 soon and a lot of countries are embarking on hegemony and superiority of AI – Just like the Cold-war era SPACE RACE that resulted in a lot of brilliant inventions, discoveries and humankind’s progress. But will the new AI war be different – Let’s wait and see..
AI is definitely not without controversies – It could potentially start the WW3 soon and a lot of countries are embarking on hegemony and superiority of AI – Just like the Cold-war era SPACE RACE that resulted in a lot of brilliant inventions, discoveries and humankind’s progress. But will the new AI war be different – Let’s wait and see..
Google is using machine learning and deep learning principles in a simple method. Let’s see a video and try it out!
Take it to next level – How can you easily integrate globally. Language will no longer be a barrier.
A perfectionist of sorts, Steve Jobs quoted - “Be a yardstick of quality. Some people aren't used to an environment where excellence is expected”
Without a focus on quality, simplicity and efficiency, APPLE wouldn’t have become the most valuable company on earth, a brilliant turn around from a company that was almost dead before Jobs 2.0 began.
Source: CA Workshop on Modern Software Factory
Source: Gartner, Inc. Top Strategic Predictions for 2017 and Beyond: Surviving the Storm Winds of Digital Disruption, 14-Oct-2016
While the defects and bugs are making a dramatic impact, the world is leaping ahead. Business is expecting agility in business delivery...
Take these for some stats
Google – which supposedly has a single code repository, refactors code by upwards of 50% each month. They have ~2 Billion LOC (and counting). Even taking a 75-95% test coverage taken up by empowered teams (as claimed by some of the engineers in published artefacts), this is a humongous testing effort. If you have a backlog of code to be verified, it could be a disaster exceeding the size of a titanic by all means. When the Cyclomatic complexity of testing is so huge, how can you test the entire code base and application flawlessly? This is a brilliant example of how one can run an efficient test strategy.
Take NetFlix that currently has over 5 Billion API Calls per day (up from Billion+ a few years ago). How would you do effective Load, Stress, Performance Testing and ensure Availability , Redundancy and Reliability of service is not impacted?
A lot of firms are moving towards a bi-modal IT (doing a transformation while running the legacy apps running) and doing continuous delivery and Testing all the time, leveraging all the fancy words such as Agile, DevOps DevQAOps etc. etc.
Additionally, nearly 41% of Global corporate workload is shifting to cloud, to ease out on Capital Expense and controlled Operational Expense strategies.
By some means, Augmented Reality, Gestural Computing, IoT is expected to take the world by storm. How are we going to test all these permutations?
AR Market $143 Billion by 2020 - HW/SW/Apps/Consulting & SI
If you take the Gartner’s Hype Cycle for Emerging Technologies for 2017 – You see a pattern. Some are in the slope of enlightenment but majority in the curve of inflated expectations and disillusionment. For the technologies emerging stronger, we need to have some solid test approach / strategy to deliver high quality outcomes
Artificial Intelligence
Internet of Things (or Everything)
Machine Learning / Deep Learning
AR/VR & Wearables
Block Chain
Drones & Vehicles
Gestural Computing
Connected Devices
Human Augmentation
Robotics
Algorithms
Smart Assistants
Are we ready for these?
Carrying on Quality - Some statistics or a tip of the ice-berg
Over 4.4 billion people got affected by a software fail – which is greater than 50% of Human Population. It is almost a number similar to the people not having access to a Toilet – But less than the number of mobile phones in use in the world!.
More than a $Trillion worth of assets affected and a cumulative impact of 315 years.
A leading airlines lost 3% market cap due t a botched software upgrade infested with bugs
Broadly speaking – New Age Test Innovation focuses on the following needs with Intelligent Testing
Rapid High Quality and Innovative Test Delivery
Test Suite Creation and Optimization (Risk Coverage )
Useful Automation – Test Smart, Self-Healing, Script less, Purposeful
Predictive and Cognitive Testing – Foresee issues reduce reactive time, resolve rapidly
Rapid Impactful Defect Finding - Intelligent Defect Detection, Pattern Analysis, Predictive Modeling
Intelligent Environment Provisioning
Management with Intelligent Metrics and Dashboard
Are we capable of building intelligent automated frameworks and leverage cognitive models to optimize our test strategy and test suites to do proactive application health analytics via rapid defect finding and scale up rapidly to do niche and special areas of testing? That remains the key
https://www.linkedin.com/pulse/ai-software-testing-jason-arbon
Explore user experience, by analyzing text from social media feeds (sentiment analysis) to spot feedback trends about what has already been released
Cluster similar bugs together by data visualization heat mapping, for easier attack by Development (via the Pareto Approach, theorizing that bugs like to nest together)
Reduce test cases that can be determined to be unnecessary before execution?
Predict if specific follow-up DevOps sprints require specific tests cases to be run, vs. being omitted because there’s no chance that the problem got addressed yet.
Reviewing specifications tell us what a program should do and how it should work. I.’s pattern matching helps us eliminate unneeded “too close’ test cases by seeing which ones are too similar. As we mentioned earlier, this may mean focusing on boundary value analysis (edge cases, literally), emphasizing state transition, or ensuring all-pairs testing.
Exploration testing session logs, via pattern recognition of verbose logging, seek activity patterns of specific warnings tracking to specific user actions, modules, forms, etc.
Known product issues, once analyzed, can have A.I. cluster similar bugs through pattern recognition, suggesting likely duplicates. Bugs from automated test cases can be auto-run on previous builds to find the causal build to help pinpoint root cause code changes.
Discussions with knowledgeable personnel (product owners, developers or Marketing, etc.) may determine code danger areas. White box-driven test design targets the actual revised code, hunting for specific code level problems. Factors may include the coder, change date, functions referenced, or specific non-standard notations. A.I. pattern-matching techniques help pinpoint applicable code based on your search parameters.
End user analysis applies to two different areas. The first is studying the frequency of specific user feedback words to help the most popular concerns bubble to the top of a list for further research. The second is end user usage analysis, where log file statistics (based on A.I. pattern searching) show how much time each type of user spends in different program areas on different actions. Early focus on these heat mapped areas concentrate attention where the most user time is spent.
Test suite optimization - Identifies duplicate/similar and unique test cases
Predicting the next - To help predict the key parameters of software testing processes based on historical data.
Log Analytics - Identifies hotspots and automatically execute test cases
Traceability - Identifies complex scenarios from the requirements traceability matrix (RTM) and extract keywords to achieve test coverage
Customer sentiment Analytics - Analyzes data from social media and provides an interactive visualization of feedback trends
Defect analytics - Identifies high-risk areas in the application which helps in risk-based prioritization of regression test cases
Its benefits include:
Improved quality – Prediction, prevention, and automation using self-learning algorithms
Faster time to market – Significant reduction in efforts with complete E2E test coverage
Cognitively – Scientific approach for defect localization, aiding early feedback with unattended execution
Traceability – Missing test coverage against requirement as well as, identifying dead test cases for changed or redundant requirement
One integrated platform – Adaptable to client technology landscape, built on open source stack
Leave the exhaustive testing to AI. Leave tapping every button, inputting obvious valid and invalid data into text fields, etc. to the machines.
Focus on the qualitative aspects of software testing that is specific to their specific app and customer.
Focus on creative and business-specific test inputs and validations. Be more creative and think of email address values that a machine with access to thousands of possible email test inputs wouldn’t think to try. Verify that cultural- or domain-specific and expectations are met. Think of test cases that will break the machine processing for your specific app (e.g., negative prices, disconnecting the network at the worst possible time, or simulating possible errors).
Record these human decisions in a way that later helps to train the bots. Schematized records of input and outputs are better than English text descriptions in paragraph form