Teste combinatório de software

Introdu¸˜o
ca Constru¸˜o de arranjos de cobertura
ca Modelos generalizados

Teste Combinat´rio de Software
o

Lucia Moura
School of Electrical Engineering and Computer Science
University of Ottawa
lucia@eecs.uottawa.ca

Palestra no SECCOM, INE, UFSC, 2012

o Lucia Moura

Introdu¸˜o

Trajet´ria geogr´ﬁca e acadˆmica
o a e Para ver todos os detalhes exibidos na tela,
use o link Imprimir ao lado do mapa.

Professora (2000-...)
EECS, University of Ottawa,
Ottawa, Canada

PhD (1999)
DCC, University of Toronto,
Toronto, Canada

Graduação e mestrado (1987, 1992)
IME, USP,
São Paulo, Brasil

Imagens ©2012 TerraMetrics, Dados cartográficos ©2012 Google, INEGI, MapLink, Tele Atlas -

o Lucia Moura

Introdu¸˜o

Ottawa, Canad´
a To see all the details that are visible on the
screen, use the Print link next to the map.

EECS

University of Ottawa

Imagery ©2012 Cnes/Spot Image, DigitalGlobe, GeoEye, Map data ©2012 Google -

o Lucia Moura

Introdu¸õ

o

Queremos testar um sistema:
um programa
um circuito
um pacote que integra v´rios softwares
a
v´rias plataformas diferentes em que um pacote tenha que
a
funcionar
um software altamente configur´vel
a
uma gui interface
uma aplica¸õ na “cloud”
ca

Queremos uma bateria de testes que dˆ uma boa cobertura do
e
sistema para detectar erros/bugs/falhas.

o Lucia Moura

Introdu¸õ

o
Primeiro isolamos os parˆmetros do sistema e poss´
a ıveis valores
os parˆmetros de entrada de um programa e seus valores
a

5, 10, 20, 25, 30
<100,000
Introduction [100,000-250,000) 6
[250,000-450,000)
> 450,000

17 opções
Example 1.1.2. For our second example, we look at the circuit diagram in11 opções 1.1.
Figure
This is a small circuit, with only five inputs - a, b, c, d, and e - each of one bit, and
three outputs. This circuit could be exhaustively tested using 25 = 32 tests, but based
6 opções
on the fact that some of the inputs do not interact, doing so is unnecessary. We instead
show that we can fully test the circuit using only eight tests by using a variable strength
(5, 4, 11, 17, 6)
covering array. We represent the circuit as a hypergraph, with one vertex for each input,

as entradas de um circuito: 5 entradas bin´rias
and one hyperedge for each output, as shown in Figure 1.2.
a
a
NAND x

b
XNOR y

c
AND

d XOR z

NOT

(2, 2, 2, 2, 2)
AND

e

Figure 1.1: An example of a small circuit with five inputs and three outputs.

componentes de uma plataforma e suas configura¸˜es
co
Component
Web Browser Operating Connection Printer
System Type Config
Config: Netscape(0) Windows(0) LAN(0) Local (0)
IE(1)Figure 1.1, taking each input
Figure 1.2: The hypergraph that arises from the circuit in Macintosh(1) PPP(1) Networked(1)
as a vertex and each output as a hyperedge overOther(2)
the inputs contributing to Linux(2)
its value. ISDN(2) Screen(2)

Table 3: Four components, each with 3 configurations
We now give a variable strength covering array for the circuit that requires only eight
(3, 3, 3, 3)
different test cases, with the guarantee that for each output, every possible combination
Teste Combinat´rio de Test Case Browser
o Software
of inputs have been tested together in some test case. OS Connection Printer Lucia Moura

Introdu¸õ

Teste combinat´rio de pares (pairwise testing)
o
Testando um sistema com k = 4 componentes cada um com v = 3
valores: Component
Web Browser Operating Connection Printer
System Type Config
Config: Netscape(0) Windows(0) LAN(0) Local (0)
IE(1) Macintosh(1)
ComponentPPP(1) Networked(1)
WebOther(2)
Browser Linux(2)
Operating Connection Screen(2)
ISDN(2) Printer
Table 3: Four components, each with 3Type
System configurations Config
4
Testar todas as possibilidades: 3Windows(0) Connection PrinterLocal (0)
Config: Netscape(0) = 81 testes. Networked(1)
Test Case Browser Macintosh(1)
IE(1) OS
LAN(0)
PPP(1)
1 NetScape Linux(2)
Windows LAN
ISDN(2) Local Screen(2)
Teste combinat´rio de pares de parˆmetros pode ser feito com 9 testes.
o 2
Other(2)
NetScape a
Linux ISDN Networked
Table 3: Four components, each with 3 configurations
3 NetScape Macintosh PPP Screen
4 IE Windows ISDN Screen
Test Case Browser Macintosh Connection Networked
5 IE OS LAN Printer
16 NetScape Windows
IE Linux LAN
PPP Local
Local
27 NetScape Windows
Other Linux PPP
ISDN Networked
Networked
38 Other Linux
NetScape Macintosh LAN
PPP Screen
Screen
49 Other
IE Macintosh
Windows ISDN
ISDN Local
Screen
5 Table 4: Test SuiteMacintosh Pairs from Table 3
IE to Cover all LAN Networked
6 IE Linux PPP Local
7 Other Windows PPP Networked
agriculture and manufacturing [64]. It has entered the software testing community, appear-
8 Other Linux LAN Screen
ing in practitioner’s guidebooks [70, 89], and provided in simple spreadsheet formats [40, 41].
The use of covering arrays in software testing was pioneered by Mandl Local Brownlie et
9 Other Macintosh ISDN [83] and
al. [9, 102], and statistical foundations were explored in [46, 47, 48, 49, 82, 91]. Empirical
Table 4: (examplo deinteractions in a software system
Colbourn 2004)
results indicate that testing Test Suite to Cover all Pairs from Table 3 indeed finds a
of all pairwise
large percentage of existing faults [45, 78]. Indeed, Burr et al. [11] provide more empirical
agriculture and manufacturing [64]. It coverage leadsthe useful code coverage as well. Dalal
results to show that this type of test has entered to software testing community, appear-
Arranjo de cobertura com forca t = 2, k = 4 parametros, v = 3 valores
ing et al. present empirical results to89], and provided in simple spreadsheet formats in a 41].
in practitioner’s guidebooks [70, argue that the testing of all pairwise interactions [40,
software system finds a large percentage of the existing faults [45]. Dunietz et al. link the
The use of covering arrays in software testing was pioneered by Mandl [83] and Brownlie et
cada cobre todas as interacoes dois a dois com N = 9 tests.
effectiveness of these methods to software code coverage. They show that high code block
al. coverage is obtained when testing all two-way interactions, but higher48, 49, sizes 91]. needed
[9, 102], and statistical foundations were explored in [46, 47, subset 82, are Empirical
results good paththat testing of all pairwise interactions reports for three software systems. a
for indicate coverage [54]. Kuhn et al. examined fault in a software system indeed finds
Teste Combinat´rio de
o Software Lucia Moura

shown to work across a variety of combinations of operating systems, databases, and
Introdu¸˜o network
ca characteristics. Constru¸˜o de arranjos de cobertura

Teste combinat´rio we hadpares operating system (Windows XP, Apple OS X,
For example, suppose
de an application that is intended to run on a variety of
platforms comprised o five components: an (pairwise testing)
of
Red Hat Enterprise Linux), a browser (Internet Explorer, Firefox), protocol stack (IPv4,
Arranjo ⋅processor (Intel,platforms.and a database tests, shownSybase, Oracle), possible of
IPv6), a
de cobertura:
3 ⋅ 2 ⋅ 2 ⋅ 2 3 = 72 possible
AMD),
With only 10
(MySQL,
in Table 1, it is
a total
to
for¸a every component5 parˆmetros, valores (3, 2, 2, least once, i.e., all possible
test t = 2, k = interacting with every other component at 2, 3), N = 10 testes
c a
pairs of platform components are covered.

Test OS Browser Protocol CPU DBMS
1 XP IE IPv4 Intel MySQL
2 XP Firefox IPv6 AMD Sybase
3 XP IE IPv6 Intel Oracle
4 OS X Firefox IPv4 AMD MySQL
5 OS X IE IPv4 Intel Sybase
6 OS X Firefox IPv4 Intel Oracle
7 RHEL IE IPv6 AMD MySQL
8 RHEL Firefox IPv4 Intel Sybase
9 RHEL Firefox IPv4 AMD Oracle
10 OS X Firefox IPv6 AMD Oracle

Table 1.taken from Khun, Kacker and Lei 2010)
(example Pairwise test configurations

testar todas as possibilidades (t = 5): 32 × 23 = 72 testes
teste de pares (t = 2): 10 testes 7
o Lucia Moura

Introdu¸˜o

Teste combinat´rio de pares (pairwise testing)
o
Arranjo de cobertura:
for¸a t = 2, k = 5 parˆmetros, valores (3, 2, 2, 2, 3), N = 10 testes
c a

(example taken from Khun, Kacker and Lei 2010)

testar todas as possibilidades (t = 5): 32 × 23 = 72 testes
teste de pares (t = 2): 10 testes
o Lucia Moura

Introdu¸õ

Por que usar teste de pares (pairwise testing)?

Economia: usamos um n´mero m´
u ınimo de testes.
exemplo: k = 20 parˆmetros com v = 10 valores cada.
a
testar todas as combina¸˜es:1020 testes(em geral = v k )
co
teste de pares:155 testes (em geral em O(v 2 log k))
Robustez: temos uma boa cobertura de erros na pr´tica.
a
a maior parte dos erros em software (75%-80%) sõ causados
a
por certos valores ou por intera¸õ de 2 valores.
ca
“Evaluating FDA recall class failures in medical devices... 98% showed that the problem could have been
detected by testing the device with all pairs of parameter settings.” (Wallace and Kuhn, 2001)
Cohen, Dalal, Fredman, Patton (1996) - AETG software
Dalal, Karunanithi, Leaton, Patton, Horowicz (1999)
Kuhn and Reilly (2002)

cobertura de pares implicam outras medidas de cobertura.
“Our initial trial of this was on a subset Nortel’s internal e-mail system where we able cover 97% of
branches with less than 100 valid and invalid testcases, as opposed to 27 trillion exhaustive testcases.”
(Burr and Young, 1998)
“The block coverage obtained for [pairwise] was comparable with that achieved by exhaustively testing all
factor combinations ...” (Dunietz et al., 1997)
Cohen, Dalal, Fredman, Patton (1996, 1997) - AETG software

o Lucia Moura

Introdu¸õ

Aumentando a for¸a da cobertura (t-way coverage)
c
podemos usar outros valores intermedi´rios de for¸a entre
a c
t = 2 (pairwise) e t = k (testar todo o espa¸o de parˆmetros).
c a
Defects Total
o “tradeoff” ´ que aumentando o t, aumenta-se a robustez,
e
productivity more than doubling on
average and more than tripling in
found
per hour
2.4X
higher
defects
found
13%
higher

three projects. The groups using pair-
mas tamb´m aumenta-se o n´mero de testes
e u
wise testing also achieved the same
or higher quality in all 10 projects; Manual Pairwise Manual Pairwise

estudos mostram que normalmente t ∈ [2, 6] ´ suficiente para
all of the defects identified by the
teams using manual test case selec- e (a) Testing method (b) Testing method

tion methods were identified by the Figure 1. Summary of results from 10 projects. Pairwise combinatorial test case
detectar todos os errors
teams using combinatorial methods.
In five projects, the combinatorial
selection versus manual test case selection: (a) testing efficiency and (b) testing
Kuhn, Wallace e Gallo (2004)
quality.
teams found additional defects that
100
had not been identified by the teams
using manual methods.
These proof-of-concept projects
successfully demonstrated to the
75
teams involved that manual meth-
ods of test case selection were not
Cumulative percent

nearly as effective as pairwise com-
binatorial methods for finding the 50
largest number of defects in the least
amount of time.
Medical devices
TESTING HIGHER-DEGREE Browser
25 Web server
INTERACTIONS NASA distributed database
Other empirical investigations
have concluded that from 50 to 97
percent of software faults could be 0
1 2 3 4 5 6
identified by pairwise combinato- Interactions
rial testing. However, what about the
remaining faults? How many failures Figure 2. Cumulative error detection rate for fault-triggering conditions. Many faults
could be triggered only by an unusual were caused by a single parameter value, a smaller proportion resulted from an
interaction involving more than two interaction between two parameter values, and progressively fewer were triggered
by three-, four-, five, and six -way interactions.
parameters?
In a 1999 study of faults arising
from rare conditions, the National because it only guarantees that all by three-, four-, five,- and six-way
Kuhn, Wallace e Gallo (2004) of parameter values will be interactions. Figure 2 summarizes
Institute of Standards and Technology pairs
o reviewed 15 years of medical device tested. A particular four-way com- these results. Thus far, a fault trig- Lucia Moura

Introdu¸õ

Teste combinat´rio de software, arranjos de cobertura
o
if it contains the minimum possible number of rows. Various authors transpose the array in
Teste combinat´riocourse this isoccasionally constructpreference. In our discussions here, ca t
the definition, and of
o t-way requer of personal thede coberturaarray. for¸
we employ the N × k format, but
a matter
arranjos tranposed covering de
for¸a teach. 3;has Nexample of a covering array =strength three with tenN = havinglinhas
c = It isv = 13 rows.
Here an
=
2 s´ımbolos; k of 10 colunas; factors 13 two levels
0 0 0 0 0 0 0 0 0 0
1 1 1 1 1 1 1 1 1 1
1 1 1 0 1 0 0 0 0 1
1 0 1 1 0 1 0 1 0 0
1 0 0 0 1 1 1 0 0 0
0 1 1 0 0 1 0 0 1 0
0 0 1 0 1 0 1 1 1 0
1 1 0 1 0 0 1 0 1 0
0 0 0 1 1 1 0 0 1 1
0 0 1 1 0 0 1 0 0 1
0 1 0 1 1 0 0 1 0 0
1 0 0 0 0 0 0 1 1 1
0 1 0 0 0 1 1 1 0 1

Definicao: Arranjos objectCoberturain developing interaction tests when all factors
This combinatorial de is fundamental
have an equal number of levels. However, systems are typically not composed of components
Um arranjo de cobertura the sameca t, of parameters v s´
(factors) that each have exactlyde for¸numberk fatores,(levels). To remove this fatorımbolos por
limitation of covering arrays, the mixed-level covering array can be used.
e tamanho N ,level covering array MCA (N;t,k, (v v),. .´, vumaanmatrixarray. × k com
A mixed denotado CA(N ; t, k, , v , e )) is λ 1 .2 k N × k N Let
{i , . . . , i } ⊆ {1, . . . , k}, and consider the subarray of size N ×t obtained by selecting columns
1 t
s´
ımbolos. .deofumMCA. There arev-´rio G tal que em cada as rows, and an
i ,. ,i
1 tthe alfabeto a v distinct t-tuples that could appear sub-arranjo
t
i=1 i

t × N ,MCA requires smallest Nappear att leastcoberta pelo menos umav vez. ))
cada the that eachem which suchonce. We coveringnotation CAN(t, k, (v , , . . . , v
to denote
t-tupla for G ´ a mixed use the array exists.
e 1 2 k

An early investigation of covering arrays appears implicitly in Marczewski [84]. Rńyi
e
o [105] determined sizes of covering arrays for the case t = v = 2 when N is even. Kleitman
Teste Combinat´rio de Software Lucia Moura

Introdu¸˜o

Teste combinat´rio de software, arranjos de cobertura
o
Teste combinat´rio t-way requer arranjos de cobertura de for¸a t
o c
for¸a t = 3; v = 2 s´
c ımbolos; k = 10 colunas; N = 13 linhas

Deﬁnicao: Arranjos de Cobertura
Um arranjo de cobertura de for¸a t, k fatores, v s´
c ımbolos por fator
e tamanho N , denotado CA(N ; t, k, v), ´ uma matrix N × k com
e
s´
ımbolos de um alfabeto v-´rio G tal que em cada sub-arranjo
a
t × N , cada t-tupla em Gt ´ coberta pelo menos uma vez.
e
o Lucia Moura

Introdu¸õ

Minimiza¸õ de arranjos de cobertura
ca
Dados t (for¸a), k (n´mero de parˆmetros) and v (#valores).
c u a
Minimize N (#tests)
CAN (t, k, v) = min{N : existe um CA(N ; t, k, v)}.

Crescimento logar´
ımitico de arranjos de cobertura
CAN (t = 2, k, v = 2) = {min N : N −1 ≥ k} =
N/2
log k(1 + o(1)) (Katona 1973, Kleitman and Spencer 1973)
t = 2, v > 2 fixed, k → ∞:
CAN (t = 2, k, v) = v log k(1 + o(1))
2
(Gargano, Korner and Vaccaro 1994)
CAN (t, k, v = 2) ≤ 2t tO(log t) log k (Naor et al 1993,1996,1998)
CAN (t, k, v) ≤ v t (t − 1) log k(1 + o(1))
(Godbole, Skipper and Sunley 1996)

o Lucia Moura

Introdu¸õ

Minimiza¸õ de arranjos de cobertura e crescimento
ca
logar´
ıtmo

Dados t (for¸a), k (n´mero de componentes) e v (#valores).
c u

Minimize N (#testes)
CAN (t, k, v) = min{N : existe um CA(N ; t, k, v)}.

Para v e t fixos CAN (t, k, v) = O(log k).

Use o m´todo guloso de densidade (Bryce & Colbourn 2007).
e
M´todo guloso de um-teste-por-vez garante N = O(log k).
e

´
Otimo para teste de software: #testes cresce com o log do
#parˆmetros!
a

o Lucia Moura

Introdu¸˜o

Constru¸˜o de arranjos de cobertura m´
ca ınimos

m´todos combinat´rios: recursivos e diretos
e o
Survey: Charlie Colbourn, “Combinatorial Aspects of Covering
Arrays”, 2004 (34 paginas)
algoritmos
m´todos gulosos:
e
• AETG (D. Cohen, Dalal, Fredman, Patton 1996, 1997),
um-teste-por-vez, tenta aproximar garantia logar´
ıtmica
• m´todo guloso de densidade ( Bryce e Colbourn 2007),
e
um-teste-por-vez, garantia logar´
ıtmica
• IPOG algoritm (J. Lei), ACTS tool/NIST (Khun e Kacker):
alternˆncia de crescimento de linhas e colunas
a
m´todos de busca heur´
e ıstica
• tabu search: Zekaoui (2006), Torres-Jimenez (2012)
• simulated annealing: M. Cohen (2003-2008), Torres-Jimenez
(2010-2012)

o Lucia Moura

Introdu¸õ

Constru¸õ de arranjos de cobertura
ca

M´todos pr´ticos, mais flex´
e a ıveis:
m´todos gulosos (r´pidos, n´mero de testes nõ ´ otimizado)
e a u a e
busca heur´
ıstica (mais lentos, resultados de melhor qualidade)
M´todo para obter os melhores arranjos de cobertura
e
poss´ıvel:
selecione os melhores resultados, usando combina¸õ de:
ca
bons ingredientes (constru¸˜es diretas e buscas heur´
co ısticas)
+ e as melhores constru¸˜es combinat´rias recursivas
co o

Ver a tabela mantida por Colbourn com os melhores
tamanhos de arranjos de cobertura conhecidos.

o Lucia Moura

Introdu¸õ

Exemplo de bons ingredientes p/ usar em constru¸õ
ca
recursiva
arranjos ortogonais: CA(N = q 2 ; t = 2, k ≤ q + 1, v = q)
(m´todo cl´ssico usando corpos finitos Fq )
e a
0000
2 3
6 0122 7
1220
6 7
6 7
2202
6 7
6 7
2021
6 7
6 7
6
6 0211 7
7
6
6 2110 7
7
4 1101 5
1012

(N otimo)
metodo de LFSR para t = 3:
CA(N = 2q 3 − 1; t = 3, k ≤ q 2 + q + 1, v = q)
(Raaphorst, Moura, Stevens 2012)
(N ´timo ou perto do ´timo)
o o

o Lucia Moura

Introdu¸˜o

Exemplo de uma boa constru¸˜o recursiva: Produtos
ca
neste exemplo: parˆmetro t = 2
a

CA(4,3)
1 2 3 4
1 0 0 0 0 CA(12=4*3,3)
2 0 1 1 2
1 2 3 4 5 6 7 8 9 10 11 12
3 0 2 2 1
4 1 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0
5 1 1 0 1 0 1 1 2 0 1 1 2 0 1 1 2
6 1 2 1 0 0 2 2 1 0 2 2 1 0 2 2 1
7 2 0 1 1 1 0 2 2 1 0 2 2 1 0 2 2
8 2 1 2 0 1 1 0 1 1 1 0 1 1 1 0 1
9 2 2 0 2 1 2 1 0 1 2 1 0 1 2 1 0
2 0 1 1 2 0 1 1 2 0 1 1
size=9 2 1 2 0 2 1 2 0 2 1 2 0
2 2 0 2 2 2 0 2 2 2 0 2
CA(3,3) with 3 disjoint rows: 0 0 0 0 1 1 1 1 2 2 2 2
0 0 0 0 2 2 2 2 1 1 1 1
0 0 0 1 1 1 1 0 0 0 0 2 2 2 2
1 1 1 1 1 1 1 2 2 2 2 0 0 0 0
2 22 2 2 2 2 0 0 0 0 1 1 1 1
0 1 2 2 2 2 2 1 1 1 1 0 0 0 0
02 1 size=9+6=15
1 0 2
OD(3,3) 1 20
2 01
2 10
CA(N1,k1,g)+OD(N2,k2,g)= CA(N1+N2,k1*k2,g)
size=6

o Lucia Moura

Introdu¸õ

Generaliza¸oes de Arranjos de Cobertura
c˜

Queremos constru¸˜es eficazes baseadas em teoria combinat´ria
co o
para lidar com restri¸˜es adicionais em teste de software:
co
N´mero de valores variado (v varia com a coluna:
u
v1 , v2 , . . . , vk )
Configura¸˜es proibidas
co
ACs evitando pares proibidos (CAFEs) e generaliza¸˜es
co
Varia¸˜es na for¸a
co c
ACs de for¸a vari´vel.
c a
Capacidade de determina¸õ dos erros
ca
locating arrays, detecting arrays, error-locating arrays (ELAs)
Outras generaliza¸˜es
co

o Lucia Moura

Introdu¸˜o
For example, suppose we had an application that is intended to run on a variety of
platforms comprised of five components: an operating system (Windows XP, Apple OS X,
N´mero de valores variados
u Red Hat Enterprise Linux), a browser (Internet Explorer, Firefox), protocol stack (IPv4,
IPv6), a processor (Intel, AMD), and a database (MySQL, Sybase, Oracle), a total of
3 ⋅ 2 ⋅ 2 ⋅ 2 ⋅ 3 = 72 possible platforms. With only 10 tests, shown in Table 1, it is possible to
Mtest every component interacting= 5,every other 2, 2, 2, 3))least once, i.e., all possible
CA(N = 10; t = 2, k with v = (3, component at
pairs of platform components are covered.

5 OS X IE IPv4 Intel Sybase
6 OS X Firefox IPv4 Intel Oracle
7 RHEL IE IPv6 AMD MySQL
8 RHEL Firefox IPv4 Intel Sybase
9 RHEL Firefox IPv4 AMD Oracle

Table 1. PairwiseKackerconfigurations
(exemplo de Khun, test and Lei 2010)

Constru¸˜es combinat´rias:
co o
Moura, Stardom, Stevens, Williams (2003) 7
Colbourn, Martirosyan, Mullen, Shasha, Sherwood, Yucas (2005)
o Lucia Moura

Introdu¸˜o

N´mero de valores variados
u
M CA(N = 10; t = 2, k = 5, v = (3, 2, 2, 2, 3))

(exemplo de Khun, Kacker and Lei 2010)

Constru¸˜es combinat´rias:
co o
Moura, Stardom, Stevens, Williams (2003)
Colbourn, Martirosyan, Mullen, Shasha, Sherwood, Yucas (2005)
o Lucia Moura

the original configurations are underlined. Note that adding the constraint also resulted in
Introdu¸õ
ca
reducing the number of Constru¸õ de arranjos de coberturaThis will not always be Modelos generalizados
ca
test configurations by one. the case,
depending on the constraints used, but it illustrates how constraints can help reduce the
problem. Even if particular combinations are testable, the test team may consider some
Configura¸oes proibidas/CAFEs
c˜
combinations unnecessary, and constraints could be used to prevent these combinations,
Internet explorer (IE) so podenumber of test configurations. proibidos: (OS X, IE) (RHL,IE)
possibly reducing the ser usado com Windows (XP) Pares

5 OS X Firefox IPv4 Intel Sybase
7 RHL Firefox IPv6 Intel MySQL
8 RHL Firefox IPv4 Intel Oracle
9 XP IE IPv4 AMD Sybase
(exemplo de Khun, Kacker and Lei 2010)
Figure 8. Test configurations for simple example with constraint.

Modelo: grafo c/ 3+2+2+2+3 vertices, 2 arestas: {OS X, IE} {RHL,IE}
Constru¸˜es Factors orias:
3.3.2 Cost
co combinat´
• Danziger, Mendelsohn, Moura, Stevens (2008, is probably the most widely
Using combinatorial methods to design test configurations 2009)
• Maltais, Moura approach because it is quick and easy to do and typically delivers
used combinatorial (2010)
Aplica¸oes:improvements toattesting. costCombinatorial testing forand can parameters can
significant
c˜
provide better test coverage lower than conventional tests,
input
be extended to
• Cohen, Dwyer, Shi (2007, much better assurance.
high strength coverage to provide 2008)

3.4 Chapter Summary
o Lucia Moura

Introdu¸õ

1 1 1 0 0 0 1 0
For¸a vari´vel: AC em grafos
c a 1
1
1
0
0
1
1
1
0
1
1
0
0
0
0
0
1 0 0 0 0 1 1 1
Testes cobrindo rela¸˜es de entrada-saida
co (nota: a matriz est´ transposta)
a
Next, we transform into a test suite for ac-
R cording to also cover the same combination for
must step 5. . Since we
I1 chose and its combination arbitrarily, we have proven
B B
I6 I2
that does cover all combinations of .
1 1 1 0 0 0 1 0
Y G The next theorem describes the relationship0
1 1 0 1 0 1 0
between
I7 I8
at step 4 and 1 at step 5.1
1 0 0 0 1 0
I5 I3
R R Theorem 3.2 1 Suppose 0
1 1was 0 1 to0 0 us-
reduced
I4 ing steps 1 to1 of 1 algorithm. If
3 our 1 0 0 0and 1 are op- 0
B 1 1 0 1
timal test suites for and 0 1respectively then
0 0
. In addition, if then .
Figure 1. The graph for the problem instance where 1 0 1 1 1 0 0 0
and , ,Proof. According to Theorem 3.1, 0 5 of our algorithm
1 0 0 0 step 1 1 1
, , , , . transforms into a test suite for whose size is
(exemplo de Cheng, Dumitrescu, . Schroeder 2003)
Hence, . On the other hand, when
, can also be transformed into a test suite for
. Let us now replace the param- Theorem 3.1in theis a test suite for each . That is, covers
following manner. For , con-
with their corresponding parameters all combinations of .each
Colora¸õ de grafos, homomorfismos de grafos
ca
eters in each in struct by setting equal to its value in .
: , , That is, if we arrange the test cases in as columns in a
, , array, and whose rows are indexed by ,
Proof.then the test suite for and is formed by considering only
Let be a particular
, , and
• Meagher, tese de PhD (2005)
. Applying reduction R1, combination indexed .by the input variables in . , let each denote
the rows of For each Since us

• Meagher, Stevens (2005), Meagher, Moura, Zekaoui (2007) by
. The table below con- the color also in , this means that in rows Thus,
is assigned to node the . in
tains a test suite for . indexed by the input parameters in cover all the com-
. Since covers test then we have a test
suite there is
• Cheng, Dumitrescu, Schroeder (2003), Cheng (2007) binations of . But , so the
case created for that coverscoverscombination of , for
also the all combinations
1 1 1 0 0 0 1 0
of .each .
Consequently, the corresponding test case of
Teste Combinat´rio de Software 0 1 0 1 0 0
o 1 1
Lucia Moura

Introdu¸õ

For¸a vari´vel: AC em grafos
c a
Testes cobrindo rela¸˜es de entrada-saida
co (nota: a matriz est´ transposta)
a

(exemplo de Cheng, Dumitrescu, Schroeder 2003)

Colora¸õ de grafos, homomorfismos de grafos
ca

• Meagher, tese de PhD (2005)
• Meagher, Stevens (2005), Meagher, Moura, Zekaoui (2007)
• Cheng, Dumitrescu, Schroeder (2003), Cheng (2007)
o Lucia Moura

Introdu¸˜o

ACs de for¸a vari´vel
c a
VCAs = ACs em hipergrafos

Se escolhe n´
ıveis de cobertura diferentes para diferentes conjuntos
de parˆmetros.
a

• Cohen, Colbourn, Collofello, Gibbons, Mugridge (2003)
• Raaphorst, Moura and Stevens (2011, 2012)
o Lucia Moura

Introdu¸˜o

Como localizar as intera¸oes que causam falha?
c˜

Example:
0 0 0 0 F
0 1 1 0 F
0 2 2 2 P
1 0 2 1 P
1 1 0 2 P
1 2 1 0 P
2 0 1 2 P
2 1 2 0 P
2 2 0 1 P
0 0 1 1 P

V´rios poss´
a ıveis conjuntos de intera¸˜es errˆneas !
co o
Como construir um arranjo que nos indique as intera¸˜es que
co
causam erros?
o Lucia Moura

Introdu¸õ

Arranjos usados para localiza¸õ de erros
ca

Yilmaz, Cohen and Porter (2006):
arranjos de cobertura+ ´rvores de classifica¸õ.
a ca
Colbourn and McClary (2008):
locating arrays, detecting arrays.
Martinez, Moura, Stevens and Panario (2010):
error-locating arrays (ELAs)
Chodoriwsky, Moura (2012): generalizacao de algoritmo
adaptativo para t ≥ 3

o Lucia Moura

&(#%R3#$&.'4$%'6%>(."(%$'6&
A2'3/% &#$&.4+% $#,-."#% ),'-./#,$% ),'-./#% &(#% 9B% '6% &(#%
Introdu¸õ
ca "2'3/% &#$&.4+5% >(1% /'% "2'
!""#$$% &'% &(#% )*+#% ),'-./#/% 01%õ de "2'3/% &#$&.4+% -.,&3*2%7*"(.4#C%"2.#4&$%"'44#"&%&(#%-.,&3*2%7*"(.4#%&(,'3+(% &#$&.4+8% M(#,#% *,#% *2,#*/1%
Constru¸a
c
&(#% arranjos de cobertura K8%AYZAVO@9YZ%
Modelos generalizados
,#7'&#% /#$D&')% *4/% '&(#,% >*1$8% <3&% &(#% >*1% '6% 4'4=:;<%
$#,-."#$5% &(#% #4&#,),.$#% /'#$% &(#% 634"&.'4*2% &#$&.4+% *4/% *))2."*&.'4%$'6&>*,#5%$&.22%4##/$%&'%0#%.4$&*22#/%'4%&(#%-.,&3*2% 6'"3$%'4%&(.$%*"&.-.&18%Y3,%>
% % % M(.$% )*)#,% /#$",.0#/% '3,% ,#$#*,"(%6',%&(#%63&3,#%$'6&>*,#%&#$&.4
)#,6',7*4"#% &#$&.4+8% 9&% .$% #*$1% &'% *"(.#-#% 6',% &(#% 7*"(.4#% *4/% &(#4% .&% "*4% 0#% &#$&#/8% E.*+,*7% .$% *$% 6'22'>%
.4% &(#% *,#*% '6% "2'
Outras generaliza¸˜es: aplica¸õ em cloud computing
co ca
:;<=0*$#/%*))2."*&.'4%$'6&>*,#%),'/3"&$%'6%<?@%$&,3"&3,#8% &#$&.4+8% :#% .4&,'/3"#/% &(#% /#6.4.&.'4% '6% "2'3/% "'7)3&
FG.+8HIJ%

*4/% &(#% /#6.4.&.'4% '6% "2'3/% &#$&.4+8% !4/% &(#45% >#% *4*21L
!A`ZY:
:#%&(*4D%'3,%"'22#*+3
!"# # $%&'(#)&*+,-.#/-01&'2#3201'4.4(56# &(#%$&3/15%*4/%6',%&(#.,%"'7
&(#%R3#$&.'4$%'6%>(."(%$'6&>*,#%&#$&.4+%),']#"&$%"*4%/'%
A2'3/% &#$&.4+% $#,-."#% ),'-./#,$% ),'-./#% &(#% 9B% '6% &(#% X;G;
"2'3/% &#$&.4+5% >(1% /'% "2'3/$% &#$&.4+5% E8% Z3,7.5% &'%:'2$D.5% A8% [,
('>% X8% /'% "2'
Jun &Meng (“Software Testing based on cloud computing” , 2011)
-.,&3*2%7*"(.4#C%"2.#4&$%"'44#"&%&(#%-.,&3*2%7*"(.4#%&(,'3+(% aNb!
&#$&.4+8% M(#,#% *,#% *2,#*/1% $#-#,*2% ),'/3"&$% *-*.2*02#%.4%& c'3$#665% *4/% E8% d*+',
,#7'&#% /#$D&')% *4/% '&(#,% >*1$8% <3&% &(#% >*1% '6% 4'4=:;<% "2'3/="'7)3&.4+% $1$&#75f%

discutem o modelo de usar cloud computing para teste de 6'"3$%'4%&(.$%*"&.-.&18%Y3,%>',D%.$%*.7#/%*&%/'.4+%*%,#$#* i&(% 9;;;?!A% 94&#,4*&.'4*2%
*))2."*&.'4%$'6&>*,#5%$&.22%4##/$%&'%0#%.4$&*22#/%'4%&(#%-.,&3*2% &(#% [,./8% :*$(.4+&'45% EA5% O

7*"(.4#% *4/% &(#4% .&% "*4% 0#% &#$&#/8% E.*+,*7% .$% *$% 6'22'>% 6',%&(#%63&3,#%$'6&>*,#%&#$&.4+%7#&('/$8% NQSjNHN8% #2*$&."% "'7)3&#%
aQb! !7*L'4%
software.
FG.+8HIJ% %
!A`ZY:V;E[;ZM% G.+8H%9**@%)2*&6',7%*,"(.&#"&3,#% % K.,&3*2.L*&.'4%&#"(4'2'+1%
!-*.2*02#J(&&)J??*>$8*7*L'48"
aHb! !8% :#.$$5% eA'7)3&.4+% .4%
M(#%7*.4%),'"#$$%'6%"2'3/%&#$&.4+%*$%6'22'>$J% ))8NUjQT5%Qhhk8%
aSb! M88`.4+5% e!% $#26=&#$&.4+% *)
:#%&(*4D%'3,%"'22#*+3#$%6',%&(#.,%&#"(4."*2%(#2)%/3,
NI% O$#,$% 2'+.4% "2'3/% &#$&.4+% ),'-./#,P$% >#0$.&#% *4/%
,#+.$&#,%3$#,%.46',7*&.'4C%
/.$$#,&*&.'45%G2',./*%94&#,4*&.'
Qhhi8%
&(#%$&3/15%*4/%6',%&(#.,%"'77#4&$%*4/%"'4&.43'3$%$3))',&
QI% O$#,$% *))21% 6',% &(#% &#$&% )2*&6',7% ,#$'3,"#$5% aTb! ZE@O%E#)*,&7#4&%'6%A'7)3
[,'3)5f%l*48%QhNh5%(&&)J??"$84/
*))2."*&.'4$%4##/%&'%/#$",.0#%&(#%"'46.+3,*&.'4%,#R3.,#7#4&$%
aUb! MAVJ%(&&)J??&"28$'3,"#6',+#84#&
X;G;X;ZA;@% akb! l8% K*,.*8% FQhhi5% G#0,3*,1% Q
'6% -.,&3*2% 7*"(.4#% #4-.,'47#4&5% @3"(% *$% ')#,*&.4+% $1$&#7%
-#,$.'45%0,'>$#,%-#,$.'45%7#7',1%$.L#5%(*,/%/.$D%$.L#5%(*,/% A2'3/8f% !7*L'4
/.$D%$)##/5%4#&>',D%0*4/>./&(5%6.,#>*225%#&"8C% <2'+8(&&)J??*>$8&1)#)*/8"'7?*
aNb!M(#% $#,-."#% ),'-./#,% '6% "2'3/% &#$&.4+% ,#-.#>% &(#% [8% Y0#,&#22.5% @8% @'7*4
HI%
E8% Z3,7.5% X8% :'2$D.5% A8% [,L#+',"L1D5% &(#="2'3/8(&72%
c'3$#665% *4/% E8% d*+','/4'-5eM(#% #3"*21)&3$% ')#4=$'3
*))2."*&.'4%*4/%"'46.+3,#%&(#%&#$&%)2*&6',7C%
SI% O4/#,% &(#% ,#4&*2% *+,##7#4&5% &(#% 3$#,% F'42.4#I%ghiJ% B,'"##/.4+$% '6% &(#% Q
"2'3/="'7)3&.4+% $1$&#75f% .4% AA[X9E%
),#=)*./%$#,-."#%6##C% %
i&(% 9;;;?!A% 94&#,4*&.'4*2% @17)'$.37% '4% A23$&#,% A'7)3&.4+%
TI%O$#,$%2'+.4%&(#%"2'3/%&#$&.4+%)2*&6',75%34&.2%&(#%#4/%
&(#% [,./8% :*$(.4+&'45% EA5% O@!J% 9;;;% A'7)3&#,% @'".#&15% Qhhi5
'6%&(#%&#$&.4+C%
NQSjNHN8%
UI%!""',/.4+%&'%&(#%62'>%',%&.7#%&'%)*1%&(#%*"&3*2%"'$&$5%
aQb! !7*L'4% #2*$&."% "'7)3&#% "2'3/% F!7*L'4% ;AQI8% aY42.
&(#%2#*$#%#4/$8%
% !-*.2*02#J(&&)J??*>$8*7*L'48"'7?#"Q?8%
9K8%<O9VE%MW;%BX9K!M;%AVYOE%M;@M9Z[%

Uma idía que proponho nesta palestra:aHb! !8% :#.$$5% eA'7)3&.4+% .4% &(#% A2'3/$5f% 4#&:',D#,5% -'28NN54
e
G.+8H%9**@%)2*&6',7%*,"(.&#"&3,#% % K.,&3*2.L*&.'4%&#"(4'2'+1%
M(#%7*.4%),'"#$$%'6%"2'3/%&#$&.4+%*$%6'22'>$J% ))8NUjQT5%Qhhk8%
BV!MGYX%YG%;ZM;XBX9@;%
X#6#,#4"#% "2'3/% &#$&.4+% ./#*$5% #4&#,),.$#$% "*4% 03.2/%
&(#.,% '>4% .4&#,4*2% "2'3/% &#$&.4+% #4-.,'47#4&% &'% #4]'1%6',% *3&'4'7."% $'6&>*,#5f% B(
aSb! M88`.4+5% e!% $#26=&#$&.4+% *)),'*"(% .&$%
NI% O$#,$% 2'+.4% "2'3/% &#$&.4+% ),'-./#,P$% >#0$.&#% *4/%
usu´rio usa arranjos de cobertura no passo 2 (pode ser invi´vel); ou
a
,#+.$&#,%3$#,%.46',7*&.'4C%
a
*/-*4&*+#$8% A'46./#4&.*2.&1% *4/% '&(#,% ,#*$'4$% 6',% &(#%
/.$$#,&*&.'45%G2',./*%94&#,4*&.'4*2%O4.-#,$.&15%.*7.5%GV5%O@!5%!3
#^&#,4*2% 4#&>',D% "*4% 4'&% *""#$$% &(#% $'6&>*,#% X_E%
Qhhi8%
#4&#,),.$#%"2'3/%&#$&%7#&('/%"*4%*2$'%"2."D%(#,#8%
QI% O$#,$% *))21% 6',% &(#% &#$&% )2*&6',7% ,#$'3,"#$5% aTb! ZE@O%E#)*,&7#4&%'6%A'7)3&#,%@".#4"#5%e@'6&>*,#%M#$&.4+%X#$#*
!3&'7*&#/%"2'3/%&#$&.4+%#4-.,'47#4&J%"(''$#%*%0#&&#,%
sofisticar o passo 2: o usu´rio s´ fornece as componentes e
a o
*))2."*&.'4$%4##/%&'%/#$",.0#%&(#%"'46.+3,*&.'4%,#R3.,#7#4&$% [,'3)5f%l*48%QhNh5%(&&)J??"$84/$38#/3?$&,+?8%
)#,6',7*4"#%7*"(.4#%',%$#,-#,%&'%.4$&*22%*3&'7*&#/%&#$&.4+%
$'6&>*,#5%MAVJ%(&&)J??&"28$'3,"#6',+#84#&?%V*$&%*""#$$#/J%*,"(%Nh5%QhNh8%
aUb! 6',% #^*7)2#% V'*/,34#,5% X'0'&% *4/% '&(#,% &''2$8%
'6% -.,&3*2% 7*"(.4#% #4-.,'47#4&5% @3"(% *$% ')#,*&.4+% $1$&#7% e a for¸a desejada; o cloud
configura¸˜es desejadas para o teste
co c M#$&.4+% +,'3)$% 2'+% .4% &(.$% 7*"(.4#% &'% /'% &(#.,% '>4%
akb! l8% K*,.*8% FQhhi5% G#0,3*,1% QSI8% eK.,&3*2% @&,#$$=6,##% M#$&.4+% .4%
634"&.'4*2%&#$&.4+%*4/%*3&'7*&#/%&#$&.4+5%*4/%/'%4'&%4##/%&'%
-#,$.'45%0,'>$#,%-#,$.'45%7#7',1%$.L#5%(*,/%/.$D%$.L#5%(*,/% .4$&*22%&(#%.4/#)#4/#4&%*3&'7*&#/%&#$&.4+%#4-.,'47#4&%.4%&(#% :#0%
A2'3/8f% !7*L'4% @#,-
testing provider gera o arranjo de cobertura adequado
/.$D%$)##/5%4#&>',D%0*4/>./&(5%6.,#>*225%#&"8C% <2'+8(&&)J??*>$8&1)#)*/8"'7?*>$?Qhhi?hQ?-.,&3*2=$&,#$$6,##=&#$&.4+
2'"*2%7*"(.4#$8%
32&.=7*"(.4#% #4-.,'47#4&% &'% 03.2/J% 3$.4+% -.,&3*2%
&(#="2'3/8(&72%
HI% M(#% $#,-."#% ),'-./#,% '6%cõ: &#$&.4+% ,#-.#>% &(#% das maquinas virtuais na cloud)
(restri¸"2'3/% configura¸˜es
a co 7*"(.4#% &#"(4'2'+1% &'% "')1% *4/% $&*,&% *% 4370#,% '6% -.,&3*2%
*))2."*&.'4%*4/%"'46.+3,#%&(#%&#$&%)2*&6',7C% 7*"(.4#$%'4%&(#%$#,-#,%R3."D215%&(#%&#$&%&#,7.4*2$%*""#$$%&'%
&(#.,% '>4% -.,&3*2% 7*"(.4#$% &'% 3)+,*/#% "3$&'7#,% /*&*5%
SI%o O4/#,% &(#% ,#4&*2% *+,##7#4&5% &(#% 3$#,% F'42.4#I%
Teste Combinat´rio de Software #^#"3&#%&(#%#4-.,'47#4&%&#$&.4+5%$&,#$$%&#$&.4+5%)#,6',7*4"#%
Lucia Moura
&#$&.4+5% *4/% '&(#,% $'6&>*,#% &#$&.4+% 01% ,#7'&#% /#$D&')% ',%

Introdu¸õ

Discussõ final
a

Teste combinat´rio de software ´ util e efetivo.
o e
Existem tools dispon´
ıveis para uso imediato em aplica¸˜es:
co
• ACTS do NIST (EUA) t ≤ 6 (open source, gratuito)
• Hexawise: comercial t ≤ 6 (gratuito para uso acadˆmico,
e
nonprofit e compania com at´ 5 usu´rios, ≥$50.000/ano)
e a
• Testcover.com: gerador autom´tico (t = 2) na
a
cloud/software as a service (SaaS); (assinatura: $100/mˆs)
e
Existe pesquisa ativa na ´rea de algor´
a ıtmos e constru¸˜es
co
combinat´rias para otimizar o n´mero de testes em arranjos
o u
de cobertura.
Temos tido algum sucesso em lidar com restri¸˜es adicionais.
co
Existe pesquisa ativa na ´rea de teste de software avaliando a
a
efetividade e adaptando teste combinat´rio de software aos
o
mais diversos tipos de aplica¸˜es.
co
o Lucia Moura

Teste combinatório de software

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Teste combinatório de software

Similar to Teste combinatório de software (20)

More from PET Computação

More from PET Computação (20)

Recently uploaded

Recently uploaded (20)

Teste combinatório de software