Boundary Value Testing for Traditional and ML Software

1. felix.dobslaw@miun.se Boundary Value Testing for Traditional and ML Software At Chalmers - WASP, Software Engineering, PhD course - 8/6/2023 Felix Dobslaw Östersund, Mid Sweden University Research lead, Software Engineering and Education Group SEE

2. felix.dobslaw@miun.se Felix ● 2007 Bsc in “Informatik” University of Hamburg ● 2009 Msc in Distributed Systems MIUN (Sundsvall) ● 2005 – 2009 Prolifics/IBM software dev (Hamburg/Remote) ● 2012 Lic. in Computational Intelligence, MIUN ● 2015 Dr. in Wireless Sensor Networks, MIUN ● 2016 → Tenure in Computer Sciences, MIUN (Östersund) ● 2016 → Entrepeneurship, ride-sharing, online ratings, ... ● 2019-2021 PostDoc in Software Engineering, Chalmers (Gothenburg) ● 2022-now research-lead in Software Engineering at MIUN (Östersund)

5. felix.dobslaw@miun.se

7. felix.dobslaw@miun.se Test Automation Augmentation

8. felix.dobslaw@miun.se inputs outputs requirements constraints

9. felix.dobslaw@miun.se Specification: inputs outputs requirements constraints Entirely implicit Greatly detailed

10. felix.dobslaw@miun.se Specification: inputs outputs requirements constraints Entirely implicit Greatly detailed Waterfall Model

11. felix.dobslaw@miun.se program Specification: tests inputs outputs requirements constraints Entirely implicit Greatly detailed Waterfall Model

12. felix.dobslaw@miun.se program Specification: tests validate Specification inputs outputs requirements constraints Entirely implicit Greatly detailed Waterfall Model

15. felix.dobslaw@miun.se program Specification: tests validate Specification inputs outputs requirements constraints Entirely implicit Greatly detailed Waterfall Model Goal vs. System

18. felix.dobslaw@miun.se program Specification: tests validate Specification inputs outputs requirements constraints Entirely implicit Greatly detailed Waterfall Model Goal vs. System Correct software depends on correct boundaries. Boundary Value Exploration reveals discrepancies between desired and actual behavior.

19. felix.dobslaw@miun.se F. Dobslaw, F. G. de Oliveira Neto and R. Feldt, "Boundary Value Exploration for Software Analysis," 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2020.

20. felix.dobslaw@miun.se Boundary Value Testing F. Dobslaw, F. G. de Oliveira Neto and R. Feldt, "Boundary Value Exploration for Software Analysis," 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2020.

21. felix.dobslaw@miun.se Boundary Value Testing BVT: “Execution of specific input pairs in order to ensure that an actual boundary is also expected.” F. Dobslaw, F. G. de Oliveira Neto and R. Feldt, "Boundary Value Exploration for Software Analysis," 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2020.

22. felix.dobslaw@miun.se Boundary Value Analysis Boundary Value Testing BVT: “Execution of specific input pairs in order to ensure that an actual boundary is also expected.” F. Dobslaw, F. G. de Oliveira Neto and R. Feldt, "Boundary Value Exploration for Software Analysis," 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2020.

23. felix.dobslaw@miun.se BVA: “Analysis of artifacts of the software development process to clarify the expected and actual boundaries of a software.” Boundary Value Analysis Boundary Value Testing BVT: “Execution of specific input pairs in order to ensure that an actual boundary is also expected.” F. Dobslaw, F. G. de Oliveira Neto and R. Feldt, "Boundary Value Exploration for Software Analysis," 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2020.

24. felix.dobslaw@miun.se Boundary Value Analysis Boundary Value Testing Boundary Value Exploration Candidates F. Dobslaw, F. G. de Oliveira Neto and R. Feldt, "Boundary Value Exploration for Software Analysis," 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2020.

25. felix.dobslaw@miun.se Boundary Value Analysis Boundary Value Testing Boundary Value Exploration Candidates F. Dobslaw, F. G. de Oliveira Neto and R. Feldt, "Boundary Value Exploration for Software Analysis," 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2020.

26. felix.dobslaw@miun.se Boundary Value Analysis Boundary Value Testing Boundary Value Exploration Candidates BVE: “A collection of techniques that select or help select inputs to detect and identify boundary candidates.” F. Dobslaw, F. G. de Oliveira Neto and R. Feldt, "Boundary Value Exploration for Software Analysis," 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2020.

27. felix.dobslaw@miun.se Boundary Value Analysis Boundary Value Testing Boundary Value Exploration Candidates BVE: “A collection of techniques that select or help select inputs to detect and identify boundary candidates.” F. Dobslaw, F. G. de Oliveira Neto and R. Feldt, "Boundary Value Exploration for Software Analysis," 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2020.

28. felix.dobslaw@miun.se Boundary Value Testing (BVT) Software Testing  Related to Partition Analysis  Decide and implement relevant test cases on the boundaries 3/13

32. felix.dobslaw@miun.se What are the boundaries? What are the boundary candidates?

33. felix.dobslaw@miun.se What are the boundaries? What are the boundary candidates? height, weight → Body Mass Index Category height in cm weight in kg method: bmi(h::int, w::int)

34. felix.dobslaw@miun.se What are the boundaries? What are the boundary candidates? height, weight → Body Mass Index Category height in cm weight in kg method: bmi(h::int, w::int) Examples Expected Boundary: Healthy/Overweight Boundary Candidate: [ (142.2, 81), (142.2, 82) ]

35. felix.dobslaw@miun.se What are the boundaries? What are the boundary candidates? height, weight → Body Mass Index Category height in cm weight in kg method: bmi(h::int, w::int) Examples Expected Boundary: Healthy/Overweight Boundary Candidate: [ (142.2, 81), (142.2, 82) ]

36. felix.dobslaw@miun.se program Specification: tests inputs outputs requirements constraints BVE Process Example

37. felix.dobslaw@miun.se program Specification: tests inputs outputs requirements constraints

38. felix.dobslaw@miun.se program Specification: tests inputs outputs requirements constraints boundary mining

39. felix.dobslaw@miun.se program Specification: tests inputs outputs requirements constraints boundary mining bmi(1,0) → “Underweight” bmi(1,1) → “Sev. Obese”

40. felix.dobslaw@miun.se program Specification: tests inputs outputs requirements constraints boundary mining bmi(142.2, 81) → “Normal” bmi(142.2, 82) → “Overweight” “clear case” bmi(1,0) → “Underweight” bmi(1,1) → “Sev. Obese”

41. felix.dobslaw@miun.se program Specification: tests inputs outputs requirements constraints boundary mining Bmi(139, 100) → BoundsError(“ w >= 140”) Bmi(140, 100) → “Sev. Obese“ bmi(142.2, 81) → “Normal” bmi(142.2, 82) → “Overweight” “clear case” not so clear case... bmi(1,0) → “Underweight” bmi(1,1) → “Sev. Obese”

42. felix.dobslaw@miun.se program Specification: tests inputs outputs requirements constraints boundary mining Bmi(139, 100) → BoundsError(“ w >= 140”) Bmi(140, 100) → “Sev. Obese“ bmi(142.2, 81) → “Normal” bmi(142.2, 82) → “Overweight” “clear case” not so clear case... bmi(1,0) → “Underweight” bmi(1,1) → “Sev. Obese” Specification Pull Request:

43. felix.dobslaw@miun.se program Specification: tests inputs outputs requirements constraints boundary mining Bmi(139, 100) → BoundsError(“ w >= 140”) Bmi(140, 100) → “Sev. Obese“ Standards simplify Tooling (e.g. web): bmi(142.2, 81) → “Normal” bmi(142.2, 82) → “Overweight” “clear case” not so clear case... bmi(1,0) → “Underweight” bmi(1,1) → “Sev. Obese” Specification Pull Request:

44. felix.dobslaw@miun.se Specification Pull Request Correction: bmi(141, 100) → BoundsError(“ w >= 142”) bmi(143, 100) → “Sev. Obese“ program Specification: tests inputs outputs requirements constraints boundary mining Bmi(139, 100) → BoundsError(“ w >= 140”) Bmi(140, 100) → “Sev. Obese“ Standards simplify Tooling (e.g. web): bmi(142.2, 81) → “Normal” bmi(142.2, 82) → “Overweight” “clear case” testing → quality assurance not so clear case... bmi(1,0) → “Underweight” bmi(1,1) → “Sev. Obese” Specification Pull Request:

45. felix.dobslaw@miun.se Specification Pull Request Correction: bmi(141, 100) → BoundsError(“ w >= 142”) bmi(143, 100) → “Sev. Obese“ program Specification: tests inputs outputs requirements constraints boundary implementation boundary mining Bmi(139, 100) → BoundsError(“ w >= 140”) Bmi(140, 100) → “Sev. Obese“ Standards simplify Tooling (e.g. web): bmi(142.2, 81) → “Normal” bmi(142.2, 82) → “Overweight” “clear case” testing → quality assurance not so clear case... bmi(1,0) → “Underweight” bmi(1,1) → “Sev. Obese” Specification Pull Request:

46. felix.dobslaw@miun.se Specification Pull Request Correction: bmi(141, 100) → BoundsError(“ w >= 142”) bmi(143, 100) → “Sev. Obese“ program Specification: tests inputs outputs requirements constraints boundary implementation boundary mining Bmi(139, 100) → BoundsError(“ w >= 140”) Bmi(140, 100) → “Sev. Obese“ Standards simplify Tooling (e.g. web): bmi(142.2, 81) → “Normal” bmi(142.2, 82) → “Overweight” “clear case” testing → quality assurance not so clear case... bmi(1,0) → “Underweight” bmi(1,1) → “Sev. Obese” Specification Pull Request:

47. felix.dobslaw@miun.se How to automatically mine boundaries?

48. felix.dobslaw@miun.se SUT x y How to automatically mine boundaries?

49. felix.dobslaw@miun.se SUT x y Assert y == yexp How to automatically mine boundaries?

50. felix.dobslaw@miun.se SUT x y Assert y == yexp How to automatically mine boundaries?

51. felix.dobslaw@miun.se SUT x y x1 x2 y1 y2 SUT SUT Assert y == yexp How to automatically mine boundaries?

52. felix.dobslaw@miun.se SUT x y x1 x2 y1 y2 ? SUT SUT Assert y == yexp How to automatically mine boundaries?

53. felix.dobslaw@miun.se SUT x y x1 x2 y1 y2 ? SUT SUT Assert y == yexp How to automatically mine boundaries? Distance Metric Examples: Jaccard, Hamming, Euclidean… Very basic one: Strlendist

54. felix.dobslaw@miun.se SUT x y x1 x2 y1 y2 ? SUT SUT Assert y == yexp How to automatically mine boundaries? Distance Metric Examples: Jaccard, Hamming, Euclidean… Very basic one: Strlendist Example: 30 and 31 are neighbors in the input space for field month: Date(2021, 30, 4) → “30/4/2021” StringLength is 9 Date(2021, 31, 4) → “ERROR, month field out of bounds.” StringLength is 33 Strlendist = |33-9| = 24 Big difference, this seems to be a boundary candidate! But is it an interesting one?...

56. felix.dobslaw@miun.se Boundary without Oracle

57. felix.dobslaw@miun.se Boundary without Oracle

58. felix.dobslaw@miun.se Boundary without Oracle $ sudo -u# -1 id -u

59. felix.dobslaw@miun.se Foundation: Diversity  Challenge: Describe relation between inputs and outputs for arbitrary data types. ? x y

60. felix.dobslaw@miun.se Foundation: Diversity  Challenge: Describe relation between inputs and outputs for arbitrary data types. x1 x2 SUT y1 y2

61. felix.dobslaw@miun.se Foundation: Diversity  Challenge: Describe relation between inputs and outputs for arbitrary data types. x1 x2 SUT y1 y2 ?

62. felix.dobslaw@miun.se Foundation: Diversity  Challenge: Describe relation between inputs and outputs for arbitrary data types.  Proposal based on Kolmogorov Complexity  Applicable for all data types  Normalized Information Distance (NID)  “Compression trick” makes it practical  CC ~ KC  Normalized Compression Distance (NCD) x1 x2 SUT y1 y2 ?

66. felix.dobslaw@miun.se Program Derivatives

67. felix.dobslaw@miun.se Derivatives

68. felix.dobslaw@miun.se Derivatives Principia Mathematica (1687)

69. felix.dobslaw@miun.se Derivatives ● Slope of the tangent line to the graph at x How sensitive to change is f in x?

70. felix.dobslaw@miun.se Derivatives ● Slope of the tangent line to the graph at x How sensitive to change is f in x? secant

71. felix.dobslaw@miun.se Derivatives ● Slope of the tangent line to the graph at x How sensitive to change is f in x? secant

72. felix.dobslaw@miun.se What is x + h? … What is h?

73. felix.dobslaw@miun.se What is x + h? … What is h? DQ = Difference Quotient

74. felix.dobslaw@miun.se Program Derivatives R. Feldt and F. Dobslaw. "Towards automated boundary value testing with program derivatives and search. International Symposium on Search Based Software Engineering. Springer, Cham, 2019.

80. felix.dobslaw@miun.se Program Derivatives P(x) is program output for input x R. Feldt and F. Dobslaw. "Towards automated boundary value testing with program derivatives and search. International Symposium on Search Based Software Engineering. Springer, Cham, 2019.

83. felix.dobslaw@miun.se Program Derivatives Hm… But … R. Feldt and F. Dobslaw. "Towards automated boundary value testing with program derivatives and search. International Symposium on Search Based Software Engineering. Springer, Cham, 2019.

84. felix.dobslaw@miun.se Program Derivatives Hm… But … … How do I select di and do? R. Feldt and F. Dobslaw. "Towards automated boundary value testing with program derivatives and search. International Symposium on Search Based Software Engineering. Springer, Cham, 2019.

85. felix.dobslaw@miun.se Program Derivatives Hm… But … … How do I select di and do? … How do I get bmin then? R. Feldt and F. Dobslaw. "Towards automated boundary value testing with program derivatives and search. International Symposium on Search Based Software Engineering. Springer, Cham, 2019.

86. felix.dobslaw@miun.se Program Derivatives Hm… But … … How do I select di and do? … How do I get bmin then? ... R. Feldt and F. Dobslaw. "Towards automated boundary value testing with program derivatives and search. International Symposium on Search Based Software Engineering. Springer, Cham, 2019.

87. felix.dobslaw@miun.se C is the compressor (e.g. bzip2) CDQ = Compression Difference Quotient PDQ = Program Difference Quotient NCD = Normalized Compression Distance Program Derivatives 9/13 How do I select di and do?

90. felix.dobslaw@miun.se Input Mutation How do I get bmin then?  May require exploring the SUT’s behavior on a number of “close values”  e.g. via Search-based Software Engineering with Mutators

91. felix.dobslaw@miun.se Case: (Julia) Date

102. felix.dobslaw@miun.se Finding Candidates

103. felix.dobslaw@miun.se Finding Candidates derivative

113. felix.dobslaw@miun.se Visible Boundary Candidate 2D - Color signifies dimension of boundary

115. felix.dobslaw@miun.se Visible Boundary Candidate 3D - Opacity signifies boundariness - “…” abbreviates single (long) year

117. felix.dobslaw@miun.se 3D zoom out - Delimited to 4 years from boundary

118. felix.dobslaw@miun.se 3D zoom out - Delimited to 4 years from boundary (manually decided)

121. felix.dobslaw@miun.se ● 2 proposed detection algorithms ● Program Derivative with crude metric (Strlendist) ● 4 SUTs quant + qual analysis (+200 quant only SUTs in revision) – Limited to unit tests with integer based inputs

122. felix.dobslaw@miun.se Preliminary Results/Conclusions

123. felix.dobslaw@miun.se Preliminary Results/Conclusions ● AutoBVA detected and selected interesting boundary candidates (potential bugs) for all 4 SUTs. ● First automated BVA in literature – Black-box, non-formal ● Identified demand for a more scalable solution with aggressive online selection (ongoing).

124. felix.dobslaw@miun.se Where else can this be useful? When does the system hit the switch? - is that at the right time, i.e. according to specification? - shall be a problem of the product owner, not the developer/data scientist. Research Council Project AQUAS 2022-2025

127. felix.dobslaw@miun.se Learnt Component Programmed Component Software Artifact Data Science Traditional Software Engineering SE/CS vs DS/ML Dobslaw, Felix, and Robert Feldt. "Similarities of Testing Programmed and Learnt Software." 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2023.

128. felix.dobslaw@miun.se Learnt Component Programmed Component Software Artifact inputs → → outputs Data Science Traditional Software Engineering SE/CS vs DS/ML Dobslaw, Felix, and Robert Feldt. "Similarities of Testing Programmed and Learnt Software." 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2023.

129. felix.dobslaw@miun.se Learnt Component Programmed Component Software Artifact inputs → → outputs Data Science Traditional Software Engineering Approach SE/CS vs DS/ML Dobslaw, Felix, and Robert Feldt. "Similarities of Testing Programmed and Learnt Software." 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2023.

130. felix.dobslaw@miun.se Learnt Component Programmed Component Software Artifact inputs → → outputs Data Science Traditional Software Engineering Approach SE/CS vs DS/ML Dobslaw, Felix, and Robert Feldt. "Similarities of Testing Programmed and Learnt Software." 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2023.

131. felix.dobslaw@miun.se Learnt Component Programmed Component Software Artifact inputs → → outputs Data Science Traditional Software Engineering Approach SE/CS vs DS/ML (+ ) Dobslaw, Felix, and Robert Feldt. "Similarities of Testing Programmed and Learnt Software." 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2023.

132. felix.dobslaw@miun.se Learnt Component Programmed Component Software Artifact inputs → → outputs Configuration Code Training Data Training Data Training Data Data Science Traditional Software Engineering Approach SE/CS vs DS/ML (+ ) Dobslaw, Felix, and Robert Feldt. "Similarities of Testing Programmed and Learnt Software." 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2023.

135. felix.dobslaw@miun.se “...the notion of correctness is not only a binary notion...” "...the notion of correctness is not clear..." Yves Le Traon (ICST Keynote 2023)

136. felix.dobslaw@miun.se Incorrect ? Unknown/ Unconsidered/ Unsure Correct Software Correctness Spectrum “...the notion of correctness is not only a binary notion...” "...the notion of correctness is not clear..." Yves Le Traon (ICST Keynote 2023)

137. felix.dobslaw@miun.se Incorrect ? Unknown/ Unconsidered/ Unsure Correct Software Correctness Spectrum “...the notion of correctness is not only a binary notion...” "...the notion of correctness is not clear..." Yves Le Traon (ICST Keynote 2023) The U’s call for common testing method to clarify and refine understanding.

138. felix.dobslaw@miun.se Incorrect ? Unknown/ Unconsidered/ Unsure Correct Software Correctness Spectrum “...the notion of correctness is not only a binary notion...” "...the notion of correctness is not clear..." Yves Le Traon (ICST Keynote 2023) The U’s call for common testing method to clarify and refine understanding. One idea here is:

139. felix.dobslaw@miun.se Incorrect ? Unknown/ Unconsidered/ Unsure Correct Software Correctness Spectrum “...the notion of correctness is not only a binary notion...” "...the notion of correctness is not clear..." Yves Le Traon (ICST Keynote 2023) The U’s call for common testing method to clarify and refine understanding. One idea here is: - automated black-box boundary analysis

140. felix.dobslaw@miun.se Loan Approval?

142. felix.dobslaw@miun.se Loan Approval? Question: What can the person do to get an approval?

143. felix.dobslaw@miun.se Loan Approval? Counterfactuals Question: What can the person do to get an approval?

148. felix.dobslaw@miun.se Loan Approval? Question: Are we making the right decisions?

149. felix.dobslaw@miun.se Loan Approval? Question: Are we making the right decisions? Is the boundary appropriate?

153. felix.dobslaw@miun.se Loan Approval? Question: Are we making the right decisions? Is the boundary appropriate? Is this a reasonable separator?

154. felix.dobslaw@miun.se Loan Approval? Question: Are we making the right decisions? Is the boundary appropriate? Is this a reasonable separator?

155. felix.dobslaw@miun.se Ongoing/Future Work Program Derivative (PD)

156. felix.dobslaw@miun.se Ongoing/Future Work Program Derivative (PD) Generalization of PD: may extend to - execution properties - memory use - exec time… - multiple/varying distance metrics - generic (ncd, jaccard...) - type specific (euclidean, geographic)

157. felix.dobslaw@miun.se Ongoing/Future Work Program Derivative (PD) Summarization: - Scalable, generalizable ways of selecting interesting candidates. - in support of production data Generalization of PD: may extend to - execution properties - memory use - exec time… - multiple/varying distance metrics - generic (ncd, jaccard...) - type specific (euclidean, geographic)

158. felix.dobslaw@miun.se Ongoing/Future Work GUI handling: - traces and uncommon/unexpected transitions Program Derivative (PD) Summarization: - Scalable, generalizable ways of selecting interesting candidates. - in support of production data Generalization of PD: may extend to - execution properties - memory use - exec time… - multiple/varying distance metrics - generic (ncd, jaccard...) - type specific (euclidean, geographic)

159. felix.dobslaw@miun.se Ongoing/Future Work GUI handling: - traces and uncommon/unexpected transitions Program Derivative (PD) How to support the tester: - process what steps, roles? - tools & method algorithms for detection/summarization features for automated test case extraction Summarization: - Scalable, generalizable ways of selecting interesting candidates. - in support of production data Generalization of PD: may extend to - execution properties - memory use - exec time… - multiple/varying distance metrics - generic (ncd, jaccard...) - type specific (euclidean, geographic)

160. felix.dobslaw@miun.se Questions/Comments?

161. felix.dobslaw@miun.se Questions/Comments? ● Find me on LinkedIn and connect. ● Chalmers and MIUN PhD positions out. – Both deadline 15/6 ● Related problems/questions? – Get in touch and we talk

Boundary Value Testing for Traditional and ML Software

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Boundary Value Testing for Traditional and ML Software

Semelhante a Boundary Value Testing for Traditional and ML Software (20)

Mais de Felix Dobslaw

Mais de Felix Dobslaw (7)

Último

Último (20)

Boundary Value Testing for Traditional and ML Software