Thinking in Properties

1. Thinking in Properties And Beyond (Testing) Susan Potter 2020-08-01

2. Intro

3. Intro finger $(whoami) • Introduced to QuickCheck in Erlang ~2010 • Adopted Haskell’s QuickCheck, Hedgehog, and ScalaCheck at work • ”Testing” in production, thinking in properties, 4 years Susan Potter Thinking in Properties 2020-08-01 1 / 41

4. Intro Agenda • An Origin Story (with code) • Mental Models (above the code) • Beyond Testing (illustrations) Susan Potter Thinking in Properties 2020-08-01 2 / 41

5. An Origin Story

6. An Origin Story Discovering Superpowers Susan Potter Thinking in Properties 2020-08-01 3 / 41

7. An Origin Story Discovering Superpowers Explore domain with types data List a = EmptyList | Cons a (List a) type Pred a = (a -> Bool) type Comp a = (a -> a -> Ordering) sortBy :: Comp a -> List a -> List a filter :: Pred a -> List a -> List a reverse :: List a -> List a last, first :: List a -> Maybe a Susan Potter Thinking in Properties 2020-08-01 3 / 41

8. An Origin Story Discovering Superpowers Explore domain with usage examples -- | Reverse the elements of a list -- >>> reverse (Cons 1 (Cons 2 (Cons 3 EmptyList)))→ -- Cons 3 (Cons 2 (Cons 1 EmptyList)) -- -- >>> reverse EmptyList -- EmptyList reverse :: List a -> List a Susan Potter Thinking in Properties 2020-08-01 4 / 41

9. An Origin Story Discovering Superpowers Encode examples as re-runnable tests describe "Lib.reverse" $ do it "returns [5,4,3,2,1] given [1,2,3,4,5]" $ do reverse [1,2,3,4,5] `shouldBe` [5,4,3,2,1] it "returns empty list given empty list" $ do reverse [] `shouldBe` [] Susan Potter Thinking in Properties 2020-08-01 5 / 41

10. An Origin Story Discovering Superpowers Rinse & Repeat Susan Potter Thinking in Properties 2020-08-01 6 / 41

11. An Origin Story Discovering Superpowers Continuous Learning Figure 1: Schedule for Erlang Factory SF 2011 where my mind was blown Susan Potter Thinking in Properties 2020-08-01 7 / 41

12. An Origin Story Harnessing Newly Found Superpowers Characteristics of property-based testing Where we have: • generators that produce random ”arbitrary” values for inputs • general rules that hold without knowing inputs upfront • shrinking of failed values • test runs assert rule multiple times using new generated values Related terms: generative testing, fuzz testing (or ”fuzzing”) Susan Potter Thinking in Properties 2020-08-01 8 / 41

16. An Origin Story Harnessing Newly Found Superpowers One General Rule: Round-tripping (two rides gets us back home) Examples: -- Assumes x is encodable and decodable roundtrip0 = x -> decode (encode x) == Just x roundtrip1 = x -> decodeBase64 (encodeBase64 x) == Right x Counter-examples: • sha256 is one way! Susan Potter Thinking in Properties 2020-08-01 9 / 41

17. An Origin Story Harnessing Newly Found Superpowers Introducing generators To make these ideas concrete we will be using hedgehog with these imports: import Hedgehog import qualified Hedgehog.Gen as Gen import qualified Hedgehog.Range as Range import Control.Monad (replicateM) import Data.Function (($), (.)) Hedgehog integrates shrinking with generation. We will not discuss this difference to QuickCheck but read Well Typed’s blog post about this here. Susan Potter Thinking in Properties 2020-08-01 10 / 41

18. An Origin Story Harnessing Newly Found Superpowers Primitive generators by example >>> replicateM 25 $ Gen.sample Gen.lower "okohcpxrkfunkmwnqujnnhxkg" >>> let currencies = [ "USD", "JPY", "EUR", "CHF", "CNY" ] >>> replicateM 5 $ Gen.sample $ Gen.element currencies ["USD","CNY","USD","JPY","USD"] >>> replicateM 5 $ Gen.sample $ Gen.choice [ Gen.ascii, Gen.unicode ]→ ['f', 'c', 'j', '1068213', '<'] Susan Potter Thinking in Properties 2020-08-01 11 / 41

19. An Origin Story Harnessing Newly Found Superpowers Generating your domain’s data, 1/2 Suppose our domain looks like this: import Data.Word (Word8, Word16) type W16 = Word16 data IP = IPv4 Word8 Word8 Word8 Word8 | IPv6 W16 W16 W16 W16 W16 W16 W16 W16 Susan Potter Thinking in Properties 2020-08-01 12 / 41

20. An Origin Story Harnessing Newly Found Superpowers Generating your domain’s data, 2/2 genW8 = Gen.word8 Range.constantBounded genW16 = Gen.word16 Range.constantBounded genIPv4 = IPv4 <$> genW8 <*> genW8 <*> genW8 <*> genW8 genIPv6 = IPv6 <$> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 genAnyIP = Gen.choice [ genIPv4, genIPv6 ] sampleIPs n = replicateM n (Gen.sample genAnyIP) Susan Potter Thinking in Properties 2020-08-01 13 / 41

21. An Origin Story Harnessing Newly Found Superpowers Sampling generated domain data >>> sampleIPs 3 [ "136.59.149.200" , "338d:2397:f612:e036:b27c:2298:4db8:b933" , "5.38.65.204" ] Susan Potter Thinking in Properties 2020-08-01 14 / 41

22. An Origin Story Harnessing Newly Found Superpowers Writing Our First Property! genList :: MonadGen m => m a -> m [a] genList = Gen.list (Range.linear 0 1000) genInt = Gen.int (Range.linear 0 100000) -- "round-tripping" property prop_reverse_reverse = property $ do xs <- forAll $ genList genInt Lib.reverse (Lib.reverse xs) === xs Susan Potter Thinking in Properties 2020-08-01 15 / 41

23. An Origin Story Harnessing Newly Found Superpowers Reviewing Our First Property! Questions about prop_reverse_reverse: • Does it assert anything about reverse ’s speciﬁcation? • Do callers of reverse need to exploit ”round-tripping”? • Does an implementation exist that typechecks yet fails this property? • Are we generating interesting data given the operation’s type? • Are we resigned to function-level property testing? Susan Potter Thinking in Properties 2020-08-01 16 / 41

28. An Origin Story Harnessing Newly Found Superpowers Quick Review: Example-based tests over time t ~ 0 → t → ∞ Quick ⌣ →  / ⌢ Coverage ? →  / ⌢ Repeatable ⌣ →  / ⌢ Documents usage  →  / ⌢ Documents contract ⌢ → ⌢ Effort  →  / ⌢ • can measure coverage • ﬁxtures provide test data • interesting ﬁxtures brittle • over time tends to ⌢ Susan Potter Thinking in Properties 2020-08-01 17 / 41

33. An Origin Story Harnessing Newly Found Superpowers Quick Review: Property-based tests initially t ~ 0 Quick ⌣ Coverage ⌣ Repeatable ⌣ Documents usage  Documents contract ⌣ /  Effort ? • Can we measure coverage ? • We need to maintain generators instead of ﬁxtures! • Not constrained by imagination! ⌣ • Am I smart enough to think up relevant and meaningful properties ? Susan Potter Thinking in Properties 2020-08-01 18 / 41

38. Mental Models

39. Mental Models Deriving Properties: Algebraic laws Law What might it be good for Idempotency e.g. event log effects, REST APIs, conﬁg effects Associativity e.g. map/reduce distribution Commutativity e.g. map/reduce local parallelism Distributivity e.g. stable or performant rewrite rules Identity element e.g. for invariance at that value for operation Round-tripping e.g. encoding/decoding Absorption e.g. boolean algebra rewrites Transitivity e.g. dependency closures Susan Potter Thinking in Properties 2020-08-01 19 / 41

40. Mental Models Algebraic laws: Idempotency (running 1+ times yields same result) Examples: idem0 = x -> abs x == abs (abs x) idem2 = x -> toUpper s == toUpper (toUpper s) • curl -XPUT https://foo.bar/resource/123 -d baz=qux Counter-example: • curl -XPOST https://foo.bar/resource -d baz=qux Algebra: Given x ∈ A and f ∈ (A → A) and f(x) = f(f(x)) then f is idempotent. Susan Potter Thinking in Properties 2020-08-01 20 / 41

41. Mental Models Algebraic laws: Associativity (brackets to the left or right) Examples: assoc0 = x y z -> (x + y) + z == x + (y + z) assoc1 = x y z -> (x ++ y) ++ z == x ++ (y ++ z) Counter-example: • (x − y) − z = x − (y − z) Algebra: Given x, y, z ∈ A, ⊕ ∈ (A → A → A) and (x ⊕ y) ⊕ z = x ⊕ (y ⊕ z) then ⊕ is associative. Susan Potter Thinking in Properties 2020-08-01 21 / 41

42. Mental Models Algebraic laws: Commutativity (any order will do) Examples: comm0 = x y -> x + y == y + x comm1 = x y -> x * y == y * x Counter-example: • x + +y = y + +x • x − y = y − x Algebra: Given x, y ∈ A, ⊕ ∈ (A → A → A) and $x ⊕ y = y ⊕x then ⊕ is commutative. Susan Potter Thinking in Properties 2020-08-01 22 / 41

43. Mental Models Algebraic laws: Distributivity (one operation over another) Examples: dist0 = x y z -> x*(y + z) == x*y + x*z Counter-example: • x + (y ∗ z) = x + y ∗ x + z where x = 1, y = 2, z = 3 1 + (2 ∗ 3) = 1 + 2 ∗ 1 + 3 ⇒ 7 = 6 Algebra: Given x, y, z ∈ A and ⊕, ⊗ ∈ (A → A → A) then ⊗ is distributive over ⊕ when x ⊗ (y ⊕ z) = x ⊗ y ⊕ x ⊗ z Susan Potter Thinking in Properties 2020-08-01 23 / 41

44. Mental Models Algebraic laws: Identity element Examples: identity0 = x -> 0 + x == x identity1 = x -> False || x == x identity2 = s -> "" ++ s == s Counter-example: -NonEmpty does not have an identity element Algebra: ∃e ∈ A, ∀a ∈ A, e ⊕ a = a then e is the identity element in A. Susan Potter Thinking in Properties 2020-08-01 24 / 41

45. Mental Models Algebraic laws: Absorption Examples: absorption0 = a b -> (a || (a && b)) == a absorption1 = a b -> (a && (a || b)) == a Algebra: Given ∧, ∨ ∈ (A → A → A) and a, b ∈ A then when a ∧ (a ∨ b) = a = a ∨ (a ∧ b) Susan Potter Thinking in Properties 2020-08-01 25 / 41

46. Mental Models Deriving Properties: Relational laws -- implicitly expect sort and last to be correct prop_max_is_last_of_sort = property $ do xs <- forAll $ genList Gen.ascii Just (max xs) === last (sort xs) prop_last_is_first_of_reversed = property $ do xs <- forAll $ genList Gen.unicode last xs === first (reverse xs) Susan Potter Thinking in Properties 2020-08-01 26 / 41

47. Mental Models Deriving Properties: Abstraction laws Using hedgehog-classes package we can check our typeclass instances according to the abstraction laws: import Hedgehog import Hedgehog.Classes import qualified Hedgehog.Gen as Gen import qualified Hedgehog.Range as Range investmentPortfolioSemigroup = lawsCheck (semigroupLaws genInvestmentPortfolio) portfolioFoldable = lawsCheck (foldableLaws genPortfolio) Susan Potter Thinking in Properties 2020-08-01 27 / 41

48. Mental Models Deriving Properties: Reﬂections, Rotations, Distortions prop_rotated_colors_same = property $ do img <- forAll $ genImage colors (rotate90 img) === colors img • normalizing audio shouldn’t change time length • reversing a list shouldn’t change length Susan Potter Thinking in Properties 2020-08-01 28 / 41

49. Mental Models Deriving Properties: Informal model checking Sometimes you can model the basic state machine of a system simply: • model of interesting parts of stateful system • not exhaustive • thinking in state machine models • generate sequence or call graph of commands • assert pre- and post-conditions or invariants • careful you don’t write a second implementation of the SUT just to test it! Susan Potter Thinking in Properties 2020-08-01 29 / 41

50. Mental Models Deriving Properties: Legacy oracles Replacing legacy systems: • bind to old lib as oracle • assert new rewritten library matches oracle for same inputs • good for e.g. calculation engines or data pipelines • might need large build engineering effort Susan Potter Thinking in Properties 2020-08-01 30 / 41

51. Mental Models Deriving Properties: Does not barf Wrapping lower-level code via FFI: • gaps between foreign input or output types and native types • runtime exceptions thrown for some input values (inform design) • sanity checking FFI wrapping Susan Potter Thinking in Properties 2020-08-01 31 / 41

52. Mental Models Deriving Properties: Metamorphic relations • Running against SUT twice with possibly different inputs • A relation exists between those inputs • Assert a relation exists between the outputs of those system runs An example across inputs and outputs, but the relation between inputs and outputs can be different: x, y ∈ Inputs, x ≤ y, x′ = SUT(x), y′ = SUT(y) then x′ ≤ y′ Susan Potter Thinking in Properties 2020-08-01 32 / 41

53. Mental Models Deriving Properties: Metamorphic relation patterns • Input equivalence • Shufﬂing • Conjunctive conditions • Disjunctive conditions • Disjoint partitions • Complete partitions • Partition difference Susan Potter Thinking in Properties 2020-08-01 33 / 41

54. Mental Models Deriving Properties: Heckle Yourself! • mutation testing • alter your code until your tests fail • if no tests fail, throw your tests out (curation) • question your assumptions Susan Potter Thinking in Properties 2020-08-01 34 / 41

55. Beyond testing

56. Beyond testing Properties of Delivery Pipelines Property: Source consistency Ensuring fast-forward only ”merges”: main() { local -r mergeBase="$(git merge-base HEAD origin/deploy)" local -r deployHead="$(git rev-parse origin/deploy)" test "${mergeBase}" = "${deployHead}" } set -e main # should exit with 0 for success Susan Potter Thinking in Properties 2020-08-01 35 / 41

57. Beyond testing Stateful Migrations (in production) Properties: pre and post conditions and invariants between migration phases • moving a stateful cluster from one datacenter to another • upgrading Elastic search into a new cluster • online schema migrations of large tables with binlog syncing and atomic rename (e.g. MySQL) Susan Potter Thinking in Properties 2020-08-01 36 / 41

60. Beyond testing System Monitoring Property: Connectedness! Given public name for service name: • resolve name to A records (IPs) • ∀ IPs should negotiate TLS handshake • ∀ IPs should make HTTP request with Host Susan Potter Thinking in Properties 2020-08-01 37 / 41

63. Beyond testing Production Data Checks Sometimes your generators don’t generate data you see in production! Legacy systems exist with no property-based testing toolchain! • Structured logging can record inputs and results; validate OOB • Run property checks against production inputs and outputs in Haskell :) Susan Potter Thinking in Properties 2020-08-01 38 / 41

64. Wrapping Up

65. Wrapping Up In Closing • Not all properties are useful • Initially hard to think up useful properties genMentalModels = Gen.choice [ genAlgebraicLaws, genRelationalLaws, genAbstrationLaws, genStateMachines, genMetamorphicRelations, genHeckleYourCode, genTestingInProduction ] Susan Potter Thinking in Properties 2020-08-01 39 / 41

66. Wrapping Up In Closing • Not all properties are useful • Initially hard to think up useful properties genMentalModels = Gen.choice [ genAlgebraicLaws, genRelationalLaws, genAbstrationLaws, genStateMachines, genMetamorphicRelations, genHeckleYourCode, genTestingInProduction ] Susan Potter Thinking in Properties 2020-08-01 39 / 41

67. Wrapping Up Questions? GitHub @mbbx6spp LinkedIn /in/susanpotter Twitter @SusanPotter Web Personal site Consulting Training Thank you for listening! Susan Potter Thinking in Properties 2020-08-01 40 / 41

68. Wrapping Up Credits • Photo by Elias Castillo on Unsplash • Photo by Juan Rumimpunu on Unsplash • Photo by LinkedIn Sales Navigator on Unsplash • Photo by Leonardo Sanches on Unsplash • Photo by Mélissa Jeanty on Unsplash • Photo by Chris Liverani on Unsplash • Photo by <a href=”https: //unsplash.com/@spanic?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText”>Damir Spanic</a> on <a href=”https://unsplash.com/s/photos/baseball-bat?utm_source=unsplash&utm_medium=referral&utm_content= creditCopyText”>Unsplash</a></span> • Photo by <a href=”https: //unsplash.com/@serrah?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText”>Serrah Galos</a> on <a href=”https://unsplash.com/s/photos/reflection?utm_source=unsplash&utm_medium=referral&utm_content= creditCopyText”>Unsplash</a></span> • Photo by Sergey Zolkin on Unsplash • Photo by Roman Mager on Unsplash • Photo by Miguel Ibáñez on Unsplash • Photo by Science in HD on Unsplash • Photo by Steve Douglas on Unsplash • Photo by Natalie Parham on Unsplash Susan Potter Thinking in Properties 2020-08-01 41 / 41

Thinking in Properties

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Thinking in Properties

Semelhante a Thinking in Properties (20)

Mais de Susan Potter

Mais de Susan Potter (17)

Último

Último (20)

Thinking in Properties