SwiftCheck

QuickCheck for Swift.

For those already familiar with the Haskell library, check out the source. For
everybody else, see the Tutorial Playground for a
beginner-level introduction to the major concepts and use-cases of this library.

Introduction

SwiftCheck is a testing library that automatically generates random data for
testing of program properties. A property is a particular facet of an algorithm
or data structure that must be invariant under a given set of input data,
basically an XCTAssert on steroids. Where before all we could do was define
methods prefixed by test and assert, SwiftCheck allows program properties and
tests to be treated like data.

To define a program property the forAll quantifier is used with a type
signature like (A, B, C, ... Z) -> Testable where A : Arbitrary, B : Arbitrary ... Z : Arbitrary. SwiftCheck implements the Arbitrary protocol for most Swift
Standard Library types and implements the Testable protocol for Bool and
several other related types. For example, if we wanted to test the property
that every Integer is equal to itself, we would express it as such:

func testAll() {
    // 'property' notation allows us to name our tests.  This becomes important
    // when they fail and SwiftCheck reports it in the console.
    property("Integer Equality is Reflexive") <- forAll { (i : Int) in
        return i == i
    }
}
Swift

For a less contrived example, here is a program property that tests whether
Array identity holds under double reversal:

property("The reverse of the reverse of an array is that array") <- forAll { (xs : [Int]) in
    // This property is using a number of SwiftCheck's more interesting 
    // features.  `^&&^` is the conjunction operator for properties that turns
    // both properties into a larger property that only holds when both sub-properties
    // hold.  `<?>` is the labelling operator allowing us to name each sub-part
    // in output generated by SwiftCheck.  For example, this property reports:
    //
    // *** Passed 100 tests
    // (100% , Right identity, Left identity)
    return
        (xs.reversed().reversed() == xs) <?> "Left identity"
        ^&&^
        (xs == xs.reversed().reversed()) <?> "Right identity"
}
Swift

Because SwiftCheck doesn't require tests to return Bool, just Testable, we
can produce tests for complex properties with ease:

property("Shrunken lists of integers always contain [] or [0]") <- forAll { (l : [Int]) in
    // Here we use the Implication Operator `==>` to define a precondition for
    // this test.  If the precondition fails the test is discarded.  If it holds
    // the test proceeds.
    return (!l.isEmpty && l != [0]) ==> {
        let ls = self.shrinkArbitrary(l)
        return (ls.filter({ $0 == [] || $0 == [0] }).count >= 1)
    }
}
Swift

Properties can even depend on other properties:

property("Gen.one(of:) multiple generators picks only given generators") <- forAll { (n1 : Int, n2 : Int) in
    let g1 = Gen.pure(n1)
    let g2 = Gen.pure(n2)
    // Here we give `forAll` an explicit generator.  Before SwiftCheck was using
    // the types of variables involved in the property to create an implicit
    // Generator behind the scenes.
    return forAll(Gen.one(of: [g1, g2])) { $0 == n1 || $0 == n2 }
}
Swift

All you have to figure out is what to test. SwiftCheck will handle the rest.

Shrinking

What makes QuickCheck unique is the notion of shrinking test cases. When fuzz
testing with arbitrary data, rather than simply halt on a failing test, SwiftCheck
will begin whittling the data that causes the test to fail down to a minimal
counterexample.

For example, the following function uses the Sieve of Eratosthenes to generate
a list of primes less than some n:

/// The Sieve of Eratosthenes:
///
/// To find all the prime numbers less than or equal to a given integer n:
///    - let l = [2...n]
///    - let p = 2
///    - for i in [(2 * p) through n by p] {
///          mark l[i]
///      }
///    - Remaining indices of unmarked numbers are primes
func sieve(_ n : Int) -> [Int] {
    if n <= 1 {
        return []
    }

    var marked : [Bool] = (0...n).map { _ in false }
    marked[0] = true
    marked[1] = true

    for p in 2..<n {
        for i in stride(from: 2 * p, to: n, by: p) {
            marked[i] = true
        }
    }

    var primes : [Int] = []
    for (t, i) in zip(marked, 0...n) {
        if !t {
            primes.append(i)
        }
    }
    return primes
}

/// Short and sweet check if a number is prime by enumerating from 2...⌈√(x)⌉ and checking 
/// for a nonzero modulus.
func isPrime(n : Int) -> Bool {
    if n == 0 || n == 1 {
        return false
    } else if n == 2 {
        return true
    }
    
    let max = Int(ceil(sqrt(Double(n))))
    for i in 2...max {
        if n % i == 0 {
            return false
        }
    }
    return true
}
Swift

We would like to test whether our sieve works properly, so we run it through
SwiftCheck with the following property:

import SwiftCheck

property("All Prime") <- forAll { (n : Int) in
    return sieve(n).filter(isPrime) == sieve(n)
}
Swift

Which produces the following in our testing log:

Test Case '-[SwiftCheckTests.PrimeSpec testAll]' started.
*** Failed! Falsifiable (after 10 tests):
4

Indicating that our sieve has failed on the input number 4. A quick look back
at the comments describing the sieve reveals the mistake immediately:

- for i in stride(from: 2 * p, to: n, by: p) {
+ for i in stride(from: 2 * p, through: n, by: p) {
Diff

Running SwiftCheck again reports a successful sieve of all 100 random cases:

*** Passed 100 tests

Custom Types

SwiftCheck implements random generation for most of the types in the Swift
Standard Library. Any custom types that wish to take part in testing must
conform to the included Arbitrary protocol. For the majority of types, this
means providing a custom means of generating random data and shrinking down to
an empty array.

For example:

import SwiftCheck
 
public struct ArbitraryFoo {
    let x : Int
    let y : Int

    public var description : String {
        return "Arbitrary Foo!"
    }
}

extension ArbitraryFoo : Arbitrary {
    public static var arbitrary : Gen<ArbitraryFoo> {
        return Gen<(Int, Int)>.zip(Int.arbitrary, Int.arbitrary).map(ArbitraryFoo.init)
    }
}

class SimpleSpec : XCTestCase {
    func testAll() {
        property("ArbitraryFoo Properties are Reflexive") <- forAll { (i : ArbitraryFoo) in
            return i.x == i.x && i.y == i.y
        }
    }
}
Swift

There's also a Gen.compose method which allows you to procedurally compose
values from multiple generators to construct instances of a type:

public static var arbitrary : Gen<MyClass> {
    return Gen<MyClass>.compose { c in
        return MyClass(
            // Use the nullary method to get an `arbitrary` value.
            a: c.generate(),

            // or pass a custom generator
            b: c.generate(Bool.suchThat { $0 == false }),

            // .. and so on, for as many values and types as you need.
            c: c.generate(), ...
        )
    }
}
Swift

Gen.compose can also be used with types that can only be customized with setters:

public struct ArbitraryMutableFoo : Arbitrary {
    var a: Int8
    var b: Int16
    
    public init() {
        a = 0
        b = 0
    }
    
    public static var arbitrary: Gen<ArbitraryMutableFoo> {
        return Gen.compose { c in
            var foo = ArbitraryMutableFoo()
            foo.a = c.generate()
            foo.b = c.generate()
            return foo
        }
    }
}
Swift

For everything else, SwiftCheck defines a number of combinators to make working
with custom generators as simple as possible:

let onlyEven = Int.arbitrary.suchThat { $0 % 2 == 0 }

let vowels = Gen.fromElements(of: [ "A", "E", "I", "O", "U" ])

let randomHexValue = Gen<UInt>.choose((0, 15))

let uppers = Gen<Character>.fromElements(in: "A"..."Z")
let lowers = Gen<Character>.fromElements(in: "a"..."z")
let numbers = Gen<Character>.fromElements(in: "0"..."9")
 
/// This generator will generate `.none` 1/4 of the time and an arbitrary
/// `.some` 3/4 of the time
let weightedOptionals = Gen<Int?>.frequency([
    (1, Gen<Int?>.pure(nil)),
    (3, Int.arbitrary.map(Optional.some))
])
Swift

For instances of many complex or "real world" generators, see
ComplexSpec.swift.

System Requirements

SwiftCheck supports OS X 10.9+ and iOS 7.0+.

Setup

SwiftCheck can be included one of two ways:

Using The Swift Package Manager

  • Add SwiftCheck to your Package.swift file's dependencies section:
.package(url: "https://github.com/typelift/SwiftCheck.git", from: "0.8.1")
Swift

Using Carthage

  • Add SwiftCheck to your Cartfile
  • Run carthage update
  • Drag the relevant copy of SwiftCheck into your project.
  • Expand the Link Binary With Libraries phase
  • Click the + and add SwiftCheck
  • Click the + at the top left corner to add a Copy Files build phase
  • Set the directory to Frameworks
  • Click the + and add SwiftCheck

Using CocoaPods

  • Add our Pod to your podfile.
  • Run $ pod install in your project directory.

Framework

  • Drag SwiftCheck.xcodeproj into your project tree
    as a subproject
  • Under your project's Build Phases, expand Target Dependencies
  • Click the + and add SwiftCheck
  • Expand the Link Binary With Libraries phase
  • Click the + and add SwiftCheck
  • Click the + at the top left corner to add a Copy Files build phase
  • Set the directory to Frameworks
  • Click the + and add SwiftCheck

License

SwiftCheck is released under the MIT license.

GitHub

https://github.com/typelift/SwiftCheck