Test suites with many example-based tests can contain a lot of repetition. While it’s possible to factor out much of the repetition using regular Haskell code, it can be useful to construct test cases from external files, which can be generated from other sources, or constructed by people not doing Haskell programming. In this video I’ll demonstrate how to create a dynamic test suite based on examples in an external CSV file. The tests can be run individually using Tasty patterns. It is easy to add new examples to the CSV, and the Haskell test code doesn’t even need recompilation.

Show Notes

The function that we’ll be testing is called pluralize. It’s a function of type Text -> Text, which, given an empty string, returns an empty string. Given any other string, we will check if the last character is an s, and if so, we’ll return the string. Otherwise, we’ll append an s at the end.

pluralize :: Text -> Text
pluralize "" = ""
pluralize t =
  case Text.last t of
    's' -> t
    _ -> t <> "s"

We can try this function out in the REPL:

*Lib> pluralize "dog"
"dogs"
*Lib> pluralize "dogs"
"dogs"
*Lib> pluralize "cats"
"cats"
*Lib> pluralize "cat"
"cats"

Now, we’d like to test this function. In a Tasty test file, we can define an Hspec test using spec_ followed by a name. Using do notaion, we can combine a bunch of test cases.

spec_pluralize :: Spec
spec_pluralize = do

  it "pluralizes 'cat' to 'cats'" $
    pluralize "cat" `shouldBe` "cats"

  it "pluralizes 'cats' to 'cats'" $
    pluralize "cats" `shouldBe` "cats"

  it "pluralizes 'dog' to 'dogs'" $
    pluralize "dog" `shouldBe` "dogs"

  it "pluralizes 'dogs' to 'dogs'" $
    pluralize "dogs" `shouldBe` "dogs"

Running the test suite in the REPL shows us that all tests pass:

*Main Lib LibTest> :main
/home/owi/coda/dynamic-test-suite/test/Main.hs
  pluralize
    pluralizes 'cat' to 'cats':  OK
    pluralizes 'cats' to 'cats': OK
    pluralizes 'dog' to 'dogs':  OK
    pluralizes 'dogs' to 'dogs': OK

All 4 tests passed (0.00s)
*** Exception: ExitSuccess

These test cases have a lot of repetition. We’d like to keep separate test cases, but not repeat all the code. Let’s instead write an external file called test/plurals.csv. Here we list all inputs and their corresponding expected outputs, like cat becoming cats.

cat,cats
cats,cats
dog,dogs
dogs,dogs

This file could be edited in a spreadsheet editor, or generated from some external source, and contain thousands of examples. Let’s rewrite our test case file to use this external CSV file, instead.

We need some new imports:

import           Control.Monad
import           Data.Semigroup
import qualified Data.Text.IO     as Text
import           Text.Printf

We remove some of the hard-coded examples from before, and instead define readExamples, an IO action return [(Text, Text)]. It’s defined as mapping the asPair function over the lines read from a file; test/plurals.csv, the file we wrote before. In case there are two items on a line, we return those in a tuple, otherwise we will fail with an error message.

readExamples :: IO [(Text, Text)]
readExamples =
  mapM asPair =<< Text.lines <$> Text.readFile "test/plurals.csv"
  where
    asPair line =
      case Text.splitOn "," line of
        [input, expected] -> pure (input, expected)
        _ -> fail ("Invalid example line: " <> Text.unpack line)

If you’re parsing more complex CSV data, consider using the Cassava package, instead of manually splitting lines with text.

Now, here comes the trick. In our spec we will use runIO to embed an IO action:

spec_pluralize :: Spec
spec_pluralize = do
  examples <- runIO readExamples
  ...

With our examples read, we can loop over them using forM_, and construct a test case for each example. We will generate test cases looking like before, but using the text strings from the file.

  ...
  forM_ examples $ \(input, expected) ->
    it (printf "pluralizes '%s' to '%s'" input expected) $
      pluralize input `shouldBe` expected

Running the test suite gives us the same result as before, when we had examples written directly with Hspec.

Now, someone figures out that some words aren’t pluralized correctly; words like fungus and schema.

cat,cats
cats,cats
dog,dogs
dogs,dogs
fungus,fungi
schema,schemata

We run the test suite, and see that we have two new failures:

*Main Lib LibTest> :main
/home/owi/coda/dynamic-test-suite/test/Main.hs
  pluralize
    pluralizes 'cat' to 'cats':        OK
    pluralizes 'cats' to 'cats':       OK
    pluralizes 'dog' to 'dogs':        OK
    pluralizes 'dogs' to 'dogs':       OK
    pluralizes 'fungus' to 'fungi':    FAIL
      expected "fungi", but got "fungus"
    pluralizes 'schema' to 'schemata': FAIL
      expected "schemata", but got "schemas"

2 out of 6 tests failed (0.00s)

If you add hundreds, or thousands, of these example, you might want to filter them to only show the failures. We can do that using by setting the environment TASTY_HIDE_SUCCESSES to the string True. Restarting the REPL and running the tests, we see only the failures:

$ TASTY_HIDE_SUCCESSES=True stack test
...

  pluralize
    pluralizes 'fungus' to 'fungi':    FAIL
      expected "fungi", but got "fungus"
    pluralizes 'schema' to 'schemata': FAIL
      expected "schemata", but got "schemas"

2 out of 6 tests failed (0.01s)

Exporting an environment variable in Vim can be done using let.

You might want to only run specific test cases. You can use the -p flag in Tasty with a pattern. Here we only run the fungus test:

$ stack test --ta '-p fungus'
...

  pluralize
    pluralizes 'fungus' to 'fungi': FAIL
      expected "fungi", but got "fungus"

1 out of 1 tests failed (0.00s)

As commented by @mwotton, if the extra-source-files field in the project’s Cabal file includes the CSV file path, stack test --file-watch will rerun tests on changes to the CSV file.

OK, let’s fix our implementation, even if somewhat naively. We pattern match on those two special cases:

pluralize :: Text -> Text
pluralize "" = ""
pluralize "fungus" = "fungi"
pluralize "schema" = "schemata"
pluralize t =
  case Text.last t of
    's' -> t
    _   -> t <> "s"

We rerun the tests, and see that they all pass.

That’s it! You can use runIO to embed IO actions in Hspec, and dynamically create your test suite.

Thank you for watching and reading!