Dynamic Test Suites in Haskell using Hspec and Tasty
Test suites with many example-based tests can contain a lot of repetition. While it’s possible to factor out much of the repetition using regular Haskell code, it can be useful to construct test cases from external files, which can be generated from other sources, or constructed by people not doing Haskell programming. In this video I’ll demonstrate how to create a dynamic test suite based on examples in an external CSV file. The tests can be run individually using Tasty patterns. It is easy to add new examples to the CSV, and the Haskell test code doesn’t even need recompilation.
The function that we’ll be testing is called
pluralize. It’s a function of type
Text -> Text, which, given an empty string, returns an empty string. Given any other string, we will check if the last character is an s, and if so, we’ll return the string. Otherwise, we’ll append an s at the end.
pluralize :: Text -> Text pluralize "" = "" pluralize t = case Text.last t of 's' -> t _ -> t <> "s"
We can try this function out in the REPL:
*Lib> pluralize "dog" "dogs" *Lib> pluralize "dogs" "dogs" *Lib> pluralize "cats" "cats" *Lib> pluralize "cat" "cats"
Now, we’d like to test this function. In a Tasty test file, we can define an Hspec test using
spec_ followed by a name. Using do notaion, we can combine a bunch of test cases.
spec_pluralize :: Spec spec_pluralize = do it "pluralizes 'cat' to 'cats'" $ pluralize "cat" `shouldBe` "cats" it "pluralizes 'cats' to 'cats'" $ pluralize "cats" `shouldBe` "cats" it "pluralizes 'dog' to 'dogs'" $ pluralize "dog" `shouldBe` "dogs" it "pluralizes 'dogs' to 'dogs'" $ pluralize "dogs" `shouldBe` "dogs"
Running the test suite in the REPL shows us that all tests pass:
*Main Lib LibTest> :main /home/owi/coda/dynamic-test-suite/test/Main.hs pluralize pluralizes 'cat' to 'cats': OK pluralizes 'cats' to 'cats': OK pluralizes 'dog' to 'dogs': OK pluralizes 'dogs' to 'dogs': OK All 4 tests passed (0.00s) *** Exception: ExitSuccess
These test cases have a lot of repetition. We’d like to keep separate test cases, but not repeat all the code. Let’s instead write an external file called
test/plurals.csv. Here we list all inputs and their corresponding expected outputs, like cat becoming cats.
cat,cats cats,cats dog,dogs dogs,dogs
This file could be edited in a spreadsheet editor, or generated from some external source, and contain thousands of examples. Let’s rewrite our test case file to use this external CSV file, instead.
We need some new imports:
import Control.Monad import Data.Semigroup import qualified Data.Text.IO as Text import Text.Printf
We remove some of the hard-coded examples from before, and instead define
readExamples, an IO action return
[(Text, Text)]. It’s defined as mapping the
asPair function over the lines read from a file;
test/plurals.csv, the file we wrote before. In case there are two items on a line, we return those in a tuple, otherwise we will fail with an error message.
readExamples :: IO [(Text, Text)] readExamples = mapM asPair =<< Text.lines <$> Text.readFile "test/plurals.csv" where asPair line = case Text.splitOn "," line of [input, expected] -> pure (input, expected) _ -> fail ("Invalid example line: " <> Text.unpack line)
If you’re parsing more complex CSV data, consider using the Cassava package, instead of manually splitting lines with
Now, here comes the trick. In our spec we will use
runIO to embed an
spec_pluralize :: Spec spec_pluralize = do examples <- runIO readExamples ...
With our examples read, we can loop over them using
forM_, and construct a test case for each example. We will generate test cases looking like before, but using the text strings from the file.
... forM_ examples $ \(input, expected) -> it (printf "pluralizes '%s' to '%s'" input expected) $ pluralize input `shouldBe` expected
Running the test suite gives us the same result as before, when we had examples written directly with Hspec.
Now, someone figures out that some words aren’t pluralized correctly; words like fungus and schema.
cat,cats cats,cats dog,dogs dogs,dogs fungus,fungi schema,schemata
We run the test suite, and see that we have two new failures:
*Main Lib LibTest> :main /home/owi/coda/dynamic-test-suite/test/Main.hs pluralize pluralizes 'cat' to 'cats': OK pluralizes 'cats' to 'cats': OK pluralizes 'dog' to 'dogs': OK pluralizes 'dogs' to 'dogs': OK pluralizes 'fungus' to 'fungi': FAIL expected "fungi", but got "fungus" pluralizes 'schema' to 'schemata': FAIL expected "schemata", but got "schemas" 2 out of 6 tests failed (0.00s)
If you add hundreds, or thousands, of these example, you might want to filter them to only show the failures. We can do that using by setting the environment
TASTY_HIDE_SUCCESSES to the string
True. Restarting the REPL and running the tests, we see only the failures:
$ TASTY_HIDE_SUCCESSES=True stack test ... pluralize pluralizes 'fungus' to 'fungi': FAIL expected "fungi", but got "fungus" pluralizes 'schema' to 'schemata': FAIL expected "schemata", but got "schemas" 2 out of 6 tests failed (0.01s)
Exporting an environment variable in Vim can be done using
You might want to only run specific test cases. You can use the
-p flag in Tasty with a pattern. Here we only run the fungus test:
$ stack test --ta '-p fungus' ... pluralize pluralizes 'fungus' to 'fungi': FAIL expected "fungi", but got "fungus" 1 out of 1 tests failed (0.00s)
As commented by @mwotton, if the
extra-source-files field in the project’s Cabal file includes the CSV file path,
stack test --file-watch will rerun tests on changes to the CSV file.
OK, let’s fix our implementation, even if somewhat naively. We pattern match on those two special cases:
pluralize :: Text -> Text pluralize "" = "" pluralize "fungus" = "fungi" pluralize "schema" = "schemata" pluralize t = case Text.last t of 's' -> t _ -> t <> "s"
We rerun the tests, and see that they all pass.
That’s it! You can use
runIO to embed IO actions in Hspec, and dynamically create your test suite.
Thank you for watching and reading!