Testing the failure cases of code is often as important as testing the successful paths. The Pandoc filter we worked on in the previous episode returns its result, and throws exceptions with formatted strings, in the IO type. This makes testing the failure cases much harder. In this episode we will introduce a function exposing the errors using the Either type, and rewrite the test suite to match the new behavior.

Show Notes

In the last episode we introduced the InclusionMode data type, making valid modes of execution explicit.

data InclusionMode = SnippetMode Text | RangeMode Range | EntireFileMode
  deriving (Show, Eq)

Our test suite still needs some changes though, to reflect how you cannot run the filter in multiple modes simultaneously. Switching over to the test runner, we see that two cases fail.

/home/owi/haskell-at-work/pandoc-include-code/test/Driver.hs
  includeCode
    includes snippet:                                                  OK
    includes snippet within start and end line:                        FAIL
      IOException of type UserError (user error (Conflicting inclusion modes: [RangeMode (Range {rangeStart = 1, rangeEnd = 3}),SnippetMode "foo"]))
    includes snippet by exact name, not prefix:                        OK
    excludes snippet outside start and end line:                       FAIL
      IOException of type UserError (user error (Conflicting inclusion modes: [RangeMode (Range {rangeStart = 2, rangeEnd = 3}),SnippetMode "foo"]))
    includes only lines between start and end line:                    OK
    dedents lines as much as specified:                                OK
    dedents lines as much as possible without removing non-whitespace: OK

2 out of 7 tests failed (0.00s)

These cases were previously valid, but now we want to instead assert that they fail. Looking at the test suite, we see that the test cases use a function includeCode that returns an IO action of a code block.

includeCode :: String -> [(String, String)] -> IO Block
includeCode fixtureName attrs = Filter.includeCode
  (Just (Format "html5"))
  ( CodeBlock
    ("", [], mconcat [[("include", "test/fixtures/" ++ fixtureName)], attrs])
    ""
  )

spec_includeCode = do
  it "includes snippet" $
    includeCode "foo-snippet.txt" [("snippet", "foo")]
    `shouldReturn` codeBlock "foo\n"
  ...

Failures in includeCode are thrown as exceptions in the IO action, with formatted strings for error messages.

includeCode :: Maybe Format -> Block -> IO Block

Testing directly against the formatted messages would be brittle, so we’ll instead provide an alternative function to includeCode that returns IO (Either InclusionError Block). The Maybe Format passed by Pandoc’s filter mechanism to includeCode is not used, so we won’t pass it to our new function.

includeCode' :: Block -> IO (Either InclusionError Block)

The existing function will use includeCode' and convert the Either value to an IO action, throwing exceptions just as before. The Kleisli arrow (>=>) is used to compose two monadic actions, analogous to the (.) operator for function composition.

-- | A Pandoc filter that includes code snippets from external files.
includeCode :: Maybe Format -> Block -> IO Block
includeCode _ = includeCode' >=> either printAndFail return

Now, let’s get those exceptions out of includeCode'. When we have a valid block to return, we do that using the Right constructor. Errors are returned by wrapping them in a Left.

includeCode' cb@(CodeBlock (id', classes, attrs) _) =
    ...
    Right Nothing -> return (Right cb)
    Left  err     -> return (Left err)
includeCode' x = return (Right x)

The case where we actually include code in the block will no longer use runInclusion', as we want the ExceptT from the regular runInclusion function. Thus, we remove runInclusion', and use runExceptT directly. We’ll also have to use runReaderT with the InclusionSpec value, and bind the resulting contents.

includeCode' cb@(CodeBlock (id', classes, attrs) _) =
  case parseInclusion (HM.fromList attrs) of
    Right (Just spec) -> runExceptT $ do
      contents <- runReaderT (runInclusion allSteps) spec
      return
        ( CodeBlock (id', classes, filterAttributes attrs)
                    (Text.unpack contents)
        )
    ...

I like using ExceptT in this local style, not exposing it to the caller, but making error handling inside the function body easier to manage.

Now that we have a new function and want to test for specific errors using their constructors, we need to extend the exports list. We add the new function, the InclusionError data type, the MissingRangePart data type, and the InclusionMode data type.

module Text.Pandoc.Filter.IncludeCode
  ( InclusionError (..)
  , MissingRangePart (..)
  , InclusionMode (..)
  , includeCode'
  , includeCode
  ) where

It’s time to fix the test suite. We’ll remove the qualified import of IncludeCode, and import everything but the old includeCode function.

import           Text.Pandoc.Filter.IncludeCode hiding (includeCode)

Our test helper will use includeCode' instead of includeCode, and return an action of type IO (Either InclusionError Block).

includeCode :: String -> [(String, String)] -> IO (Either InclusionError Block)
includeCode fixtureName attrs = includeCode'
  ( CodeBlock
    ("", [], mconcat [[("include", "test/fixtures/" ++ fixtureName)], attrs])
    ""
  )

All assertions now need to check for Left or Right results. We begin by expecting all cases to pass with Right values. A macro in Neovim makes this process a little less painful.

  it "includes snippet" $
    includeCode "foo-snippet.txt" [("snippet", "foo")]
    `shouldReturn` Right (codeBlock "foo\n")

  it "includes snippet within start and end line" $
    includeCode "foo-snippet.txt"
                  [("snippet", "foo"), ("startLine", "1"), ("endLine", "3")]
    `shouldReturn` Right (codeBlock "foo\n")

  it "includes snippet by exact name, not prefix" $
    includeCode "foo-snippets.txt" [("snippet", "foo")]
    `shouldReturn` Right (codeBlock "foo\n")

  it "excludes snippet outside start and end line" $
    includeCode "foo-snippet.txt"
                  [("snippet", "foo"), ("startLine", "2"), ("endLine", "3")]
    `shouldReturn` Right (codeBlock "")

  it "includes only lines between start and end line" $
    includeCode "some-text.txt" [("startLine", "2"), ("endLine", "3")]
    `shouldReturn` Right (codeBlock "world\nthis\n")

  it "dedents lines as much as specified" $
    includeCode "indents.txt" [("dedent", "1")]
    `shouldReturn` Right (codeBlock "zero\n two\none\n  three\n       eight\n")

  it "dedents lines as much as possible without removing non-whitespace" $
    includeCode "indents.txt" [("dedent", "8")]
    `shouldReturn` Right (codeBlock "zero\ntwo\none\nthree\neight\n")

Checking the test runner output, the two cases failing before are now showing the difference between the expected Right, and the actual Left.

/home/owi/haskell-at-work/pandoc-include-code/test/Driver.hs
  includeCode
    includes snippet:                                                  OK
    includes snippet within start and end line:                        FAIL
      expected Right (CodeBlock ("",[],[]) "foo\n"), but got Left (ConflictingModes [RangeMode (Range {rangeStart = 1, rangeEnd = 3}),SnippetMode "foo"])
    includes snippet by exact name, not prefix:                        OK
    excludes snippet outside start and end line:                       FAIL
      expected Right (CodeBlock ("",[],[]) ""), but got Left (ConflictingModes [RangeMode (Range {rangeStart = 1, rangeEnd = 3}),SnippetMode "foo"])
    includes only lines between start and end line:                    OK
    dedents lines as much as specified:                                OK
    dedents lines as much as possible without removing non-whitespace: OK

2 out of 7 tests failed (0.00s)

These cases should return Left values, so let’s change the first failing test and its description.

it "rejects snippet mode and range mode" $
  includeCode "foo-snippet.txt"
                [("snippet", "foo"), ("startLine", "1"), ("endLine", "3")]
  `shouldReturn` Left (ConflictingModes [RangeMode (range 1 3), SnippetMode "foo"])

We’ll define the range helper function for this test suite, to give us a Range value given a line and a column. Note that this function is partial, but we can live with that in our test code.

range l c = fromJust (mkRange l c)

We need to import mkRange and fromJust.

import           Data.Maybe                     (fromJust)
import           Text.Pandoc.Filter.Range       (mkRange)

We also need to enable OverloadedStrings to construct the Text value for the SnippetMode constructor.

{-# LANGUAGE OverloadedStrings #-}

Now we have only one failing test case left. Converting it to expect an error would result in the same test we just wrote, so we’ll just remove it.

it "excludes snippet outside start and end line" $
  includeCode "foo-snippet.txt"
                [("snippet", "foo"), ("startLine", "2"), ("endLine", "3")]
  `shouldReturn` Right (codeBlock "")

All tests pass, and we’re done!

/home/owi/haskell-at-work/pandoc-include-code/test/Driver.hs
  includeCode
    includes snippet:                                                  OK
    rejects snippet mode and range mode:                               OK
    includes snippet by exact name, not prefix:                        OK
    includes only lines between start and end line:                    OK
    dedents lines as much as specified:                                OK
    dedents lines as much as possible without removing non-whitespace: OK

All 6 tests passed (0.00s)

Remaining to be implemented in this Pandoc filter series is the .numberLines feature. Keep an eye out for the next video!