criterion-1.1.4.0: Robust, reliable performance measurement and analysis

Copyright | (c) 2009-2014 Bryan O'Sullivan
License | BSD-style
Maintainer | bos@serpentine.com
Stability | experimental
Portability | GHC
Safe Haskell | None
Language | Haskell98

Criterion

Contents

- Benchmarkable code
- Creating a benchmark suite
  - Running a benchmark
- For interactive use

Description

Core benchmarking code.

Synopsis

- data Benchmarkable
- data Benchmark
- env :: NFData env => IO env -> (env -> Benchmark) -> Benchmark
- bench :: String -> Benchmarkable -> Benchmark
- bgroup :: String -> [Benchmark] -> Benchmark
- nf :: NFData b => (a -> b) -> a -> Benchmarkable
- whnf :: (a -> b) -> a -> Benchmarkable
- nfIO :: NFData a => IO a -> Benchmarkable
- whnfIO :: IO a -> Benchmarkable
- benchmark :: Benchmarkable -> IO ()
- benchmarkWith :: Config -> Benchmarkable -> IO ()
- benchmark' :: Benchmarkable -> IO Report
- benchmarkWith' :: Config -> Benchmarkable -> IO Report

# Benchmarkable code

data Benchmarkable #

A pure function or impure action that can be benchmarked. The `Int64` parameter indicates the number of times to run the given function or action.

# Creating a benchmark suite

data Benchmark #

Specification of a collection of benchmarks and environments. A benchmark may consist of:

- An environment that creates input data for benchmarks, created with `env`.
- A single `Benchmarkable` item with a name, created with `bench`.
- A (possibly nested) group of `Benchmark`s, created with `bgroup`.

Instances

Show Benchmark # |
Methods showsPrec :: Int -> Benchmark -> ShowS # show :: Benchmark -> String # showList :: [Benchmark] -> ShowS #

env #

Arguments

:: NFData env |
=> IO env | Create the environment. The environment will be evaluated to normal form before being passed to the benchmark.
-> (env -> Benchmark) | Take the newly created environment and make it available to the given benchmarks.
-> Benchmark |

Run a benchmark (or collection of benchmarks) in the given environment. The purpose of an environment is to lazily create input data to pass to the functions that will be benchmarked.

A common example of environment data is input that is read from a file. Another is a large data structure constructed in-place.

Motivation. In earlier versions of criterion, all benchmark inputs were always created when a program started running. By deferring the creation of an environment when its associated benchmarks need the its, we avoid two problems that this strategy caused:

- Memory pressure distorted the results of unrelated benchmarks. If one benchmark needed e.g. a gigabyte-sized input, it would force the garbage collector to do extra work when running some other benchmark that had no use for that input. Since the data created by an environment is only available when it is in scope, it should be garbage collected before other benchmarks are run.
- The time cost of generating all needed inputs could be significant in cases where no inputs (or just a few) were really needed. This occurred often, for instance when just one out of a large suite of benchmarks was run, or when a user would list the collection of benchmarks without running any.

Creation. An environment is created right before its related benchmarks are run. The `IO` action that creates the environment is run, then the newly created environment is evaluated to normal form (hence the `NFData` constraint) before being passed to the function that receives the environment.

Complex environments. If you need to create an environment that contains multiple values, simply pack the values into a tuple.

Lazy pattern matching. In situations where a "real" environment is not needed, e.g. if a list of benchmark names is being generated, `undefined` will be passed to the function that receives the environment. This avoids the overhead of generating an environment that will not actually be used.

The function that receives the environment must use lazy pattern matching to deconstruct the tuple, as use of strict pattern matching will cause a crash if `undefined` is passed in.

Example. This program runs benchmarks in an environment that contains two values. The first value is the contents of a text file; the second is a string. Pay attention to the use of a lazy pattern to deconstruct the tuple in the function that returns the benchmarks to be run.

    setupEnv = do
      let small = replicate 1000 (1 :: Int)
      big <- map length . words <$> readFile "/usr/dict/words"
      return (small, big)

    main = defaultMain [
       -- notice the lazy pattern match here!
       env setupEnv $ \ ~(small,big) -> bgroup "main" [
       bgroup "small" [
         bench "length" $ whnf length small
       , bench "length . filter" $ whnf (length . filter (==1)) small
       ]
     ,  bgroup "big" [
         bench "length" $ whnf length big
       , bench "length . filter" $ whnf (length . filter (==1)) big
       ]
     ] ]

Discussion. The environment created in the example above is intentionally not ideal. As Haskell's scoping rules suggest, the variable `big` is in scope for the benchmarks that use only `small`. It would be better to create a separate environment for `big`, so that it will not be kept alive while the unrelated benchmarks are being run.

bench #

Arguments

:: String | A name to identify the benchmark.
-> Benchmarkable | An activity to be benchmarked.
-> Benchmark |

Create a single benchmark.

bgroup #

Arguments

:: String | A name to identify the group of benchmarks.
-> [Benchmark] | Benchmarks to group under this name.
-> Benchmark |

Group several benchmarks together under a common name.

## Running a benchmark

nf :: NFData b => (a -> b) -> a -> Benchmarkable #

Apply an argument to a function, and evaluate the result to head normal form (NF).

whnf :: (a -> b) -> a -> Benchmarkable #

Apply an argument to a function, and evaluate the result to weak head normal form (WHNF).

nfIO :: NFData a => IO a -> Benchmarkable #

Perform an action, then evaluate its result to head normal form. This is particularly useful for forcing a lazy `IO` action to be completely performed.

whnfIO :: IO a -> Benchmarkable #

Perform an action, then evaluate its result to weak head normal form (WHNF). This is useful for forcing an `IO` action whose result is an expression to be evaluated down to a more useful value.

# For interactive use

benchmark :: Benchmarkable -> IO () #

Run a benchmark interactively, and analyse its performance.

benchmarkWith :: Config -> Benchmarkable -> IO () #

Run a benchmark interactively, and analyse its performance.

benchmark' :: Benchmarkable -> IO Report #

Run a benchmark interactively, analyse its performance, and return the analysis.

benchmarkWith' :: Config -> Benchmarkable -> IO Report #

Run a benchmark interactively, analyse its performance, and return the analysis.