Ppx_expect¶

Ppx_expect is a framework for writing tests for OCaml code printing textual output.

There is pretty comprehensive documentation about Inline expectation tests and the ppx_expect tool for writing them:

Here, we will just cover enough to get started using them inside the Octez codebase.

How to run tests¶

If you want to run tests and see whether they succeed, build the standard runtest alias:

dune runtest src/lib_stdlib
# or dune build @@src/lib_stdlib/runtest

The result would look like this:

File "src/lib_stdlib/bloomer.ml", line 97, characters 0-817: random_read_writes (0.006 sec)
File "src/lib_stdlib/bloomer.ml", line 123, characters 0-1339: peek and poke work with bits = [1 .. Sys.int_size - 7] (0.012 sec)
File "src/lib_stdlib/bloomer.ml", line 163, characters 0-739: sequential_read_writes (0.021 sec)
File "src/lib_stdlib/bloomer.ml", line 182, characters 0-705: read_over_write (0.001 sec)
File "src/lib_stdlib/bloomer.ml", line 282, characters 0-835: consistent_add_mem_countdown (0.275 sec)
File "src/lib_stdlib/bloomer.ml", line 309, characters 0-1531: consistent_add_countdown_count (2.079 sec)
File "src/lib_stdlib/bloomer.ml", line 456, characters 4-137 (11.966 sec)
File "src/lib_stdlib/bloomer.ml", line 461, characters 4-1695: <<match Sys.getenv_opt "BLOOMER_TEST_GNUPLOT_PA[...]>> (0.000 sec)

A test can fail because it generates an output different from what is expected or because it raises an exception. Failures are reported as a diff in both cases.

Different output:

File "src/proto_alpha/lib_client/limit.ml", line 1, characters 0-0:
------ src/proto_alpha/lib_client/limit.ml
++++++ src/proto_alpha/lib_client/limit.ml.corrected
File "src/proto_alpha/lib_client/limit.ml", line 57, characters 0-1:
 |      if eq x y then Result.return_some x
 |      else error_with "Limit.join: error (%s)" where
 |
 |let%expect_test "join" =
 |  let pp_print_err fmt = function
 |    | Result.Error _ -> Format.pp_print_string fmt "error"
 |    | Ok x ->
 |        Format.(
 |          pp_print_option
 |            ~none:(fun fmt () -> pp_print_string fmt "None")
 |            pp_print_bool)
 |          fmt
 |          x
 |  in
 |  let print x = Format.fprintf Format.std_formatter "%a" pp_print_err x in
 |  print (join ~where:__LOC__ Bool.equal (Some true) (Some true)) ;
-|  [%expect {| false |}] ;
+|  [%expect {| true |}] ;
 |  print (join ~where:__LOC__ Bool.equal None None) ;
 |  [%expect {| None |}] ;
 |  print (join ~where:__LOC__ Bool.equal None (Some true)) ;
 |  [%expect {| true |}] ;
 |  print (join ~where:__LOC__ Bool.equal (Some true) None) ;
 |  [%expect {| true |}] ;
 |  print (join ~where:__LOC__ Bool.equal (Some true) (Some false)) ;
 |  [%expect {| error |}]
 |
 |let get ~when_unknown = function
 |  | None -> error_with "Limit.get: %s" when_unknown
 |  | Some x -> ok x
 |
 |let%expect_test "get" =
 |  let pp_print_err fmt = function
 |    | Result.Error _ -> Format.fprintf fmt "error"

Exception raised:

File "src/lib_stdlib/bloomer.ml", line 1, characters 0-0:
------ src/lib_stdlib/bloomer.ml
++++++ src/lib_stdlib/bloomer.ml.corrected
File "src/lib_stdlib/bloomer.ml", line 309, characters 0-1:
 |    let bloomer = create ~hash ~index_bits ~hashes ~countdown_bits in
 |    let rec init n acc =
 |      if n = 0 then acc
 |      else
 |        let x = Random.int (1 lsl 29) in
 |        add bloomer x ;
 |        assert (mem bloomer x) ;
 |        init (n - 1) (x :: acc)
 |    in
 |    let all = init 1000 [] in
 |    for _ = 0 to (1 lsl countdown_bits) - 2 do
 |      List.iter (fun x -> assert (mem bloomer x)) all ;
 |      countdown bloomer
 |    done ;
 |    List.iter (fun x -> assert (not (mem bloomer x))) all
 |  done
+|[@@expect.uncaught_exn {|
+|  (* CR expect_test_collector: This test expectation appears to contain a backtrace.
+|     This is strongly discouraged as backtraces are fragile.
+|     Please change this test to not include a backtrace. *)
+|
+|  "Assert_failure src/lib_stdlib/bloomer.ml:287:4"
+|  Raised at Tezos_stdlib__Bloomer.(fun) in file "src/lib_stdlib/bloomer.ml", line 287, characters 4-18
+|  Called from Expect_test_collector.Make.Instance_io.exec in file "collector/expect_test_collector.ml", line 262, characters 12-19 |}]

How to add tests¶

Ppx_expect is based on ppx_inline_test machinery. That is, it collects tests automatically for you.

To add a new test, make sure you have the following in your library stanza in the dune file:

(inline_tests)
(preprocess (pps ppx_expect))

In the manifest, just add the following argument to the library containing expect tests:

~inline_tests:ppx_expect

Adding new tests is then just a matter of adding let%expect_test at top level:

let%expect_test "optional name" =
  print_endline "hello world";
  [%expect {||}]

Running test with the example above will fail and show you a diff between your source and the corrected one:

 |let%expect_test "optional name" =
 |  print_endline "hello world";
-|  [%expect {||}]
+|  [%expect {| hello world |}]

If you agree with the diff, just ask dune to promote the source and you’re done:

dune runtest --auto-promote

Where to put the tests¶

Expect tests can live next to the implementation or in a different library dedicated to tests (e.g. if you don’t want to polute your source/binary or if your want to only test the exposed API).

Integration with Lwt¶

Ppx_expect can be used in combination with Lwt, see the package description. This integration has not been tested on the Octez codebase yet, hence some work will be needed to a have specific support for the codebase.