Alcotezt: An Alcotest Compatibility Wrapper for Tezt ==================================================== Alcotezt is a compatibility wrapper for tests originally written in :ref:`Alcotest ` (now deprecated), enabling running them with :doc:`Tezt `. Alcotezts (i.e. tests wrapped with Alcotezt) are programmed against an API that is compatible with Alcotest. But, under the hood, they are registered with Tezt. This means that they are executed in the same way as all Tezt tests are (see :ref:`running-alcotezts` below for more details). Alcotezts are typically declared through the ``tezt`` manifest function. This will: - Create a *test library* that registers the tests in the modules given as argument to ``tezt``. - Create a *test executable* ``main.exe`` in the test folder. The test executable links with the test library and consists of a single call to Tezt's `Test.run `__. - Finally, link the test library with Tezt's main entrypoint at ``tezt/tests/main.exe`` so that it registers all Alcotezts. .. _running-alcotezts: Running Alcotezts ----------------- For a given folder ``$TEST_DIR``, the Alcotezts contained therein can be invoked in three ways: 1. By executing the ``main.exe`` runner binary generated by Manifest. Through ``dune``, this is done with ``dune exec $TEST_DIR/main.exe``. This will execute the tests in ``$TEST_DIR`` (but not in its subdirectories). 2. Through Dune, by building the ``runtest`` alias in the test target folder. That is, executing ``dune build @$TEST_DIR/runtest``. This will execute the tests in ``$TEST_DIR`` and its subdirectories. 3. By executing the full Tezt test suite, via the main entrypoint at ``tezt/tests/main.exe``. Through Dune, this is done with ``dune exec tezt/tests/main.exe``. Execution through the test executable +++++++++++++++++++++++++++++++++++++ The advantage of passing through the Manifest-generated runner is that it only depends on the tests themselves and the tested modules (and their recursive dependencies), and so is faster to compile. Furthermore, you can give arguments to Tezt to control execution, e.g. passing ``--list`` to list the tests or ``--verbose`` to see debug output. For example, to run the tests of :src:`src/lib_clic/test`:: dune exec src/lib_clic/test/main.exe This will execute the Alcotezts in :src:`src/lib_clic/test`. To pass arguments to Tezt, append ``-- `` to the above command:: dune exec src/lib_clic/test/main.exe -- --list Execution through the Dune ``runtest`` alias ++++++++++++++++++++++++++++++++++++++++++++ The ``runtest`` alias can be used to execute all tests, including Alcotezts, in a folder and its recursive sub-folders. For example, to run all tests of protocol Alpha, run:: dune build @src/proto_alpha/runtest On the other hand, there is no convenient way to pass arguments to the underlying tests with this method. The Alcotezts can be dynamically disabled in the ``runtest`` alias by setting the environment variable ``RUNTEZTALIAS`` to ``false``. For instance, to run all the unit tests in protocol Alpha exception the Alcotezts, you can run:: RUNTEZTALIAS=false dune build @src/proto_alpha/runtest Execution through the Tezt main entrypoint ++++++++++++++++++++++++++++++++++++++++++ Finally, Alcotezts can be executed through the Tezt main entrypoint at ``tezt/tests/main.exe``. All Alcotezts are linked with this binary, so that all system, integration and unit tests registered with either Tezt or Alcotezt can be executed by invoking:: dune exec tezt/tests/main.exe This is used to run all tests in the CI in a unified and load-balanced manner. This is less convenient for local use though, as there is currently no way of executing the subset of tests that corresponds to a specific package, file or folder. A forthcoming release of Tezt will ameliorate this. Notable Differences Between Alcotest and Alcotezt ------------------------------------------------- Test naming +++++++++++ First of all, some definitions: - In Alcotest, a "test suite" is a sequence of "tests" that are each composed of a sequence of "test cases". - In Tezt, there are only "tests", no "test suites". With Alcotezt, each Alcotest "test case" becomes a Tezt "test". The Alcotest suite, test and case name are used to form the title of the Tezt test with following format: ``SUITE_NAME: TEST_NAME (TEST_CASE_NAME)``. Given an Alcotest consisting of a suite ``Suite A`` that contains a test ``Test a`` that itself contains the test cases ``Test case a1`` and ``Test case a2``. It also contains a test ``Test b`` with the test case ``Test case b1``: .. code:: ocaml let () = Alcotest.run "Suite A" [ ( "Test a", [ ("Test case a1", `Quick, fun () -> ...); ("Test case a2", `Quick, fun () -> ...); ] ); ( "Test b", [ ("Test case b1", `Quick, fun () -> ...); ] ); ] Running it in with Alcotest produces:: Testing `Suite A'. This run has ID `3F91T9S2'. [OK] Test a 0 Test case a1. [OK] Test a 1 Test case a2. [OK] Test b 0 Test case b1. Full test results in `/home/tezos/_build/_tests/Suite A'. Test Successful in 0.000s. 2 tests run. And running it with Alcotezt produces:: [17:07:42.289] [SUCCESS] (1/2) Suite A: Test a (Test case a1) [17:07:42.289] [SUCCESS] (2/3) Suite A: Test a (Test case a2) [17:07:42.290] [SUCCESS] (3/3) Suite A: Test b (Test case b1) Test Output +++++++++++ Alcotezt redirects ``Format``'s output to Tezt's `Log.debug `__. To see the debug output of an Alcotezt, pass the ``--verbose`` flag to Tezt. See the section :ref:`running-alcotezts` above for more information on how to pass flags to Tezt when executing Alcotezts. There is no way to redirect the output of ``Printf``. Consequently, the output of Alcotezts that call this module directly cannot be hidden. Integration with the ``runtest`` aliases ++++++++++++++++++++++++++++++++++++++++ Alcotezts are registered as a dependency on the ``runtest`` alias. However, they are not executed through this alias in the CI. Instead, they run through the Tezt main runner to enable load balancing.