How to add or update opam dependencies

When a merge request (MR) introduces a new dependency to an opam package, or updates an existing dependency to an different version of an opam package, additional steps must be taken in the development and merge process. This document explains those steps.

If you have already read this guide and only need a refresher, skip to the TL;DR.

Background

The Octez project is built under a system that is somewhat stricter than the default for OCaml project. Specifically, the Octez project maintains a dedicated opam package repository that is a strict subset of the opam default one; all binaries are built with dependencies from this subset only.

For this reason, adding or updating a dependency requires to work both on the main codebase and on the dedicated opam package repository. Moreover, work between those two components must happen in a specific order.

The rest of this document explains the process from the point-of-view of a developer (you). The instructions below assume you have already set up your work environment but that you installed development dependencies (make build-dev-deps instead of make build-deps).

Local work

The simplest way of working locally (i.e., on your own machine) on the Octez codebase, using a new dependency is to install it using opam.

Because you have used make build-dev-deps in order to install the Octez dependencies, you have access to the default opam repository in addition to the dedicated one.

Install your dependency: opam install foo

Add the dependency to the relevant declarations in manifest/main.ml. And then use make -C manifest to update the opam and dune files accordingly.

For example, if you are modifying the Shell using the new dependency, you must add an entry in the ~deps list of the let octez_shell = entry of the manifest/main.ml and then run make -C manifest. You should see the changes propagated onto opam/tezos-shell.opam and src/lib_shell/dune.

Add dependencies to build files: both opam files and dune files must be updated.

You can work on your feature, using the types and values provided by your new dependency.

Making an MR

Even though you can compile and run code locally, the CI will fail. That is because the CI runs not in an environment based on the content of the opam/ directory, but instead based on pre-built Docker images built by the CI of the dedicated opam-repository.

You must follow the steps below in order to produce the necessary Docker images, allowing your work to eventually be merged.

First, in your local copy of Octez, update the full_opam_repository_tag variable in the scripts/version.sh file. You should set this variable to the hash of the HEAD commit on the default opam repository. (Note: this is not always necessary, but it is simpler for you to do so than to check whether it is necessary to do so.)

Second, still in your local copy of Octez, execute the scripts/update_opam_repo.sh script. This script uses the content of your opam/ directory to create a file called opam_repo.patch. This file represents the diff between the current dedicated opam repository and the dedicated opam repository that your MR needs.

Note that the diff may include a few more changes than what you strictly need. Specifically, it might include some updates of some other dependencies. This is not an issue in general but it might explain some changes unrelated to your work.

Third, create an MR on the dedicated opam repository that includes your patch. This is the opam repository MR, its role is to prepare the environment for your existing Octez MR.

In order to create the opam repository MR:

  • If you haven’t already done so, clone the dedicated opam repository.

  • Create a branch from the repository’s master and switch to it.

  • Apply the patch generated by scripts/update_opam_repo.sh (git apply <path-to-file>/opam_repo.path).

  • Commit the applied patch.

  • Push your branch.

  • Create the opam repository MR from this branch.

You can test the MR locally using the command OPAM_REPOSITORY_TAG=<commit-id> make build-deps. This will rebuild the dependencies locally using the <commit_id> of the opam-repository.

Fourth, back in your local copy of Octez, update the variables in the .gitlab-ci.yml and scripts/version.sh files. Specifically, set the build_deps_image_version and the opam_repository_tag variables to the hash of the HEAD commit of the opam repository MR. Commit this change with a title along the lines of “CI: use dependency foo”.

This commit will point the build scripts and CI to the modified opam-repository and the associated Docker images. Do note that the CI on your branch of Octez will only be able to run after the CI on your branch of opam-repository has completed.

Fifth, still in your local copy of Octez, push these changes and open or update the MR. Make sure you add links referencing the opam-repository MR from the Octez MR and vice-versa. This gives the reviewers the necessary context to review.

That’s it. You now have two MRs:

  • The opam-repository MR from tezos/opam-repository:<your-branch> onto tezos/opam-repository:master updates the environment in which the Octez libraries and binaries are built.

  • The Octez MR from <your-organisation>/tezos:<your-branch> onto tezos/tezos:master uses this new environment.

Merging the MR

This section is for the Octez merge team. It is the last step in the lifetime of the MRs you have opened. Understanding the basics of this process may help you when communicating with the reviewers and the mergers of your MR. Understanding all the minutiae and details is not necessary. For this reason, this final section is addressed to whichever member of the Octez merge team takes care of this MR (you).

After the iterative review-comment-edit process has reached a satisfying fixpoint, you can merge the two MRs opened by the developer. To avoid interference with other MRs, it is better to perform all the steps described below relatively quickly (the same day).

First, mention the MR on the #opam-repo Slack channel and make sure there isn’t another merge ongoing.

Second, merge the opam-repository MR. Make sure that this merge doesn’t introduce a merge commit. After the merge, the HEAD commit of master should be the HEAD of the branch you just merged. This is important because it ensures the Docker images have the same name.

Second, assign the Octez MR to margebot for merging.

TL;DR

As a developer:

  • You have an Octez MR from <your-organisation>/tezos:<your-branch> onto tezos/tezos:master introducing a dependency to foo.

  • You amend the manifest/main.ml file to declare the dependency.

  • You propagate the changes to opam and dune files by running make -C manifest

  • You update the full_opam_repository_tag to the HEAD commit hash from the public default opam repository.

  • You execute scripts/update_opam_repo.sh.

  • You open an opam repository MR from tezos/opam-repository:<your-branch> onto tezos/opam-repository:master that includes the generated patch.

  • You update build_deps_image_version and opam_repository_tag to the hash of the HEAD commit of your opam repository MR.

  • You push the changes to your Octez MR.

  • You update the description of your MRs to include links.

As a merger:

  • You test, review, etc. the code.

  • You merge the opam repository MR.

  • You make sure the commit hash has been preserved by merging (no squashing, no merge-commits)

  • You assign the Octez MR to margebot.