How to add or update opam dependencies¶
When a merge request (MR) introduces a new dependency to an opam package, or updates an existing dependency to an different version of an opam package, additional steps must be taken in the development and merge process. This document explains those steps.
If you have already read this guide and only need a refresher, skip to the TL;DR.
The Octez project is built under a system that is somewhat stricter than the default for OCaml project. Specifically, the Octez project maintains a dedicated opam package repository that is a strict subset of the opam default one; all binaries are built with dependencies from this subset only.
For this reason, adding or updating a dependency requires to work both on the main codebase and on the dedicated opam package repository. Moreover, work between those two components must happen in a specific order.
The rest of this document explains the process from the point-of-view of
a developer (you). The instructions below assume you have already
set up your work environment
but that you installed development dependencies
make build-dev-deps instead of
The simplest way of working locally (i.e., on your own machine) on the
Octez codebase, using a new dependency is to install it using
Because you have used
make build-dev-deps in order to install the
Octez dependencies, you have access to the default opam repository in
addition to the dedicated one.
Install your dependency:
opam install foo
Add the dependency to the relevant declarations in manifest/main.ml. And
make -C manifest to update the opam and dune files accordingly.
For example, if you are modifying the Shell using the new
dependency, you must add an entry in the
~deps list of the
let octez_shell = entry of the manifest/main.ml and then run
make -C manifest. You should see the changes propagated onto
opam/tezos-shell.opam and src/lib_shell/dune.
Add dependencies to build files: both opam files and dune files must be updated.
You can work on your feature, using the types and values provided by your new dependency.
Making an MR¶
Even though you can compile and run code locally, the CI will fail. That is
because the CI runs not in an environment based on the content of the
opam/ directory, but instead based on pre-built Docker images built by
the CI of the dedicated
You must follow the steps below in order to produce the necessary Docker images, allowing your work to eventually be merged.
First, in your local copy of Octez, update the
full_opam_repository_tag variable in the scripts/version.sh
should set this variable to the hash of the
HEAD commit on
the default opam repository.
(Note: this is not always necessary, but it is simpler for you to do so
than to check whether it is necessary to do so.)
Second, still in your local copy of Octez, execute the
scripts/update_opam_repo.sh script. This script uses the content of
your opam/ directory to create a file
opam_repo.patch. This file represents the diff between the current
dedicated opam repository and the dedicated opam repository that your MR
Note that the diff may include a few more changes than what you strictly need. Specifically, it might include some updates of some other dependencies. This is not an issue in general but it might explain some changes unrelated to your work.
Third, create an MR on the dedicated opam repository that includes your patch. This is the opam repository MR, its role is to prepare the environment for your existing Octez MR.
In order to create the opam repository MR:
If you haven’t already done so, clone the dedicated opam repository.
Create a branch from the repository’s
masterand switch to it.
Apply the patch generated by scripts/update_opam_repo.sh (
git apply <path-to-file>/opam_repo.path).
Commit the applied patch.
Push your branch.
Create the opam repository MR from this branch.
You can test the MR locally using the command
OPAM_REPOSITORY_TAG=<commit-id> make build-deps. This will rebuild the
dependencies locally using the
<commit_id> of the opam-repository.
Fourth, back in your local copy of Octez, update the variables in the
.gitlab-ci.yml and scripts/version.sh files. Specifically, set
build_deps_image_version and the
to the hash of the
HEAD commit of the opam repository MR. Commit
this change with a title along the lines of “CI: use dependency
This commit will point the build scripts and CI to the modified opam-repository and the associated Docker images. Do note that the CI on your branch of Octez will only be able to run after the CI on your branch of opam-repository has completed.
Fifth, still in your local copy of Octez, push these changes and open or update the MR. Make sure you add links referencing the opam-repository MR from the Octez MR and vice-versa. This gives the reviewers the necessary context to review.
That’s it. You now have two MRs:
The opam-repository MR from
tezos/opam-repository:masterupdates the environment in which the Octez libraries and binaries are built.
The Octez MR from
tezos/tezos:masteruses this new environment.
Merging the MR¶
This section is for the Octez merge team. It is the last step in the lifetime of the MRs you have opened. Understanding the basics of this process may help you when communicating with the reviewers and the mergers of your MR. Understanding all the minutiae and details is not necessary. For this reason, this final section is addressed to whichever member of the Octez merge team takes care of this MR (you).
After the iterative review-comment-edit process has reached a satisfying fixpoint, you can merge the two MRs opened by the developer. To avoid interference with other MRs, it is better to perform all the steps described below relatively quickly (the same day).
First, mention the MR on the
#opam-repo Slack channel and make sure
there isn’t another merge ongoing.
Second, merge the opam-repository MR. Make sure that this merge doesn’t introduce a merge commit. After the merge, the HEAD commit of master should be the HEAD of the branch you just merged. This is important because it ensures the Docker images have the same name.
Second, assign the Octez MR to margebot for merging.
As a developer:
You have an Octez MR from
tezos/tezos:masterintroducing a dependency to
You amend the manifest/main.ml file to declare the dependency.
You propagate the changes to
dunefiles by running
make -C manifest
You update the
full_opam_repository_tagto the HEAD commit hash from the public default opam repository.
You execute scripts/update_opam_repo.sh.
You open an opam repository MR from
tezos/opam-repository:masterthat includes the generated patch.
opam_repository_tagto the hash of the
HEADcommit of your opam repository MR.
You push the changes to your Octez MR.
You update the description of your MRs to include links.
As a merger:
You test, review, etc. the code.
You merge the opam repository MR.
You make sure the commit hash has been preserved by merging (no squashing, no merge-commits)
You assign the Octez MR to margebot.