JS monorepos in prod 2: project versioning and publishing

One great advantage of a monorepo is to maintain coherent versions between packages and to automatize the version creation and the publication of packages. This article covers the versioning and publishing strategies and best practices before continuing with commit enforcement, unit tests and CI/CD integration in the following articles:

Versioning and publishing strategies

Remember the usage of the --independent flag in the lerna init command. It tells Lerna about our versioning strategy. In our case, the packages are all Gatsby plugins, and it makes sense to group them inside a single Git repository. They are however independent packages, with their own release cycle and maturity. Thus, we don’t wish to share a single version for all of them.

It is a time to use Lerna to manage the versioning. Running the lerna version command does the few things:

• Prompts the user a choice of version for each package since those are managed independently.
• Saves the version inside the package.json files.
• Creates a tag with the package name and the version.
• Commits the changes and push them to the remote server.

But first, Lerna can customize the commit message. In the lerna.json configuration file, add the entry:

  ...
  "command": 
    ...
    "version": 
      ...
      "message": "chore(release): publish"
    
  

And commit the change:

git commit -a -m 'build: customize lerna versioning message'

We can run lerna version to release an initial version for our two packages:

lerna version
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Assuming all packages changed
? Select a new version for gatsby-remark-title-to-frontmatter (currently 0.0.0) Premajor (1.0.0-alpha.0)
? Select a new version for gatsby-caddy-redirects-conf (currently 0.0.0)
  Patch (0.0.1)
  Minor (0.1.0)
  Major (1.0.0)
  Prepatch (0.0.1-alpha.0)
❯ Preminor (0.1.0-alpha.0)
  Premajor (1.0.0-alpha.0)
  Custom Prerelease
  Custom Version 

Lerna present us a list of possible incremental versions, including patch, minor and major version as well as prerelease numbers and custom release such as alpha, beta and rc. We are free to choose a different value for each package. Once it is ready, Lerna asks for a final confirmation:

lerna version
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Assuming all packages changed
? Select a new version for gatsby-remark-title-to-frontmatter (currently 0.0.0) Premajor (1.0.0-alpha.0)
? Select a new version for gatsby-caddy-redirects-conf (currently 0.0.0) Preminor (0.1.0-alpha.0)
Changes:
 - gatsby-remark-title-to-frontmatter: 0.0.0 => 1.0.0-alpha.0
 - gatsby-caddy-redirects-conf: 0.0.0 => 0.1.0-alpha.0
? Are you sure you want to create these versions? Yes
lerna info execute Skipping releases
lerna info git Pushing tags...
lerna success version finished

The gatsby-remark-title-to-frontmatter package is a major alpha release preparing version 1.0.0. The gatsby-caddy-redirects-conf package is a minor alpha release preparing version 0.1.0.

Private and local NPM registry

The source code is now available on GitHub and the commit is tagged with our version. The tags are [email protected] and [email protected]. The tags reflect the package names and versions.

Our two packages are not yet available to the community. For other users to use them from within their package.json as dependencies, they need to be published on an NPM registry. Lerna provide the command lerna publish for this. Before describing how lerna publish works, a local NPM registry is setup for the sake of testing.

Hosting your private NPM registry has multiple advantages. It provides guarantees that no external parties have access to your packages. It speeds up the downloading process when hosted inside your network. It is free assuming you use an open source registry project and that you have the machine to host it.

For the sake of testing and because NPM publications are not revocable after 48 hours, a local NPM registry is used as an alternative to the official NPM registry. Verdaccio is a simple registry that can be started on your host machine. Skip this step if you wish to publish your packages directly on the official NPM registry. To install it, you can use Docker, Kubernetes or NPM. For example, using NPM:

npm install -g verdaccio
verdaccio

I have personally used minikube and Helm:

minikube start
😄  minikube v1.14.0 on Darwin 10.15.7
✨  Using the hyperkit driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
🐳  Preparing Kubernetes v1.19.2 on Docker 19.03.8 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" by default
helm repo add verdaccio https://charts.verdaccio.org
"verdaccio" has been added to your repositories
helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "verdaccio" chart repository
Update Complete. ⎈Happy Helming!⎈
helm install npm verdaccio/verdaccio
NAME: npm
LAST DEPLOYED: Thu Dec  3 11:03:33 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace default -l "app=verdaccio,release=npm" -o jsonpath=".items[0].metadata.name")
  kubectl port-forward --namespace default $POD_NAME 8080:4873
  echo "Visit http://127.0.0.1:8080 to use your application"
export POD_NAME=$(kubectl get pods --namespace default -l "app=verdaccio,release=npm" -o jsonpath=".items[0].metadata.name")
kubectl port-forward --namespace default $POD_NAME 4873:4873

In both cases, you can open your web browser and navigate to http://127.0.0.1:4873. The welcome screen asks you to create an account:

npm adduser --registry http://localhost:4873
Username: david
Password:
Email: (this IS public) [email protected]
Logged in as david on http://localhost:4873/.

All future lerna publish commands can refer to this registry by inserting the --registry flag. Remove this flag to publish your packages on the public and official NPM registry.

To avoid passing the --registry flag on every lerna publish, the lerna.json can store its location:

  "command": 
    "publish": 
      "registry": "http://localhost:4873/"
    
  

Storing the registry address in lerna.json implies that it will be committed to Git and shared with your collaborators.

Alternatively, Verdaccio can be globally defined as your default registry inside your ~/.npmrc file:

npm set registry http://localhost:4873
cat ~/.npmrc | grep registry=
registry=http://localhost:4873/

NPM publication

It is a time to publish our packages on the NPM registry of our choice with lerna publish.

There are two interesting strategies allowing Lerna to know which version it shall publish. The first one is to publish packages in the latest commit where the version is not present in the registry with the from-package argument.

lerna publish from-package
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna WARN Unable to determine published version, assuming "gatsby-remark-title-to-frontmatter" unpublished.
lerna WARN Unable to determine published version, assuming "gatsby-caddy-redirects-conf" unpublished.
Found 2 packages to publish:
 - gatsby-remark-title-to-frontmatter => 1.0.0-alpha.0
 - gatsby-caddy-redirects-conf => 0.1.0-alpha.0
? Are you sure you want to publish these packages? No

The second one is to look for the tag in the current commit with the from-git argument. In our case, both have the same consequences.

lerna publish from-git
info cli using local version of lerna
lerna info versioning independent
Found 2 packages to publish:
 - gatsby-caddy-redirects-conf => 0.1.0-alpha.0
 - gatsby-remark-title-to-frontmatter => 1.0.0-alpha.0
? Are you sure you want to publish these packages? Yes
lerna info publish Publishing packages to npm...
lerna WARN ENOLICENSE Packages gatsby-caddy-redirects-conf and gatsby-remark-title-to-frontmatter are missing a license.
lerna WARN ENOLICENSE One way to fix this is to add a LICENSE.md file to the root of this repository.
lerna WARN ENOLICENSE See https://choosealicense.com for additional guidance.
lerna success published gatsby-remark-title-to-frontmatter 1.0.0-alpha.0
lerna http fetch PUT 201 http://localhost:4873/gatsby-remark-title-to-frontmatter 824ms
lerna success published gatsby-caddy-redirects-conf 0.1.0-alpha.0
lerna http fetch PUT 201 http://localhost:4873/gatsby-caddy-redirects-conf 873ms
Successfully published:
 - [email protected]
 - [email protected]
lerna success published 2 packages

For the sake of clarity, the notice output logs are filtered.

Package content filtering

Not every file needs to be published. Content which is not stored in Git shall probably not be published. Secret, certificates, private configuration are such examples. By default, NPM and Yarn look at your .gitignore files and import its directives unless there is a .npmignore.

But being stored and shared in Git doesn’t necessarily mean it shall be publish in NPM. Your packages must be as small as possible. For example, there is no need to publish your tests.

You could copy the .gitignore file at the root of your packages into a .npmignore file and start adding new rules. There is however, to my opinion, a much better approach. Use the files property from your package.json file. From the package.json documentation:

The optional files field is an array of file patterns that describes the entries to be included when your package is installed as a dependency. File patterns follow a similar syntax to .gitignore, but reversed: including a file, directory, or glob pattern (*, **/*, and such) will make it so that file is included in the tarball when it’s packed. Omitting the field will make it default to ["*"], which means it will include all files.

However, certain files will be included such as the README, CHANGELOG and LICENSE files.

The gatsby-remark/title-to-frontmatter/package.json and gatsby/caddy-redirects-conf/package.json files are enriched with:

Then, the changes are committed:

git commit -a -m 'build: include lib in published packages'

If we were making changes to a package with a doc folder and its associated content, it would be committed to Git but not published.

Selective package publication

Notice how Lerna is complaining about our packages which does not provide a license file. It also proposes us to place a LICENSE.md file at the root of the repository. Let do just that and release a new version.

curl \
  https://raw.githubusercontent.com/adaltas/node-csv/master/LICENSE \
  -o LICENSE.md
git add LICENSE.md
git commit -m 'build: MIT license'
lerna publish from-git
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna notice from-git No tagged release found
lerna success No changed packages to publish

This time, running lerna publish has no effect even if there are some changes since the last commit. This is because the commit didn’t affect any of the published packages defined as Yarn workspaces. This is also because the root package is not published, since it is marked as private inside the package.json file:

cat package.json | grep private
  "private": true,

Selective package publication based on content

The lerna publish command only publish packages with changes. It can go a step further by not taking into account changes in some selected files.

The command.publish.ignoreChanges is an array of globs that won’t be included in lerna changed/publish.

It can be globally defined in your lerna.json configuration file, for example to not publish new release if the changes where only in tests:

  ...
  "ignoreChanges": [
    "**/test/**"
  ]

We need to commit the changes and create new versions before testing it:

git commit -a -m "build: lerna version ignore test"
lerna version
...
Changes:
 - gatsby-remark-title-to-frontmatter: 1.0.0-alpha.0 => 1.0.0-alpha.1
 - gatsby-caddy-redirects-conf: 0.1.0-alpha.0 => 0.1.0-alpha.1
? Are you sure you want to create these versions? Yes
lerna info execute Skipping releases
lerna info git Pushing tags...
lerna success version finished

Once a new test is committed in one of the packages, no version is proposed:

echo 'console.warn("TODO")' >  gatsby-remark/title-to-frontmatter/test/index.js
git add gatsby-remark/title-to-frontmatter/test/index.js
git commit -m 'test(title-to-frontmatter): todo tests'
lerna version
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Looking for changed packages since [email protected]
lerna info ignoring diff in paths matching [ '**/test/**' ]
lerna success No changed packages to version 

Files modification can be filtered from lerna version and lerna change by appling the --ignore-changes flag or modifying the lerna.json file with the ignoreChanges property. For example, to discard a new version when only a typo was made to a README file.

Automatic incremental versions

Let’s make some change to one of our package. The gatsby-caddy-redirects-conf package, currently at version 0.1.0-alpha.1, deserves a README.

echo \
 '# Package `gatsby-caddy-redirects-conf`' \
  > gatsby/caddy-redirects-conf/README.md
git add gatsby/caddy-redirects-conf/README.md
git commit -m "docs(title-to-frontmatter): new readme file"

Instead of having Lerna asking us which version to set, we tell Lerna to automatically increment the current prerelease version:

lerna version prerelease
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Looking for changed packages since [email protected]
lerna info ignoring diff in paths matching [ '**/test/**' ]
Changes:
 - gatsby-caddy-redirects-conf: 0.1.0-alpha.1 => 0.1.0-alpha.2
? Are you sure you want to create these versions? Yes
lerna info execute Skipping releases
lerna info git Pushing tags...
lerna success version finished

Since we indicate which version we wish with the prerelease SemVer keyword, Lerna doesn’t need to ask us the targeted version and, instead, only asks to validate its suggestion.

Note, to graduate from a prerelease cycle with the Conventional Commit, the command looks like lerna version --conventional-commits --conventional-graduate. We cover the Conventional Commit in the follow up article.

CI/CD, separation between versioning and publishing

If you try to run the lerna publish command without executing lerna version before, you will notice that it first creates a version of the package and then publishes the package. Why did we call lerna version if it is optional?

Versioning is separated from publishing because I don’t execute the two commands in the same location. For example, I can run lerna version manually, when I feel confident that my package deserves a new release, and automate lerna publish remotely on a CI/CD platform, when all the checks and tests successfully pass and once a new version has been detected.

Cheat sheet

• Custom commit message from Lerna
  Modify lerna.json:

  
    ...
    "command": 
      ...
      "version": 
        ...
        "message": "chore(release): publish"
      
    
  

  Commit the change:

  git commit -a -m 'build: customize lerna versioning message'

• Create a new version:
  Automatic increment the prerelease version:
  Unless the modified files match a pattern:

  lerna version --ignore-changes '**/*.md' '**/__tests__/**'

• Plushing a new version
  Or conditionnaly if there is a release tag:
• Using a custom repository
  Add the registry flag:

  lerna publish --registry http://localhost:4873/

  Or edit lerna.json:

  
    "command": 
      "publish": 
        "registry": "http://localhost:4873/"
      
    
  

  Or edit ~/.npmrc:

  npm set registry http://localhost:4873

• Content filtering
  In ‘package.json’:

Conclusion

The Lerna version and publish commands come with many arguments which are worth to investigate. We will cover next how to enforce the commit message format, how to run unit tests and how to automate the publication of packages in a CI/CD environment.