Package management updates in Zeek 4

Our latest Zeek release includes a number of improvements around zkg, Zeek’s package manager. They aim to make zkg more accessible and familiar to users of other package managers. In this blog post I’d like to walk you through these changes.

zkg is now a core component of Zeek: users installing Zeek from source will notice that the installed distribution now includes zkg by default. After installing Zeek into $PREFIX, zkg resides in $PREFIX/bin alongside the zeek binary itself, ready to use. If you don’t already have zkg’s required Python dependencies installed, zkg will tell you how to retrieve them. If your PATH environment includes both $PREFIX/bin and a pre-existing zkg installation directory, check which zkg version you’re now invoking by default!

The newly installed zkg comes with helpful pre-configuration. Its default configuration file resides in $PREFIX/etc/zkg/config, its package state in $PREFIX/var/lib/zkg. This means that for the first time zkg’s state ties closely to a specific Zeek installation, so parallel Zeek installations automatically separate their package state, preventing tricky compatibility problems that have come up in the past.  

Separate installation of zkg via pip remains possible, and we continue to upload new zkg releases to PyPI. For users who’d rather not bundle zkg with their Zeek installation, the configure script provides --disable-zkg.

If you prefer managing zkg packages from a user account other than the one owning the Zeek installation, maintaining zkg’s state separately remains possible. To simplify this approach, zkg now includes a user mode, enabled via the --user flag, that makes zkg keep all package state in $HOME/.zkg/. For example, zkg --user autoconfig  now automatically establishes a configuration in $HOME/.zkg/ that keeps all package and plugin state in that directory, and zkg --user env reports environment variables accordingly. Power users can also continue to compose their own state management via custom config files that define state_dir, script_dir, and plugin_dir entries as needed.

A few words on transitioning from existing zkg installations. You have two options:

• Switch to the new zkg and its defaults. There’s no automatic migration of your existing packages, so you’ll need to reinstall them. If you want to migrate your existing packages, we suggest you first grab a package bundle from the old installation via zkg bundle, then install it in the new one via zkg unbundle. If you want to be extra sure you start from a clean slate, you can delete $HOME/.zkg/.
• Keep the existing installation. Your old PyPI-installed zkg will continue to work with the existing state. The new, Zeek-bundled zkg can adopt the existing state in several ways: you can add --user to your zkg invocations as described above, explicitly say zkg --configfile=$HOME/.zkg/config, or set the ZKG_CONFIG_FILE environment variable to $HOME/.zkg/config. (As always, using packages across Zeek versions risks compatibility problems. Plugins, in particular, require reinstallation since Zeek 4 implies a version switch.) 

Since your previously installed zkg and the Zeek-bundled one can coexist, you’re free to experiment with both to figure out your preferred approach. If you’re concerned about your currently installed packages, we recommend first doing a backup via zkg bundle.

The Zeek-bundled zkg installation does not automatically load the package folder that zkg uses to manage package load state. Just as with older Zeek versions, you need to add it to your startup scripts (e.g. via local.zeek) if you’d like to use it.

Users of our binary packages benefit from the same immediate availability of zkg, which can prove handy for package testing and quick deployment to CI environments.

zkg has received other new features since the last Zeek release, so take a look at the CHANGES file for the details.

Bundling zkg with Zeek is a first step in a broader push to simplify and streamline package management with Zeek that will extend well into the 4.x release cycle. We welcome your feedback and feature suggestions, so please feel free to get in touch on the mailing lists or our Slack.