Zeek 3.0.0 RC1 released

• New analyzers for NTP and MQTT, and extended analyzers for DNS (SPF/DNSSEC), RDP, SMB, and TLS.

• Support for decapsulating VXLAN tunnels.

• Support for logging in UTF8.

• Several extensions of the scripting language:

• Closures for anonymous functions

• • Iteration over key/value pairs of a table through for ( key, value in t ) …)

• • Python-style vector slicing (v[2:4])

• • A new data structure, paraglob, for efficiently matching strings against large list of globs.

• See the NEWS file for more detailed release notes, and CHANGES for the complete list of changes

Upgrading to Zeek 3

The following summarizes the main naming-related changes that you will encounter after installing Zeek 3.0.0. Unless otherwise noted, the Bro 2.6 names and paths will continue to work with this release, but often trigger deprecation warnings.

• The names of all executables that had “bro” in their name have changed: bro -> zeek, bro-config -> zeek-config, broctl -> zeekctl, bro-cut -> zeek-cut. Zeek 3.0.0 installs wrappers under the old names that will let them continue to work.
• The default install prefix is now /usr/local/zeek instead of /usr/local/bro. If your existing installation used the previous default and you are using the new default when upgrading, we’ll symlink /usr/local/zeek to /usr/local/bro. Certain subdirectories get similar treatment: share/bro, include/bro, and lib/bro.
• Along with BroControl becoming ZeekControl, installation directories and files with broctl in their name have changed to use zeekctl instead. However, these changes remain backwards compatible with previous Bro installations by continuing to pull from existing locations where customizations might have been made. For example, if you have a broctl.cfg file from a previous installation, installing Zeek over it will retain that file and even symlink the new zeekctl.cfg to it.
• The new extension for Zeek scripts is .zeek. This leads to two major changes:

• All scripts ending in .bro have been renamed to .zeek. In particular, $prefix/share/bro/site/local.bro has been renamed to local.zeek. However, if you have an existing local.bro file from a previous Bro installation—possibly with customizations made to it—Zeek will install a symlink local.zeek file that points to that pre-existing local.bro. In that case, you may want to just copy local.bro into the new local.zeek location to avoid confusion, but things should generally also work properly without intervention.
• The search logic for the @load script directive now prefers files ending in .zeek, but will still fallback to loading a .bro file if it exists. E.g. @load foo will first check for a foo.zeek file to load and then otherwise foo.bro. Note that @load foo.bro (with the explicit .bro file suffix) prefers the opposite order: it first checks for foo.bro and then falls back to a foo.zeek, if that exists.

• Changes affecting scripts:
• The events bro_init, bro_done, and bro_script_loaded are now deprecated; use zeek_init, zeek_done, and zeek_script_loaded instead. Any existing event handlers for the deprecated versions will automatically alias to the new events such that existing code will not break, but their usage will emit deprecation warnings.
• The functions bro_is_terminating and bro_version function are deprecated and replaced by functions named zeek_is_terminating and zeek_version. The old names likewise continue to work with deprecation warnings.

• The namespace used by all the builtin plugins that ship with Zeek have changed to use Zeek::.
• Any Broker topic names used in scripts shipped with Zeek that previously were prefixed with bro/ are now prefixed with zeek/ instead. In the case where external applications were using a bro/ topic to send data into a Bro process, a Zeek process still subscribes to those topics in addition to the equivalently named zeek/ topic. In the case where external applications were using a bro/ topic to subscribe to remote messages or query data stores, there’s no backwards compatibility and external applications must be changed to use the new zeek/ topic. The NEWS have a list of the most common topic names that one may need to change.
• The Broxygen component, which is used to generate our Doxygen-like scripting API documentation, has been renamed to Zeekygen. This likely has no breaking or visible changes for most users, except in the case one used it to generate their own documentation via the –broxygen flag, which is now named –zeekygen. Besides that, various documentation in scripts has also been updated to replace Sphinx cross-referencing roles and directives like :bro:see: with :zeek:see:.

 

Upgrading to the Zeek Package Manager

The external package manager switched its name as well, from bro-pkg to zkg. On PyPI, both the old bro-pkg and new zkg packages share the same code-base, so you may continue using bro-pkg if you want, but it’s easy enough to switch for sake of consistency: run pip uninstall bro-pkg && pip install zkg.  Either way, a wrapper script is provided that forwards from bro-pkg to zkg. 

 

Renaming External Packages  

It’s up to a package’s maintainer whether they want to rename a package that’s been using “bro” in its name—there’s nothing about such a package name that will be incompatible with Zeek 3.0.0. If you do want to rename your package, we recommend the following process, assuming it’s hosted on GitHub:

• Rename your GitHub repository from bro-foo to zeek-foo. GitHub will automatically provide a redirect from the old URL to the new URL, so people who had installed a package using the old URL will still be fine going forward.
• Add an alias to the package’s metadata: aliases = zeek-foo bro-foo. This tells zkg that old and new names are referring to the same package, and it will create corresponding symlinks so that explicit @load bro-foo directives will continue to work. See the documentation for more on aliases.
• Optionally, update the depends metadata field. The special dependencies zeek and zkg are replacing bro and bro-pkg, respectively, and zkg treats them as aliases. Note, however, that existing bro-pkg installations won’t recognize the new names yet, so you might want to leave them in there to support users who have not yet upgraded. See the documentation for more.
• Re-register the renamed package, zeek-foo with central package source. Follow the normal directions to update your index file: remove the old URL for bro-foo and add the URL for zeek-foo.

 

Common Issues When Upgrading 

• If you were running Bro as the bro user and intend to use a zeek user now, don’t forget to remove/update any potential cron jobs you may have.
• If you’re installing Zeek on an old Bro host, remember to first shut down the old cluster using broctl.
• Symptoms of overlapping Bro/Zeek installations:

• Plugins may have failing symbol problems depending on if you run Zeek or Bro.
• zkg packages may fail to install with an error that btest can’t find init-bare.bro.  This may be caused by certain packages using an old version of the get-bro-env script or bro_dist metadata substitution in combination with having the bro-pkg/zkg configuration set to use a mismatched Bro/Zeek sourcetree. 

• Not remembering to update zkg configuration (i.e. updating the paths in ~/.zkg/config or ~/.bro-pkg/config in case you’re now using a different source/installation path for Zeek 3.0.0)
• Not updating PATH environment variable (to either remove an old /usr/local/bro path or to add the new /usr/local/zeek path)
• Plugins will generally need to be recompiled for Zeek 3.0.0 (as is usually the case with new versions). Plugins that require –bro-dist have been seen to have build issues. The best solution is to switch the plugin to the new skeleton code. However, we will try to address any specific issues if you file a ticket with instructions on how to reproduce.
• If you run the BHR scripts, you may need to change those to run as the zeek user as well as the permissions on the queue directory.
• Not remembering to update both where an external processes (e.g. cron job) writes Intel files into the old installation tree and where the Intel configuration (e.g. Intel::read_files) expects to read such files in the case you choose to use the new default installation path. e.g. if Intel was previously written to /usr/local/bro and you now want to use /usr/local/zeek, remember to update both the Zeek configuration and whatever external process may be writing the Intel files.

Feedback 

We realize that we may have missed some places where the name change can impact existing setups. We need your help to close those gaps: if you’re running into any issues upgrading from Bro 2.6 to Zeek 3.0.0, please let us know. If it’s something that we can/should fix, please file a ticket on GitHub. If you have advice for others on how to adapt their setups, scripts, or packages, please leave a comment on this blog posting or email the Zeek mailing list. We’ll be updating this blog posting once the final 3.0.0 release comes out.

 

Contributors

Thanks to Mike Dopheide, Jon Siwek, and Justin Azoff for contributing to this blog posting.