Zeek 4.0 LTS Release

(This is an updated version of an earlier posting announcing a 4.0 release candidate.)

We are very excited to make Zeek 4.0 available. This is an LTS release with support for the coming year. Some highlights coming with this version:

• Plugin API support for adding “packet analyzers” for parsing link-layer protocols. That functionality had remained hardcoded so far, preventing Zeek packages from adding low-level analyzers of their own. Architecturally, this is also the first step towards supporting non-IP based protocols. (In fact, IP is now a packet analyzer itself as well.)
• zkg has been integrated into the Zeek distribution. This means that you usually won’t need to install the package manager separately anymore (unless your OS packaging system makes you). The integration also moves zkg’s runtime state from the user’s home directory into the Zeek installation itself, making it easier to keep packages aligned with Zeek versions.
• A new unknown_protocols.log recording unexpected link-layer protocols. It can be enabled by loading the policy/misc/unknown-protocols script.
• New log filter hooks streamlining customization of log content.
• Extensions to the analyzers for DNS, SSH, and DCE-RPC.
• HTTP DPD changes to analyze large sessions where a single side exceeds the DPD buffer size. You may see more entries in your http.log now.
• Various new string processing functions similar in spirit to what Python provides.
• Improvements to packet loss reporting through capture-loss.zeek.
• A new global zeek_script_args variable passing command-line arguments into Zeek scripts (think: “hash-bang”)
• Composite-index support for Broker-backed script tables.
• A new dictionary implementation that improves performance substantially.

To help with transition for people still on the previous 3.0 LTS track, we will continue to provide updates for critical issues in 3.0.x for another two months.

As always, there are many more smaller pieces going into this release as well: check out NEWS for a summary and CHANGES for the exhaustive list. For users still on a 3.0.x version, upgrading will also give you everything else that we added over the previous year, including: a new core I/O loop no longer wasting CPU cycles on low traffic volume; Broker-backed script tables facilitating persistence and cluster sharing; and an experimental version of a new supervisor framework. See the announcements for 3.1 and 3.2 for more information. Over the course of the 3.x development cycle, we merged more than 400 development branches, leading to more than 2200 commits to the main branch in total.

People working on plugins, or on the Zeek code base itself, will be happy to hear that we also continued our effort to modernize the Zeek C++ code base, including moving many more classes into either the zeek or zeek::detail namespaces (as begun with 3.2). We also almost have clang-format configuration ready, but ran into a last-minute glitch with the tool that is currently preventing us from moving ahead. Once that’s fixed, we will do a one-time global reformatting of all the C++ code.

We also made a concerted effort to close out tickets during this cycle. If you are following the GitHub issue tracker, you may have noticed quite a few old items getting addressed finally. Overall, we closed more than 50 old tickets, and are now down from over 100 that were open when 3.2 came out to about 40 (and that’s including new ones filed recently).

We have also formalized a new platform support policy: By aligning our CI setup with the major OS versions, we are turning it into our bellwether for 3rd party dependencies that we can assume users to have in place.

We have more stuff in the pipeline that didn’t quite make it for 4.0 anymore, and there are also some new big ticket items on the horizon for the 4.x series. You can always follow the links on the roadmap page to track what’s already being worked on.