Zeek 5.0

The Zeek team is quite excited to announce Zeek 5.0: it’s a major release providing a lot of new functionality, both extending Zeek itself and also growing its ecosystem through new capabilities and tools. Thanks to everybody who has contributed to this release, many threads are coming together here. Let’s take a closer look at what’s new.

What’s new in Zeek 5.0?

Zeek 5.0 is introducing two major new blocks of functionality:

• Built-in Spicy integration. Spicy is our recent parser generator that makes it easy to create robust analyzers for network protocols and file formats. Think of Spicy as a domain-specific scripting language for all your protocol parsing needs. While Spicy has been available externally for a while, it has now graduated into a core, built-in component of Zeek. That not only means that Spicy is now available by default for any Zeek package aiming to provide Zeek with new analyzers, but also that in the future we can start relying on Spicy for Zeek’s built-in analyzers as well. We will maintain and support Spicy along with Zeek, providing the same stability and maintenance guarantees for the Spicy toolchain as for Zeek itself. To make it easier to get started writing Spicy analyzers, we have added a Spicy template to the Zeek package manager, enabling developers to quickly bootstrap a skeleton Spicy analyzer.
• New management framework for deploying Zeek. As our existing ZeekControl has become quite dated, in Zeek 4.2 we started a journey towards a modern replacement. The 5.0 release now ships with a fully functional version of Zeek’s new Management Framework, able to operate single-system Zeek clusters at the time (i.e., setups in which all Zeek processes reside on a single machine). The framework comes with built-in with support for log rotation and archival through the zeek-archiver, making log processing more robust. At the core of the framework’s management model resides Zeek itself: The new framework leverages Zeek’s built-in process supervisor to manage all the cluster nodes (workers, loggers, etc.), along with a Broker-based control API and a command-line client for interaction. See the Management Framework documentation for how to get started. The new framework remains experimental for now, and we will continue to extend and refine it over the 5.x development period, including adding support for multi-system Zeek clusters to complete the ZeekControl replacement. In the meantime, we’re very interested in any feedback you may have on this new system.

That’s not all there is in Zeek 5.0. Further highlights include:

• Zeek now automatically warns about unused functions, hooks, and event handlers in your Zeek scripts. This catches instances where the script writer has introduced typos in names, or has forgotten to remove code that’s no longer needed.
• Zeek now provides WebSocket support for exchanging events with external clients, as an alternative to communication through the Broker library. Call Broker::listen_websocket() to enable it. Zeek will then start listening on port 9997 for incoming WebSocket connections over which to exchange JSON-encoded Zeek events. See the Broker documentation for a description of the message format.
• A new option –profile-scripts[=file] instructs Zeek to write (upon termination) a profile of every function body to the given file along with some aggregate profiles, including CPU time and approximate memory accumulated during its execution.
• Zeek now supports generation and replay of event traces via a new –event-trace command-line option. These traces record the events that occur during an execution in a manner that you can then later replay to get the same effect, but without the traffic source. You can also edit the recording to introduce differences between the original, such as introducing corner-cases to aid in testing, or anonymizing sensitive information. For details, see the documentation.
• Zeek now lets you decrement Counting Bloom Filters using the new bloomfilter_decrement() built-in function.. Furthermore, two types of Bloom Filters can now be intersected using the new bloomfilter_intersect().
• Zeek now features limited TLS decryption capabilities. This feature is experimental and currently only works for TLS 1.2 connections that use the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 cipher suite. Furthermore Zeek requires access to the pre-master secret of each TLS connection. For more details and examples how to use this functionality, see the TLS Decryption documentation.
• Two new built-in functions, val_footprint() and global_container_footprints(), offer substitutes for the deprecated (and often misleading) val_size() and global_sizes() functions. A value’s “footprint” is the number of elements it includes either directly or indirectly. The number is not meant to be precise, but rather comparable: larger footprint correlates with more memory consumption.
• Zeek’s (still experimental) script compilation machinery gained substantial new functionality and improved performance:
  • Compiling to ZAM abstract machine code now benefits from major performance improvements, and learned to support cross-compilation to another target platform.
  • Compiling scripts to C++ is now “feature complete”, i.e., the compiler should be able to process any script—other than those potentially affected by @if conditionals—regardless of which Zeek language constructs it uses.
• We substantially re-architected Broker’s transport layer to prepare for new functionality and easier maintenance. Broker is now also hiding CAF—the underlying, low-level communication library— behind the scenes, avoiding a runtime dependency. While unfortunately these changes break interoperability with older Broker versions, Broker now includes a new versioning handshake that will facilitate backwards compatibility with future protocol changes.
• In our ongoing effort to modernize Zeek’s C++ codebase, Zeek now uses the c-ares library for performing DNS requests, replacing an old custom implementation of a DNS resolver. Switching to this library simplifies the DNS code, and adds support for IPv6 lookups as well as the ability to support more DNS request types in the future.

As always, this release comes again with many additional smaller improvements and bug fixes as well. See the NEWS file for more information, and CHANGES for the exhaustive log. 

Zeek Ecosystem

Along with Zeek itself, there are a couple of recent extensions to the Zeek ecosystem that we would like to call out as well: New tooling that helps developers write Zeek scripts, and a new version of our experimental endpoint monitor, the Zeek Agent.

Tooling

On the tooling side, we have two new utilities available to support script developers:

• zeek-format. A tool that automatically (re-)formats Zeek scripts, relieving developers from manually needing to get indenting and spacing right. Our philosophy here is similar to gofmt, and the opposite of clang-format: there is only One Way to layout Zeek scripts, and this tool provides it. Accordingly, it features zero options for tweaking the formatting.
• Zeek Language Server. An experimental language server for Zeek scripts that enables code editors (such as, VSCode and vim) to provide features like code completion, go-to-definition, and on-demand documentation. It’s pretty cool, make sure to give it a try and let us know how it works for you.

Endpoint Monitoring

The Zeek Agent is an experimental endpoint agent that sends host information to Zeek for central monitoring. It supports Linux, macOS, and Windows systems (with the latter being brand new; binary builds for Windows are coming soon). On the Zeek side, the host activity—such as current processes, open sockets, or the list of users on the system—shows up as script-layer events, just as network activity does. Starting with Zeek 5.0, agents now communicate with Zeek through its new WebSocket interface (see above) to reduce the agents’ footprint. The agent comes with a Zeek script package that adds a number of new log files to Zeek that record endpoint information received from agents. The package also provides a API to custom scripts for controlling agents and processing the events they send.

The current version of the Zeek Agent (“v2”) is a reimplementation of an earlier incarnation. It retains the original SQL-style data model, but reduces the complexity of deployment and code base substantially. We’re continuing to extend the agent’s capabilities on all three platforms, and are very interested in any feedback you may have.

Support and Feedback

Zeek 5.0 is our new long-term-support (LTS) release, and we will support it with bug fixes for at least a full year. We will continue to maintain the current 4.0.x series until 5.1 comes out later this year (presumably around the end of October). Give a Zeek 5.0 a try, and let us know what you think: Join the Zeek Slack for any feedback, or file a ticket on GitHub if you spot a problem.