How to Write Zeek Logs in JSON & TSV Simultaneously

Zeek can write the same log stream in multiple formats simultaneously. If you need JSON for your SIEM and TSV for long-term archives, you’re in luck: a few lines of code handles both.

There are several reasons one might want two log formats simultaneously. For example, you could log TSV files to disk for long-term archiving, and send off json-logs to your SIEM platform that only keeps a couple of months of data accessible.

This topic comes up regularly in our community. There are already two packages that handle it [1], [2], but if your setup has specific requirements or you just want to understand what’s happening under the hood, a few lines of code handles it directly.

How Does Zeek Logging Work?

Zeek logging is built around two abstractions: log streams and log filters.

To create a new Zeek log file inside of Zeek, first create a Log Stream. If you have seen a Zeek script that writes to a log file, you probably have seen a line that looks like:

Log::write(Foo::LOG, rec);

This sends the information of the record to the Foo::LOG log stream.

A log stream can have one or several Log Filters attached to it that define what information gets written out, how, and to where. When you create a Log Stream, a default filter is created that writes the information to a file.

You can easily add additional log filters to a log stream and write all (or a subset) of the information to a different file, in a different format. Here’s how.

Adding a Second Log Filter

To log conn.log in JSON in addition to TSV, copy the default filter, change the log writer to the ASCII writer with JSON enabled, and add it as a new filter:

event zeek_init()
    {
    local filter = copy(Log::get_filter(Conn::LOG, "default"));
    filter$name = "json"; # the filter needs a different name
    filter$path = "conn.json";
    filter$writer = Log::WRITER_ASCII;
    filter$config["use_json"] = "T";
    
    Log::add_filter(Conn::LOG, filter);
    }

Once this is in place, Zeek will write two files: conn.log in TSV format and conn.json in JSON format.

Note that you have to copy the filter that you get from Log::get_filter – otherwise you end up editing the existing default filter ☺

Community Contribution

This also is an example of how community questions turn into project improvements: Someone recently asked about writing logs in multiple formats in the Zeek Slack workspace and ended up contributing a PR to one of the packages above to add a feature he was missing.

Wrapping Up

The Zeek logging framework is very flexible and you can customize nearly any aspect of Zeek logs with only a couple of lines of code. Log filters let you rename fields, exclude fields, change log rotation, set filter-specific field name maps, and more. It’s even possible to have dynamic logfile names, automatically creating a differently named logfile for each hostname, for example.

Once you understand how log filters work, the framework opens up considerably. The logging framework documentation covers most of the features and is surprisingly short.