Reducing Zeek JSON Log Size: Field Name Mapping and Log Filtering Hooks

Zeek’s JSON logs can get large quickly, especially if you’re ingesting them into a SIEM that charges by data volume. There are two ways to address this: filtering out uninteresting log lines, and renaming fields to reduce per-event size. Both take only a few lines of code.

Renaming Fields to Reduce Log Size

This approach came from a community member, Mark, who noticed he could save multiple gigabytes per day in log volume just by renaming log fields to shorter lengths. To my knowledge, this is the first time someone brought up this rather inventive idea and it’s a great reason to point out how to do this in Zeek.

The Zeek logging framework has a feature called Field Name Mapping. It allows you to specify a Zeek table that maps current field names in the log files to the names you would like to use. For example:

redef Log::default_field_name_map = {
     ["id.orig_h"] = "o_h",
     ["id.orig_p"] = "o_p",
     ["id.resp_h"] = "r_h",
     ["id.resp_p"] = "r_p"
};

This reduces the length of these field names down to three bytes. The default_field_name_map applies to all log files, but the Field Name Mapping documentation also shows how to apply this to a single field only.

Reducing Log Size via Hooks

The more standard approach is to skip uninteresting log lines entirely—like connections that never get established. The easiest way to do this is to use log filtering hooks. We recently published a blog post on this if you want to go deeper: Reduce conn.log from 35GB to 5GB with a Simple Hook.

The short version: every default Zeek log exposes a hook you can use to veto individual log lines. For example, to skip all connection log lines from known scanners:

hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)

{
    if ( rec$id$orig_h in Scan::known_scanners)
        break; # break causes the hook to return false – which skips the log line
}

Wrapping Up

The main take away is that the Zeek logging framework is very flexible; you can customize nearly any aspect of Zeek logs with only a couple of lines of code. It’s possible to remove information, add metainformation, change filenames, write a subset of information to different files, change the logging format, and more.

If you want a slightly deeper dive into the logging framework, take a peek at the logging framework documentation. It’s surprisingly short and shows off most of the features.

And if you’re working with multiple log formats, the companion post on writing Zeek logs in JSON and TSV simultaneously covers another way the logging framework gives you flexibility without much code.