Revisiting Weird-ness: Investigating DNS NOTIFY at Scale

At ESnet, our Zeek deployment was logging 4.2 million weirds per day. When we investigated, 90% were DNS-related and 98% of those came from a single opcode. After submitting a PR to fix it, daily weirds dropped to 1.3 million. Here’s how we tracked it down.

What is weird.log?

One of the most interesting log files that Zeek produces is weird.log. It captures protocol-level anomalies;  basically, whenever there’s something unexpected at the protocol level, that’s a weird. “Weirds” are hardcoded by whoever wrote the analyzer. They can also be generated by scripts, but that’s rare.

In simpler terms: any protocol traffic not complying with RFC standards, or any unknown traffic that isn’t parsed by Zeek, is flagged as weird.

If you run Zeek, you have encountered weird.log. The specific weirds you see depend on what traffic your deployment monitors–which is why your weird.log probably looks different from examples in blog posts or conference talks.

Our Zeek deployment monitors datacenter traffic: it can see east-west (internal hosts traffic) as well as north-south traffic (local-to-internet traffic). Here is the breakdown of some of the popular DNS-related weirds during a 24hrs window on a weekday that we see getting triggered in our log file:

DNS_unknown_opcode

DNS_truncated_len_lt_hdr_len

DNS_truncated_RR_rdlength_lt_len

DNS_Conn_count_too_large

DNS_truncated_quest_too_short

Breaking down the pattern

90% of the DNS-related weirds were unknown opcode parsing errors. Breaking that down further, we tried to find out the largest portion of that 90%:

DNS_unknown_opcode

DNS_unknown_opcode

DNS_unknown_opcode

DNS_unknown_opcode

DNS_unknown_opcode

For all the DNS unknown opcodes recorded, it turns out the opcode=4, which is for DNS Notify messages, was constituting the 98% of the “DNS_unknown_opcode” weirds. 

What is DNS NOTIFY?

DNS NOTIFY (defined in RFC 1996 and RFC 8765) enables a primary DNS server to immediately inform secondary (slave) servers of changes to zone data. This alert triggers a zone transfer that ensures the secondary servers’ data remains synchronized with the primary. The recording of numerous messages in our weird.log file made sense: our Zeek sensors monitor internal traffic, which includes communication between our DNS servers. When you have Zeek monitoring only north-south traffic, you miss these internal DNS notifications. Given our vantage point for Zeek, it was prudent to solve this alert, as it was triggering millions of DNS NOTIFY message alerts every day.

The Quick Fix

Since Zeek already supports standard query/response (opcode 0) parsing in DNS messages, NOTIFY messages can be treated indifferently to be supported built-in without any additional parsing code. The fix is straightforward: enable opcode 4 and don’t exclude it from the packets parsing src code in DNS.cc file. We submitted a PR to enable standard QR parsing for DNS NOTIFY messages and avoid the flood of “DNS_unknown_opcode (4)” in the zeek weird.log file, which is merged into the master and should be available in the next minor release.

The Results

After updating our datacenter cluster to Zeek 8.2.0-dev.534 (which includes this fix), we immediately saw a decline in DNS-based weirds from 90% to 13% of total DNS-related weirds:

DNS_truncated_len_lt_hdr_len

DNS_unknown_opcode

DNS_truncated_RR_rdlength_lt_len

DNS_Conn_count_too_large

DNS_truncated_quest_too_short

Overall weird alerts dropped from ~4.2M/day to ~1.3M/day. The graph below shows the decrease in weird count split by “name” field”, where y-axis is showing the total number of each type of “name” (weird) recorded over 10 days (04/06/26-04/16/26):

Takeaway

If one weird is triggering millions of times in your environment, investigate it. In our case, fixing one opcode reduced our daily weird count by 70% and saved us on SIEM ingestion costs. 

Check your weird.log and sort by count. If one weird dominates, dig into what’s triggering it. You might find a simple fix that cleans up your logs.