The Cure for Exceptional Zeek Package Testing (Part 1)
This is Part 1 of a three-part series. Read Part 2 and Part 3.
Introduction
As someone who ships a lot of Zeek packages to production, I’m always looking for ways to improve package testing. In the common, simple case, you have a small PCAP that exercises the behavior you need to generate your log or Notice and you can write a simple btest for it. But what if our case isn’t so simple? Perhaps a PCAP isn’t available or cannot be shared. Or maybe your script depends on events generated by some external service. If you’ve found yourself in a similar situation, read on! If this intro seems full of jargon to you, you may want to familiarize yourself with Zeek scripting and BTest first.
I’ll walk you through four examples over the next three blogs in this series of some tricks I’ve picked up for writing tests in these exceptional cases. All the examples will use BTest, Zeek’s generic test driver. We’ll discuss the first one here, and the remaining in Part 2 and Part 3.
- Writing good unit tests for auxiliary functions.
- Hand-crafting events to exercise your package’s functionality.
- Automatically generating events from a PCAP using Zeek’s built-in event tracing flag.
- Using a Zeek log and the Input Framework to drive your package’s functionality.
The fourth method can also be used for testing/debugging your package (if possible) on long-term logs when storing the PCAPs is too costly. I’ve used this technique to refine an algorithm on months of log files when I recognized false negatives in production. All that doing so required was writing a short driver script to run the Zeek package based on the log files.
Let’s go over our Zeek package we would like to test and how to write idiomatic unit tests with BTest.
Motivating Example: ip-distance
Rather than just describe the techniques, I want to show them in a fully fleshed out, albeit simple, Zeek package. ip-distance generates a new log, ip_distance.log, that computes two versions of Metcalf’s distance metric for IPv4 IP addresses. The slide deck explains why you may want such a metric, but the most important bit is what we actually compute. For a given pair of IPv4 addresses a.b.c.d and w.x.y.z we can compute the distance d with:
which is the Euclidean distance weighted by the number of IPs in each CIDR level. For example, the IP pairs (10.0.0.0, 10.0.0.1) and (10.0.0.0, 11.0.0.0) both only “differ” by one in their dotted quad representation but are actually much farther apart when considering the CIDR boundaries. The first represents an inclusive range that contains just those two IP addresses, while the second contains 16,777,217 IPs. The distance metric above accounts for this.
ip-distance’s main.zeek contains the meat of the package:
1: module IPDistance; 2: 3: export { 4: redef enum Log::ID += { LOG }; 5: 6: type Info: record { 7: uid: string &log; 8: distance: double &log; 9: norm_distance: double &log; 10: }; 11: } 12: 13: global a = pow(2, 24); 14: global b = pow(2, 16); 15: global c = pow(2, 8); 16: 17: function octets(ip: addr): vector of int 18: { 19: local cip = addr_to_counts(ip)[0]; 20: 21: # The + ensures that the values are considered `int`s instead of `count`s. 22: return +vector((cip & 0xFF000000) >> 24, 23: (cip & 0x00FF0000) >> 16, 24: (cip & 0x0000FF00) >> 8, 25: cip & 0x000000FF); 26: } 27: 28: function distance(ip1: addr, ip2: addr): double 29: { 30: local oct1 = octets(ip1); 31: local oct2 = octets(ip2); 32: return sqrt(a * pow(oct1[0] - oct2[0], 2) + 33: b * pow(oct1[1] - oct2[1], 2) + 34: c * pow(oct1[2] - oct2[2], 2) + 35: pow(oct1[3] - oct2[3], 2)); 36: } 37: 38: const MAX_IPV4_DISTANCE = distance(0.0.0.0, 255.255.255.255); 39: 40: function norm_distance(ip1: addr, ip2: addr): double 41: { 42: return distance(ip1, ip2) / MAX_IPV4_DISTANCE; 43: } 44: 45: event connection_state_remove(c: connection) 46: { 47: if ( is_v6_addr(c$id$orig_h) || is_v6_addr(c$id$resp_h) ) 48: return; 49: 50: local info = Info($uid=c$uid, $distance=distance(c$id$orig_h, c$id$resp_h), 51: $norm_distance=norm_distance(c$id$orig_h, c$id$resp_h)); 52: Log::write(LOG, info); 53: } 54: 55: event zeek_init() &priority=5 56: { 57: Log::create_stream(IPDistance::LOG, [$columns=Info, $path="ip_distance"]); 58: }
Note the auxiliary function octets, which is used by the main metric functions distance and norm_distance. octets returns the integers from the dotted-quad representation of the IP as a vector of int with some bit twiddling. So a.b.c.d becomes vector(a, b, c, d). With this, we can compute our distance metric (d) and a normalized variant bounded by [0.0, 1.0].
While we could manually construct PCAPs that exercise the edge case behavior in these functions, it’s much easier to make unit tests in BTest to ensure they work as we expect.
Unit Tests
BTest doesn’t have a script-layer unit test library that you might be used to in other languages like pytest for Python. However, it’s easy to emulate their basic functions using BTest and the assert statement that was added in Zeek 6.1.0. Below is the source for unit-test.zeek:
1: # @TEST-DOC: Run unit tests 2: # @TEST-EXEC: zeek $PACKAGE %INPUT 3: 4: module IPDistance; 5: 6: function double_equal(v1: double, v2: double, tol: double &default = 0.00001): bool 7: { 8: local diff = v1 - v2; 9: 10: if ( diff < 0.0 ) 11: diff = -diff; 12: 13: return diff <= tol; 14: } 15: 16: function every_T(v: vector of bool): bool 17: { 18: for ( _, x in v ) 19: if ( ! x ) 20: return F; 21: 22: return T; 23: } 24: 25: event zeek_init() 26: { 27: assert double_equal(0.0, 0.1) == F; 28: assert double_equal(0.0001, 0.0002) == F; 29: assert double_equal(0.1 + 0.2, 0.3) == T; 30: 31: assert every_T(octets(0.0.0.255) == vector(0, 0, 0, 255)); 32: assert every_T(octets(192.0.0.255) == vector(192, 0, 0, 255)); 33: assert every_T(octets(0.0.0.0) == vector(0, 0, 0, 0)); 34: assert every_T(octets(255.255.255.255) == vector(255, 255, 255, 255)); 35: 36: assert double_equal(distance(0.0.0.255, 0.0.0.255), 0.0); 37: assert double_equal(distance(0.0.0.255, 0.0.1.0), 255.501468); 38: assert double_equal(distance(255.255.255.255, 0.0.0.0), MAX_IPV4_DISTANCE); 39: assert double_equal(distance(0.0.0.0, 255.255.255.255), MAX_IPV4_DISTANCE); 40: assert double_equal(distance(0.0.0.0, 1.0.0.0), 4096.0); 41: assert double_equal(distance(1.1.1.1, 1.1.1.1), 0.0); 42: assert double_equal(distance(0.0.0.255, 0.0.0.255), 0.0); 43: 44: assert double_equal(norm_distance(0.0.0.255, 0.0.0.255), 0.0); 45: assert double_equal(norm_distance(0.0.0.255, 0.0.1.0), 0.000244); 46: assert double_equal(norm_distance(255.255.255.255, 0.0.0.0), 1.0); 47: assert double_equal(norm_distance(0.0.0.0, 255.255.255.255), 1.0); 48: assert double_equal(norm_distance(0.0.0.0, 1.0.0.0), 0.003914); 49: assert double_equal(norm_distance(1.1.1.1, 1.1.1.1), 0.0); 50: assert double_equal(norm_distance(0.0.0.255, 0.0.0.255), 0.0); 51: }
For those unfamiliar with BTest, lines 1–2 are keywords that tell BTest what to do. The first is just for documentation and the second explains how to execute the test. The second line runs zeek and includes our package and the input of unit-test.zeek.
Since we are going to need to compare floating point numbers in our test, we first define a helper function double_equal (lines 6–14) to compare the output of our distance function to their expected values (if you’re unsure why this is necessary, try computing the obviously true statement 0.1 + 0.2 == 0.3 in your Python REPL to see what I mean).
To test our octets function, we will need to compare vectors. When comparing two vector types with ==, Zeek returns a vector of bool where each respective element of the two vectors is compared. It errors if the vectors are not the same length:
$ zeek -e 'print vector(1, 2, 3) == vector(2, 2, 2);' [F, T, F] $ zeek -e 'print vector(1, 2, 3) == vector(2, 2, 2, 4);' expression error in <command line>, line 1: vector operands are of different sizes (vector(1, 2, 3) == vector(2, 2, 2, 4)) fatal error in <command line>, line 3: failed to execute script statements at top-level scope
We can check that two vectors, v1 and v2, are equal if all the elements from the result of v1 == v2 are T, which every_T (lines 16–23) does.
The bulk of the test is in our zeek_init event handler defined in lines 25–51. It’s a series of assert statements. If one of them fails, btest with the -d flag will show exactly which test case failed. If we run this after inverting the expected result of the first test we see:
¡ btest -c testing/btest.cfg -d ipdistance.unit-test [ 0%] ipdistance.unit-test ... failed % 'zeek $PACKAGE .../ip-distance/testing/.tmp/ipdistance.unit-test/unit-test.zeek' failed unexpectedly (exit code 1) % cat .stderr error in .../ip-distance/testing/.tmp/ipdistance.unit-test/unit-test.zeek, line 27: assertion failure: IPDistance::double_equal(0.0, 0.1, 0.00001) == T fatal error: errors occurred while initializing 1 of 1 test failed
As I mentioned, this isn’t the only way one could write unit tests with BTest. Prior to assert, unit tests in BTest would print the comparisons or print the results and compare the baselines. In fact, this is how I used to do them. assert expresses your intent and clearly flags which test is failing, so I tend to use assert more these days.
Unit tests are helpful with functions, but any interesting Zeek package is driven by events. The next two blog posts will walk you through how to test in those exceptional cases.