Within Zeek, there are two separate parser generators: Binpac (the old one) and Spicy (the new one). Both allow users to write protocol parsers by declaring what the protocol looks like rather than writing C++ code to parse it.

Binpac parsers are difficult to write. The primary documentation is the README, or, more likely, looking at existing parsers. Then, the developer has to use custom C++ logic in order to implement even standard parser features.

Spicy changes this. Spicy has real, comprehensive documentation so you can get started quickly. No C++ knowledge is required either – just a simple language to write a parser with. You also get a range of features: lookahead parsing, error recovery, sinks, and many more.

The barrier for writing protocol analyzers in Zeek (and elsewhere!) has never been lower. But, as we adopt Spicy as the primary tool for creating new protocol parsers, what impact could that have on performance?

What features does Spicy even have?

First, I’ll give a bird’s eye view of Spicy’s unique features. It’s important that there is context when analyzing Spicy’s performance: a more minimal runtime may provide more performance, but if it can’t do what you need, you won’t even start writing a parser.

Safety

Consider this: the network traffic that an analyzer sees is inherently untrusted – in fact, that’s the purpose of analyzing it. If the traffic triggers a vulnerability within the parser, then the parser itself is compromised – the very thing that is meant to ensure the traffic is safe!

Spicy parsers add runtime checks in order to ensure they’re safe. For example, data cannot accidentally be used before it was parsed. Integers won’t unsafely overflow. Data is trimmed when it’s no longer needed. These are all aspects the developer would have to do themselves.

Binpac parsers will have to use custom C++ code in order to perform basic functions. For example, the ASN1 Binpac parser has custom logic to turn an 8 byte string into an integer, which is provided in Spicy. It’s easy to accidentally do something unsafe in C++ – it’s the responsibility of the programmer to make a safe parser. In fact, that ASN1 conversion function needed a patch to make it safe to run. When writing Binpac parsers, developers must be careful not to introduce buffer overflows, access uninitialized pointers, and more. Spicy was designed to eliminate these classes of problems.

Error Recovery

During capture we might not see all transmitted data, or network traffic could deviate from the specification. When parsing inevitably fails, or your parser is imperfect, the parser would ideally pick up where it knows the traffic is good and continue. This is possible in Spicy!

This truly is a superpower. There are few parser generators that give you true control over how your parser recovers after failing. Even mature programming language parser generators rarely have any control over synchronizing input – it’s a unique problem to network protocols. Not only is this possible in Spicy, but it can be done with very few lines of code!

Without this, Binpac parsers each have to implement their own logic in C++. That is almost never done – it’s hard. If the developer gets this wrong, the parser can easily suffer catastrophic failures. Spicy simplifies this process, removing a significant barrier for entry and enabling completely new analyses.

Backtracking

Spicy can go back in the input in various ways. You may want to check parsing of a field, then backtrack if it doesn’t work with &try. You can even (safely) change where in the data stream you’re parsing from – maybe in the middle, maybe at the beginning, maybe just “reparsing” one field.

The Spicy LDAP analyzer, for example, uses this in order to check if a message is encrypted before “real” parsing.

Sinks

Imagine your protocol may receive data out-of-order. It must then reassemble these pieces in-order and send it off to something else to parse. Spicy gives you a mechanism for that (and more!) via sinks.

You can see this in action in QUIC in order to reassemble fragmented or unordered data, or even PostgreSQL in order to parse SSL data.

Filters

If your protocol needs a transformation to apply before “real” parsing, you can use Spicy’s filters to perform that transformation. This may be to decompress Zlib content, or to decode base64 input (both of which are available in Spicy’s runtime library). You can also create your own filters for your own use case.

For example, some data parsed by the Spicy ZIP analyzer may be Gzip compressed, so it uses the builtin Zlib filter.

Independence from Zeek

Spicy is also independent from Zeek. You can use Spicy parsers in your own programs with Spicy’s custom host applications API. You can also use a foreign function interface (FFI) in order to use C++ code in other languages, like Rust.

This part is possible through Binpac, but nowhere near as feasible due to other limitations. Binpac relies on Zeek’s regular expressions and lacks a documented API.

Everything is easier

Spicy parsers are easier to write for the average user when compared to Binpac. There’s functional documentation that stays up to date, so you don’t need to dive into other parsers or the source code in order to figure out how to write your parser. There are more standard library utilities for, say, matching ASCII text, which was more difficult in Binpac. It’s easier to debug Spicy parsers with hooks and options. There’s a more active community – see some Spicy packages on packages.zeek.org.

Not only are there big features that you can’t find elsewhere, but even small tasks are easier in Spicy.

So what?

Spicy just gives you more in an easy-to-digest form. It is a production-ready parser generator that gives many more people access to writing protocol parsers for Zeek and other tools. Not all of these are necessary in every parser, but when they are, they’re uniquely powerful features. Without Spicy, many people simply would not write the parsers.

Binpac parsers almost always require custom C++ logic in order to perform their functions. The DHCP analyzer, for example, does a substantial amount of parsing in custom C++ functions! This can create inherent safety risks. The developer also has to understand the Zeek code, as many of these parsers create internal Zeek objects. This may be difficult, or impossible, for some.

Spicy’s features may produce overhead, though. Not only is there overhead for using it, but just having the ability to use it creates some overhead that needs to be taken out when the feature is not in use. This is an active point of development. How much of an impact does that overhead have on performance?

Measuring performance

In order to measure performance, we need a way to determine how long Binpac and Spicy take to parse some simple constructs. To get a true comparison for the parsers themselves, this should not involve Zeek.

Binpac

The first thing we need is a Binpac parser. This is roughly the simplest parser you can make in Binpac (with all of the generic boilerplate cut out):

type BytesLength = record {
    length: uint64;
    data: bytestring &length=length;
    rest: uint16;
} &byteorder = bigendian;

That parser will parse length bytes, then some 2 byte sequence signifying the end. Thankfully, the Binpac README has a section on running Binpac parsers standalone. That rips out the regular expressions library to do it, but we can just run that test in a Zeek plugin.

In order to run this, we will run the parser standalone with some programmatically generated input. Here’s what I came up with:

std::string make_input() {
    std::uint64_t N = 100000000;
    std::string number = big_endian(N);
    std::string repeated(N, 'A');
    return number + repeated + 'B' + 'B';
}

Basically: the number of bytes as an 8 byte big endian number, then that many A’s, then two B’s. We pick a big number because I happen to be measuring time in seconds, but this could work for a small number of bytes, too.

Then, there is some time_parser function, which will return the number of seconds (as a double) taken by one run of the parser. The specific implementation details aren’t the focus here. We can then time multiple runs and average them to get a rough estimate of the time taken.

Given this minimal Binpac parser and the input length of 100,000,000 bytes, we get an average of 0.005 seconds to execute over 5 runs. This represents the fastest benchmark for Binpac, given its simplicity. The remaining results will come later.

Spicy

Spicy also has a documentation section on “custom host applications” – so we can time the parser execution in much the same way.

First, we’ll create a unit with the same functionality as Binpac:

module Benchmark;

public type WithUnit = unit {
    length: uint64;
    inner: bytes &size=self.length;
    end_: uint16;
};

This is the same thing as the Binpac parser. We’ll copy the make_input exactly so they receive the same input. The timing function is also the same, just with different semantics for Spicy’s API.

With that, the Spicy parser averages at 0.005 seconds to parse. The same as Binpac. That’s probably expected.

But, what about more complex parsers?

A note on methodology

Performance analysis may take a few forms. One could be looking for hot paths and optimizing them. Another could be an end-to-end test with real data in order to measure the impact on real code. These are neither. These are microbenchmarks – they only test very narrow functionality. Overoptimizing for microbenchmarks is bound to lead to disappointment and frustration. But, they can be a useful tool for guiding where the biggest “bang for your buck” will be.

There has been performance analysis work for Spicy using other techniques, but for this comparison, a direct feature-by-feature comparison with minimal parsers seems apt. This analysis can point out where Spicy falls behind Binpac.

Results

This timing approach was done for a few different parsers, which are summarized here (time in seconds):

You can find the Spicy parsers (plus some that didn’t make the cut) here and Binpac parsers here. These are still microbenchmarks, but they’re a bit closer to testing specific functionality that real parsers use. There are a couple of takeaways:

1. Spicy is not inherently slower – it is generally comparable. But, a couple of important cases are significantly slower than Binpac
2. Units parse a decent amount slower than Binpac’s records
3. Regular expressions parse a lot slower than Binpac’s regular expressions

For now, we will investigate point 2 further. Oftentimes units are necessary for the structure of a protocol, so minimal units are relatively common in current Spicy parsers. Many real parsers have a relatively small proportion of bytes parsed through regular expressions. Currently, it’s unknown if the difference is how Spicy parses the regular expression, or the particular implementation of the regular expressions. Regardless, units simply occur more – and real Spicy parsers are slower without any regular expressions, but with lots of units.

What will we do about it?

In order to find what unit optimizations worthwhile, I took the generated C++ code with the WithUnit benchmark and made 8 potential patches on it. Spicy doesn’t always remove code generated from unused features, so most of these patches are simply removing that unnecessary code. These are applied sequentially then benchmarked (so patch 2 includes 1 and 2). We can see how just removing some code will get closer and closer to Binpac’s performance numbers:

You can find the generated code changes with patches for each step here.

After manually cutting down a lot of the code in the parser, without even adding in any true optimizations ourselves, we get close to Binpac’s numbers. Not all of these are feasible to implement generally, but a lot of these are. Some of them simply remove unnecessary code. Who knows what speedups we can get with more advanced optimizations.

This benchmark does initially indicate that yes, Spicy parsers are slower than Binpac parsers. These patches show that the generated parser code can be patched and have similar execution time to Binpac. We can use these manual changes in order to guide which optimizations should be prioritized, and, hopefully, automatically get most Spicy parsers’ execution time down to equal Binpac parsers’ execution time.

It’s good that there is room for optimization. We know where to look and how to perform optimizations that would make a real performance impact on common patterns in Spicy parsers. The next step is implementation – more on that at the end.

What can you do about it?

Now, you may worry about performance. That’s a perfectly valid worry depending on your needs. What steps can you take in order to make sure these performance impacts don’t negatively affect your use case?

First, ask whether a parser’s performance is a bottleneck. There are a lot of aspects to analyzing network traffic. Depending on your use case, the performance impact may be negligible. Within Zeek, for example, script-layer execution, concurrent parsing, cluster I/O, and the site’s traffic mix might dwarf the work done by a given Spicy parser. If the parser would otherwise not get written, then the performance impact may not even matter.

Next, Spicy code can be optimized. Run through the runtime performance section in the wiki for tips on how to improve. In the future, these recommendations may get solidified into some automatic tooling (like a linter or warnings). Until then, that page has good recommendations.

Finally, you can also defer to writing custom C++ code using custom extensions. If there’s performance critical code and you need a custom, efficient library in C++, that may be useful. This approach, though, would be recommended only as a last resort.

What’s Next?

Overall, Spicy can be slower than Binpac, but this is not an inherent limitation. We are working to improve Spicy’s performance. We’re working on control-flow based optimizations in order to cut down on unnecessary work. There are also opportunities to remove overhead passed via function calls from unused features, such as errors without synchronization.

Spicy is uniquely good for creating protocol parsers. Pain points that are common for network protocols are easy in Spicy. Users who could otherwise never write a protocol parser now can. Creating those features requires possibly generating too much code. The next steps are to cut down the particularly impactful areas via well-known compiler optimization techniques.

Binpac did not optimize the code in a superior way to Spicy. Binpac simply does not have as many features, so it does not need any optimizations during code generation. It generates very little code because it does very little. To account for this, Binpac gives the user the ability to extend the parsers with C++ – if they have the expertise. Spicy also gives this to users, but not because it’s necessary, it’s just a bonus.

If you want to follow along with this progress, there are two discussions open in the Spicy repository: one for units, one for regular expressions. I have a repository that houses some of my initial investigations. Finally, you can watch for Spicy pull requests. These may include upcoming optimizations, or just new microbenchmarks.

Or, as always, we welcome contributions. 🙂

P.S.

There may be other ways to improve Spicy’s performance, such as introducing new language features or a “fast mode” which disallows many features. This is tempting, but still a significant amount of work. The goal is to improve the performance without changing the language. There’s a lot of work to be done. Perhaps in the future it will be worth it to dive more into other solutions when the standard use case is more performant.