Spicy vs. Binpac Performance, One Year Later

One year has passed since I last wrote about Spicy’s performance (you can read about that here). In that post, I compared it to Binpac. This will be a simple followup: how much progress have we made? How much would this translate to real analyzers?

I won’t bury the lede. This table shows the Binpac numbers from last time, the new Spicy numbers (Zeek version 8.2, Spicy version 1.16), then the old Spicy numbers (Zeek version 7.2, Spicy version 1.13):

In case you missed it, Binpac is the old way of generating protocol parsers, whereas Spicy is newer. Spicy is meant to make the process easier and provide a bunch of features, but it’s certainly a maximal approach. Since Binpac is quite barebones, often requiring custom C++ code, Spicy will be slower in most real analyzers.

These numbers are slightly different from the previous post, since I recalculated them. These have variance, so the regex case (for example) shows a bit of a slowdown, but that’s simply noise. In fact, there are cases that aren’t covered, like how Spicy will now handle characters near the end of a regex character class just as fast as the beginning. But that was an implementation detail.

Clearly, the two unit cases sped up, about 1.6x faster. That’s a pretty significant speedup and something we are proud of. Does this translate to real analyzers?

Real Analyzers

The short answer is: yes, but the effect is more modest. Let’s take the Spicy SSL analyzer (Zeek will use the Binpac analyzer by default, but you can switch to the Spicy version).

I have a PCAP with 9000 SSL connections, each just saying “hello.” It’s a benchmark, but end-to-end within Zeek, with real analysis. I ran this on my laptop (so there’s a decent amount of variance) to get a feel for how much faster Spicy is now. How much faster is it now, compared to 7.2?

The answer: about 1.07x faster. Zeek 8.2 ran in 1.339 seconds end-to-end on average, while 7.2 ran in 1.430 seconds on average. This was measured with hyperfine; you can see the full output at the end of this post.

Note that a decent chunk of that time is in Zeek. How much did just the Spicy parser speed up? I instrumented both and found Zeek 7.2 had 758ms in Spicy parsing, while current Zeek (8.2) spent 660ms. That’s about 1.15x faster.

We can get a bit more scientific with this, but that’s not the point. These performance improvements help a real analyzer! Of course, real analyzers won’t hit the highs of the microbenchmarks. Some analyzers would see more improvements, whereas others use all of Spicy’s features and won’t speed up as much.

Will your analyzer see a speedup? It’s certainly possible. We have primarily targeted ways to reduce generated code within Spicy. If you use all of Spicy’s features, then you probably won’t see as much speedup. This is by design: Spicy first wants to make it easy to write feature rich parsers, then wants to cut any unused features down. The primary goal is, and always will be, making writing parsers easier and safer.

Hopefully that’s exciting!

What’s Next?

Spicy is in a good spot. We’ve achieved a decent speedup on the slowest cases. We have a bunch of awesome features that make writing parsers easier and safer. We plan on implementing optimizations when they make sense and maintain Spicy for now. Now, we are looking for some next steps in the future of Spicy.

Now, the pressing concern is our users: we want you to tell us what could be better in Spicy. Is performance a non-issue? Is it just as hard as it was before to write a parser? Could we tweak the language to be easier to use? Let us know by reaching out on the Zeek Slack – there’s a #Spicy channel for that.

Happy parsing 🙂

Appendix: Hyperfine Output

$ hyperfine "~/.local/zeek/bin/zeek -Cr ssl-bench.pcap Log::default_writer=Log::WRITER_NONE" "~/.local/zeek-7.2/bin/zeek -Cr ssl-bench.pcap Log::default_writer=Log::WRITER_NONE"
Benchmark 1: ~/.local/zeek/bin/zeek -Cr ssl-bench.pcap Log::default_writer=Log::WRITER_NONE
  Time (mean ± σ):      1.339 s ±  0.005 s    [User: 1.235 s, System: 0.086 s]
  Range (min … max):    1.333 s …  1.352 s    10 runs

Benchmark 2: ~/.local/zeek-7.2/bin/zeek -Cr ssl-bench.pcap Log::default_writer=Log::WRITER_NONE
  Time (mean ± σ):      1.430 s ±  0.016 s    [User: 1.330 s, System: 0.085 s]
  Range (min … max):    1.414 s …  1.457 s    10 runs

Summary
  ~/.local/zeek/bin/zeek -Cr ssl-bench.pcap Log::default_writer=Log::WRITER_NONE ran
    1.07 ± 0.01 times faster than ~/.local/zeek-7.2/bin/zeek -Cr ssl-bench.pcap Log::default_writer=Log::WRITER_NONE