AWS provides a feature that allows mirroring your infrastructure’s network traffic to a separate system for analysis purposes. This is called AWS Traffic Mirroring. If you’d like to use Zeek’s network traffic analysis capabilities in such a cloudy environment, this blog post explains how to do so using the recently published UDP-based packet source plugin to consume VXLAN encapsulated mirrored traffic and forwarding Zeek logs directly to Kafka.
While we focus on AWS Traffic Mirroring, other cloud providers have similar features. For example, GCP’s Network Security Integration uses GENEVE instead of VXLAN, but conceptually should work the same.
UDP-based Packet Source
In contrast to other packet sources like AF_PACKET, NETMAP, PF_RING, etc. which read packets directly from a raw network interface, the plugin presented in this post provides a packet source component that turns Zeek into a high-performance UDP server, consuming mirrored traffic from a UDP socket.
In short, the plugin supports the following interface specification, stripping the VXLAN header and sending the encapsulated payload into Zeek’s packet processing pipeline:
zeek -i udp::0.0.0.0:4789:vxlan
The udp:: prefix selects the packet source plugin to use. The packet source is initialized with the remainder of the string (0.0.0.0:4890:vxlan) as the interface path. If left out, the built-in libpcap-based packet source is used. Another common prefix is af_packet:: which selects the built-in AF_PACKET packet source on Linux.
Support for GENEVE encapsulation, a combination of GENEVE and VXLAN that’s used by AWS GWLB setups or raw and skip configurations exist as well. By default, the data link type (dlt) value of the packet source is set to DLT_EN10MB. Some example usages:
zeek -i udp::0.0.0.0:6081:geneve+vxlan zeek -i udp::0.0.0.0:10000:raw zeek -i udp::[::]:4242:skip=16:dlt=raw
A nice side-effect of using this packet source is that Zeek runs unprivileged. There’s no need for the CAP_NET_RAW capability as we’re not directly accessing a network interface. Further, the packet source strips the encapsulation layer, relieving Zeek from tracking, analyzing and logging VXLAN tunnel connections. This happens in default setups with other packet sources that read raw packets, but logging information about the VXLAN tunnels is generally not very useful. For horizontally scaling Zeek across multiple processes, the packet source sets SO_REUSEPORT on the UDP socket. This, in turn, allows multiple Zeek processes to listen on the same port and the Linux kernel will load-balance packets among the sockets using flow-hashing on the outer IP/UDP header — exactly what Zeek or other DPI systems generally require. See the packet source’s README for a few more details about this load-balancing idea and also the requirements on the packet mirroring infrastructure.
The implementation uses recvmmsg() to receive packets. This should provide decent performance. Zeek is a stateful passive networking monitor with fairly heavy Deep Packet Inspection (DPI) and extensive scripting capabilities. The packet path is usually not the bottleneck for Zeek. Further, within AWS, it’s possible to launch additional EC2 instances and use a UDP-based network load balancer (NLB) as a packet broker in front for further horizontal scaling across instances. The plugin also contains an experimental io_uring implementation using io_uring_prep_recvmsg_multishot for further single instance performance experimentation.
Different architectures for deploying Zeek with AWS Traffic Mirroring.
An additional avenue to explore for performance is to use multiple network interfaces on a single instance with an NLB in front. For production deployments, a dedicated interface for the mirror traffic is recommended. For simplicity we use only a single interface per instance here. The various configurations are depicted in the diagram above.
Setting up EC2 instances
We assume you have access to an AWS console and a basic understanding of working in such an environment. AWS’s Traffic Mirroring Get started guide is a great way to get familiar with its main concepts.
Create the Source EC2 Instance
If you already have instances that support traffic mirroring, you may skip this step. Otherwise, create an ephemeral t4g.micro Ubuntu 24.04 instance and ensure you have SSH access and note down the ENI identifier of its NIC. It’ll be used as the mirroring source when creating the Traffic Mirroring Session later.
Create an EC2 Instance to run Zeek
The second EC2 instance will be running Zeek. Place it in a security group that allows incoming traffic on port 4789/udp (VXLAN) and ensure you have SSH access to it.
SSH
Add both instances to ~/.ssh/config, named aws-tmt (traffic mirror target) and aws-tms (traffic mirror source). Our console output will use these hostnames:
# ~/.ssh/config
Host aws-tmt
Hostname 18.219.xxx.yyy
User ubuntu
IdentityFile ~/.ssh/aws-tmt.key
Host aws-tms
Hostname 18.223.xxx.yyy
User ubuntu
IdentityFile ~/.ssh/aws-tms.key
Setup a Traffic Mirror Session (TMS)
In the AWS console, navigate to VPC -> Traffic mirror sessions -> Create a traffic mirror session.
- As Mirror source, select the ENI of the EC2 instance to monitor.
- As Mirror target, create a new target of type “Network interface” and select the ENI of the instance that’ll run Zeek. The other types Network Load Balancer and Gateway Load Balancer Endpoint are for more advanced setups.
- For testing, set Session number to 1 and VNI to 4711.
- For the filter, create a new one to mirror all inbound and outbound traffic from 0.0.0.0/0 (anywhere). We also exclude port 22 such that the SSH traffic isn’t mirrored.
Once the traffic mirror session has been created, we use tcpdump on the target instance to verify the source instance’s traffic is mirrored encapsulated within VXLAN:
# DNS lookup zeek.org via 8.8.4.4 on the source instance, aws-tms: ubuntu@aws-tms:~$ dig @8.8.4.4 zeek.org +short 192.0.78.212 192.0.78.150 # Running tcpdump on the target instance root@aws-tmt:/home/ubuntu# tcpdump -n -i ens5 'udp port 4789' tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ens5, link-type EN10MB (Ethernet), snapshot length 262144 bytes 17:38:58.959337 IP 172.31.38.236.65512 > 172.31.45.253.4789: VXLAN, flags [I] (0x08), vni 4711 IP 172.31.38.236.46496 > 8.8.4.4.53: 40021+ [1au] A? zeek.org. (49) 17:38:58.999909 IP 172.31.38.236.65512 > 172.31.45.253.4789: VXLAN, flags [I] (0x08), vni 4711 IP 8.8.4.4.53 > 172.31.38.236.46496: 40021 2/0/1 A 192.0.78.212, A 192.0.78.150 (69)
As we can see, tcpdump outputs information about the IP/UDP+VXLAN encapsulation as well as the encapsulated DNS requests and response packets. We also see the VXLAN VNI 4711. This confirms the traffic mirror session is functional. Yay!
Setting up Zeek
Now that we verified the traffic mirroring setup, we create a zeek user on the target system and install Zeek using the binary packages hosted at the OpenSuse Build Service repositories. Additionally, we install build dependencies for the zeek-packet-source-udp and zeek-kafka plugins and subsequently install these using Zeek’s package manager zkg.
Creating the zeek User
root@aws-tmt:/home/ubuntu# useradd --system --user-group --no-create-home --shell /bin/bash zeek
Installing Zeek
# Repos and keys root@aws-tmt:/home/ubuntu# echo 'deb http://download.opensuse.org/repositories/security:/zeek/xUbuntu_24.04/ /' | sudo tee /etc/apt/sources.list.d/security:zeek.list deb http://download.opensuse.org/repositories/security:/zeek/xUbuntu_24.04/ / root@aws-tmt:/home/ubuntu# curl -fsSL https://download.opensuse.org/repositories/security:zeek/xUbuntu_24.04/Release.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/security_zeek.gpg > /dev/null # Package installation root@aws-tmt:/home/ubuntu# sudo apt update && apt install cmake g++ kcat librdkafka-dev zeek-8.0-zkg zeek-8.0-core-dev zeekctl-8.0 ...
Installing Zeek Plugins
root@aws-tmt:/home/ubuntu# /opt/zeek/bin/zkg install zeek-packet-source-udp ... root@aws-tmt:/home/ubuntu# /opt/zeek/bin/zkg install zeek-kafka ...
To verify the installation of Zeek and the plugins was successful, run zeek -NN to list all installed plugins and their provided components. You should see Zeek::PacketSourceUDP and Seiso::Kafka show at the end of the list:
/opt/zeek/bin/zeek -NN
...
Seiso::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
[Writer] KafkaWriter (Log::WRITER_KAFKAWRITER)
...
Zeek::PacketSourceUDP - A packet source listening on a UDP socket.(dynamic, version 0.1.0)
[ConnKey Factory] PACKETSOURCE_UDP_GENEVE_VNI_FIVETUPLE
(CONNKEY_PACKETSOURCE_UDP_GENEVE_VNI_FIVETUPLE, enabled)
...
[Packet Source] UDP (interface prefix "udp"; supports live input)
Populating Zeek’s node.cfg
As the zeek user, edit /opt/zeek/etc/node.cfg to configure the Zeek cluster that will run on the aws-tmt host. We’ll use a single logger, manager, proxy and two worker processes.
[manager] type=manager host=localhost [logger-1] type=logger host=localhost [proxy-1] type=proxy host=localhost [worker-1] type=worker host=localhost interface=udp::0.0.0.0:4789:vxlan lb_procs=2 lb_method=custom
The node.cfg file is from an era where it was commonplace to run Zeek clusters spanning multiple physical nodes, sometimes running just a single process per physical system. In our scenario, all Zeek processes run on the same system, so it may appear a bit verbose on first sight.
With Zeek 8.1, there’s a new zeek-systemd-generator and a zeek.conf file that makes it much easier to deploy a single node cluster. It’s not ready for multi node deployments, however.
# zeek.conf for an equivalent single node cluster interface=udp::0.0.0.0:4789:vxlan workers = 2 $ ln -s /opt/zeek/bin/zeek-systemd-generator /etc/systemd/ system-generators/ $ systemctl daemon-reload && systemctl restart zeek.target
Zeek’s local.zeek
We add the following lines to the end of Zeek’s local.zeek file located at /opt/zeek/share/zeek/site/local.zeek. This instructs Zeek to send all logs in JSON format to the hosted Kafka instance using certificate-based authentication instead of writing logs to the local disk. If you don’t care about sending logs to Kafka, you may comment out the corresponding lines. In a default Zeekctl setup, log files are produced into the /opt/zeek/logs/ directory.
# Additions to local.zeek redef LogAscii::use_json = T; redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "kafka-zeek-demo-zeek-0b1f.l.aivencloud.com:10147", ["security.protocol"] = "SSL", ["ssl.key.location"] = "/opt/zeek/etc/ssl/certs/zeek.aivencloud.key", ["ssl.certificate.location"] = "/opt/zeek/etc/ssl/certs/zeek.aivencloud.cert", ["ssl.ca.location"] = "/opt/zeek/etc/ssl/certs/zeek.aivencloud.ca", ); redef Kafka::send_all_active_logs = T; redef Kafka::topic_name = ""; # Use Kafka for logging - comment this out if you want to write to the local disk. redef Log::default_writer = Log::WRITER_KAFKAWRITER; # Include VXLAN VNI in conn_id$ctx record. @load policy/frameworks/conn_key/packet_source/udp/vxlan_vni
The configured Kafka user has permissions for automatic topic creation. This means the first log record written to a given path will create the topic in the Kafka cluster. Some Kafka setups default this to false. In general, automatic topic creation is a bit of a gotcha to watch out for here. This demo uses a one-off Kafka instance hosted by Aiven, but should work with Amazon MSK or any self-hosted Kafka instance as well. It’s also useful to have kcat using a config file available for debugging.
Deploy and Start Zeek
To deploy Zeek, we run zeekctl deploy as the zeek user. The output of zeekctl status confirms that all processes were started.
zeek@aws-tmt:/opt/zeek$ /opt/zeek/bin/zeekctl deploy checking configurations ... installing ... ... starting ... starting logger ... starting manager ... starting proxy ... starting workers ... zeek@aws-tmt:/opt/zeek$ /opt/zeek/bin/zeekctl status Name Type Host Status Pid Started logger-1 logger localhost running 45072 16 Feb 13:28:47 manager manager localhost running 45133 16 Feb 13:28:48 proxy-1 proxy localhost running 45201 16 Feb 13:28:50 worker-1-1 worker localhost running 45270 16 Feb 13:28:51 worker-1-2 worker localhost running 45269 16 Feb 13:28:51
We can also double check the worker processes to ensure they are using the correct interface definition:
zeek@aws-tmt:/opt/zeek$ ps -f -p 45269,45270 UID PID PPID C STIME TTY TIME CMD zeek 45269 45257 0 13:28 ? 00:00:06 /opt/zeek/bin/zeek -i udp::0.0.0.0:47 89:vxlan -U .status -p zeekctl -p zeekctl-live -p local -p worker-1-2 local.zeek zeekctl base/frameworks/cluster zeekctl/auto zeek 45270 45258 0 13:28 ? 00:00:05 /opt/zeek/bin/zeek -i udp::0.0.0.0:47 89:vxlan -U .status -p zeekctl -p zeekctl-live -p local -p worker-1-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto
Setting up the Zeekctl Cronjob
A crashed Zeek process is not automatically restarted when using Zeekctl. The mechanism to do so with Zeekctl is a plain old cronjob running regularly. As the zeek user, run crontab -e and ensure the following command is part of the crontab file:
# m h dom mon dow command
* * * * * /opt/zeek/bin/zeekctl cron
Verification
On the source host, we run the following ad-hoc while loop to generate some mirror traffic towards the target:
ubuntu@aws-tms:~$ while true ; do dig @8.8.4.4 zeek.org +short ; sleep 1 ; curl -s -L http://zeek.org >/dev/null ; sleep 1; done
On the target host, we can use kcat to consume from the end of the conn topic and stream the produced log records to stdout. The kcat-cert.config file contains the same credentials as those used by Zeek and placed in the local.zeek file:
ubuntu@aws-tmt:~$ kcat -F kcat-cert.config -t conn -o end
{"ts":1771248545.97429,"uid":"CIypzy311X5bnrKYLf","id.orig_h":"172.31.38.236","id.orig_p":
34615,"id.resp_h":"8.8.4.4","id.resp_p":53,"id.ctx.vxlan_vni":4711,"proto":"udp","service"
:"dns","duration":0.018145084381103516,"orig_bytes":49,"resp_bytes":69,"conn_state":"SF",
"local_orig":true,"local_resp":false,"missed_bytes":0,"history":"Dd","orig_pkts":1,"orig_
ip_bytes":77,"resp_pkts":1,"resp_ip_bytes":97,"ip_proto":17}
{"ts":1771248551.369471,"uid":"C2BP0mdRieaCGCnw6","id.orig_h":"172.31.38.236","id.orig_p":
44088,"id.resp_h":"192.0.78.150","id.resp_p":443,"id.ctx.vxlan_vni":4711,"proto":"tcp","se
rvice":"ssl","duration":0.10626697540283203,"orig_bytes":795,"resp_bytes":178878,"conn_sta
te":"SF","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"ShADadFf","orig_
pkts":53,"orig_ip_bytes":2935,"resp_pkts":131,"resp_ip_bytes":184130,"ip_proto":6}
{"ts":1771248551.348247,"uid":"CSrgUt4ZI5hNs9BVti","id.orig_h":"172.31.38.236","id.orig_p"
:35458,"id.resp_h":"192.0.78.150","id.resp_p":80,"id.ctx.vxlan_vni":4711,"proto":"tcp","se
rvice":"http","duration":0.1283268928527832,"orig_bytes":71,"resp_bytes":441,"conn_state":
"SF","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"ShADadFf","orig_pkts"
:6,"orig_ip_bytes":331,"resp_pkts":4,"resp_ip_bytes":613,"ip_proto":6}
If you suspect issues with the Kafka setup, or want to log to files on the target system instead of to Kafka, remove the Kafka specific lines from local.zeek and find logs in the /opt/zeek/logs/current directory. The zeek-kafka plugin also supports logging records to a single topic instead of a topic per log stream. Some downstream consumers prefer that over a topic per log stream.
ConnKey Framework Integration
If you look closely at the conn entries above, you can see that the VXLAN VNI 4711 configured for the traffic mirror session is included as id.ctx.vxlan_id field in the record. This is the result of loading the policy/frameworks/conn_key/packet_source/udp/vxlan_vni script. Not only does this enable logging of the VXLAN VNI, but it also switches Zeek’s connection key implementation to one that includes the VXLAN VNI. Further, Zeek scripts have access to the vxlan_id value for a given connection c via c$id$ctx$vxlan_id. This allows disambiguating overlapping IP ranges when mirroring traffic with non-related but conflicting IP ranges to a single Zeek cluster deployment. The ConnKey framework can be particularly interesting for multi-tenancy deployments.
Summary
In this blog post we showed how to leverage Zeek’s UDP-based packet source in an AWS Traffic Mirroring environment and how to forward Zeek logs to a hosted Kafka instance. Once Zeek logs are available within Kafka, they can be forwarded via connectors to a SIEM or other data platform for analysis and threat hunting investigations.
RSS - Posts