As a quick followup to Part1, I want to bring attention to the Reminder about Events and Module Namespaces. This is something that I’d forgotten about (or never knew) that will save you a lot of headaches when converting scripts to use Broker. While I’m at it, when testing data stores, it’s important to keep in mind any data expiration times you have set. More than once I got back to testing the next morning and wondered where my persistent data had gone only to eventually realize it had just been correctly expired.
In Part 1, I discussed using Broker to interact with persistent data stores, so now I want to go over a couple options for synchronizing data across your Bro cluster. We’ll also look at how to debug Broker communication to verify things are working as you expected. As a reminder, below I’ve included a basic example of the old method of using &sychronized, which is now depreciated.
Publish Data to the Manager
There are several common use cases for Broker communication and it comes down to where your data needs to be for your policy to work as expected. Do the workers need access to the full view of the data or can the proxies or manager make the decisions? The first case we’ll examine is using Broker to publish the addresses to the manager, check out the &sychronized example near the bottom to see the script we started from.
Broker uses a pub/sub model in combination with events to communicate between nodes so we need to send a message to the recipient, but also define the event that will be processed as a result. If you aren’t familiar with pub/sub and the concept of topics, here is an overview.
@load base/frameworks/cluster
module EX;
export {
global known_clients: set[addr] &create_expire=1day &synchronized;
global add_client: event(newaddr: addr);
}
event add_client(newaddr: addr){
if(newaddr !in known_clients){
add known_clients[newaddr];
}
}
event connection_established(c: connection){
if(c$id$resp_h == 192.150.187.43){
Broker::publish(Cluster::manager_topic,EX::add_client,
c$id$orig_h);
}
}
The event for adding the address to the known_clients set will be handled by the manager, so we moved the check for existence in the set to just before adding the client data. The event definition is exported for global use. Then in the connection_established event, we replace the add to the set with our Broker::publish call. We need to specify the pub/sub topic, but Bro pre-defines Cluster::manager_topic for us already and the manager is automatically subscribed to it. The only other arguments are the event we want the manager to execute, along with any arguments. Don’t forget to remove &synchronized!
Re-deploying the new script on the cluster, let’s see the result:
[bro@host ~]$ broctl deploy
[bro@host ~]$ broctl print EX::known_clients
logger EX::known_clients = {
}
manager EX::known_clients = {
198.128.111.111
}
proxy-1 EX::known_clients = {
}
proxy-2 EX::known_clients = {
}
worker-1 EX::known_clients = {
}
worker-2 EX::known_clients = {
}
worker-3 EX::known_clients = {
}
worker-4 EX::known_clients = {
}
This method of offloading work to the manager has pretty clear tradeoffs. The workers aren’t all storing copies of the set in memory and they also don’t have to do any additional processing work. However, you’re putting that load on the manager. There may be a better way, which we’ll look at when we discuss distributing work among the proxies.
Making it Work in Standalone Mode
If you do any testing by running Bro in a standalone mode, or would like your policies to support people who do, you’ll need to do a little more work to make your script do both.
event connection_established(c: connection){
if(c$id$resp_h == 192.150.187.43){
@if( Cluster::is_enabled())
Broker::publish(Cluster::manager_topic,EX::add_client,
c$id$orig_h);
@else
event EX::add_client(c$id$orig_h);
@endif
}
}
Sometimes you’ll see additional checks for which kind of cluster node is processing the event, but that’s an added layer of complication we won’t discuss right now.
Example:
@if ( ! Cluster::is_enabled() || Cluster::local_node_type() == Cluster::MANAGER )
For the rest of these examples I will skip the standalone code to make things simpler.
Distribute Work Among Proxies
Another option when the workers don’t need access to the data is to distribute the processing amongst the proxy nodes. We could publish the data as we did with the manager, but as there may be more than one proxy, we don’t want them to multiply the effort. Conveniently, Bro provides a “highest random weight” hashing algorithm to spread the load evenly. All we have to do is change the Broker::publish line:
event connection_established(c: connection){
if(c$id$resp_h == 192.150.187.43){
Cluster::publish_hrw(Cluster::proxy_pool, c$id$orig_h,
EX::add_client, c$id$orig_h);
}
}
The Cluster::publish_hrw function distributes events across a set of nodes, here we’re using the proxy pool. The second argument is the value that will be hashed to determine the distribution. Finally, we give it the event to send along with any arguments.
[bro@host ~]$ broctl deploy
[bro@host ~]$ wget -q https://www.bro.org
[bro@host ~]$ broctl print EX::known_clients
logger EX::known_clients = {
}
manager EX::known_clients = {
}
proxy-1 EX::known_clients = {
}
proxy-2 EX::known_clients = {
198.128.111.111
}
worker-1 EX::known_clients = {
}
worker-2 EX::known_clients = {
}
worker-3 EX::known_clients = {
}
worker-4 EX::known_clients = {
}
Note that it only wrote to one of the proxies as expected. If you test the policy from multiple clients you should end up with a fairly even distribution across proxies. Obviously, all we’re doing in the example is adding an address to a set, but in policies where additional logic is needed, this could be a huge relief off both the workers and the manager. A good example of this usage is in the core policies known-hosts.bro, known-services.bro, and known-certs.bro.
Syncing Data Between Workers
Instead of the manager or proxies, let’s imagine we want the workers to all have the same data so they can make decisions on it immediately. We need to be cautious with this method because it means duplicating the memory footprint across all workers, but there are cases where being able to lookup data in a set or table will save more additional work.
Workers aren’t connected directly to each other so in order to publish data to them we need to relay it through either the manager or the proxies. To do this, we need to establish two things: a topic for the worker to publish to and we need to handle that event on the manager or proxies to forward the event to the other workers. I’ll be using a method that can be seen in the core Bro distribution as part of scripts/base/protocols/irc/dcc-send.bro.
First let’s discuss the Broker topic. We could relay the event through the master node, but ideally we should keep the load on the proxies as that’s what they’re for. It would also be nice to distribute the load evenly across all available proxies which we can do using a provided round robin function, Cluster::rr_topic(). The result is that we can build a new function that determines which proxy or manager topic to use:
function example_relay_topic(): string{
local rval = Cluster::rr_topic(Cluster::proxy_pool, “example_rr_key”);
if ( rval == “” )
# No proxy is alive, so relay via manager instead.
return Cluster::manager_topic;
return rval;
}
This simple function tries to round robin through our proxies, but if for some reason we don’t have any available, it will fall back to using the manager node. The second parameter is just a key that the Cluster framework uses internally. With the topic deciding function in place, we can modify the core Broker::publish line.
event connection_established(c: connection){
if(c$id$resp_h == 192.150.187.43){
Broker::publish(example_relay_topic(),
EX::add_client, c$id$orig_h);
}
}
Note that all we’ve really done at this point is send the EX::add_client event to one of the proxies, we still need to have them forward it to the workers. That can be accomplished within that same event:
event add_client(newaddr: addr){
@if ( Cluster::local_node_type() == Cluster::PROXY ||
Cluster::local_node_type() == Cluster::MANAGER )
Broker::publish(Cluster::worker_topic,EX::add_client,newaddr);
@else
if(newaddr !in known_clients){
add known_clients[newaddr];
}
@endif
}
And then we can test just like before:
[bro@host ~]$ broctl print EX::known_clients
logger EX::known_clients = {
}
manager EX::known_clients = {
}
proxy-1 EX::known_clients = {
}
proxy-2 EX::known_clients = {
}
worker-1 EX::known_clients = {
198.128.111.111
}
worker-2 EX::known_clients = {
198.128.111.111
}
worker-3 EX::known_clients = {
198.128.111.111
}
worker-4 EX::known_clients = {
198.128.111.111
}
So there we have it, the known client address was distributed to all of the workers.
Debugging Broker Communication
Clearly there’s a lot more to Broker than I’ve covered here and it can start to get confusing so let’s look at a way to dig deeper into where these messages are going. To start, you’ll need to recompile Bro from source and include the –enable-debug flag.
Now we need to pass the -B broker argument to bro which we can do by setting BroArgs in $BROPATH/etc/broctl.cfg. (Other debugging options can be see with bro -B help.)
[bro@host ~]$ head -n 4 /usr/local/bro/etc/broctl.cfg
## Global BroControl configuration file.
BroArgs = -B broker
[bro@host ~]$ broctl deploy
We need to look at the debug logs for the individual nodes which will be found in the $BROPATH/spool/[nodename]/ directories while the cluster is running. Let’s look at the proxies first:
[bro@host ~]$ grep add_client /usr/local/bro/spool/proxy*/debug.log
/usr/local/bro/spool/proxy-1/debug.log:1530925958.459474/1530925958.460614 [broker] Process event: EX::add_client: [198.128.111.111]
/usr/local/bro/spool/proxy-1/debug.log:1530925958.459474/1530925958.460633 [broker] Publishing event: EX::add_client([198.128.111.111]) -> bro/cluster/worker
Here we can see where proxy-1 received the relay event and then published the event to the workers. The second proxy never saw anything because of the round robin nature of the relay. The logs also contain the event called and arguments which can be very useful. Now the workers:
[bro@host ~]$ grep add_client /usr/local/bro/spool/worker*/debug.log
/usr/local/bro/spool/worker-1/debug.log:1530925958.531330/1530925958.531552 [broker] Process event: EX::add_client [198.128.111.111]
/usr/local/bro/spool/worker-2/debug.log:1530925958.532718/1530925958.533077 [broker] Process event: EX::add_client [198.128.111.111]
/usr/local/bro/spool/worker-3/debug.log:1530925958.527955/1530925958.528265 [broker] Process event: EX::add_client [198.128.111.111]
/usr/local/bro/spool/worker-4/debug.log:1530925958.531130/1530925958.531427 [broker] Process event: EX::add_client [198.128.111.111]
As you can see, all of the workers received the event as expected.
&synchronized
As mentioned at the start, I’m including this section simply as a reminder of what a simple policy would have looked like with the old &synchronized attribute.
@load base/frameworks/cluster
module EX;
export {
global known_clients: set[addr] &create_expire=1day &synchronized;
}
event connection_established(c: connection){
# bro.org clients
if(c$id$resp_h == 192.150.187.43 && c$id$orig_h !in known_clients){
add known_clients[c$id$orig_h];
}
}
All we’re doing is creating a set of addresses that we see as clients connecting to bro.org. To test this, I’ve setup a cluster with four workers and two proxies (realistically we only need one, but two will be important for a later example). We can use broctl print on the command line to check values on a running cluster:
[bro@host ~]$ wget -q https://www.bro.org
[bro@host ~]$ broctl print EX::known_clients
logger EX::known_clients = {
}
manager EX::known_clients = {
}
proxy-1 EX::known_clients = {
}
proxy-2 EX::known_clients = {
}
worker-1 EX::known_clients = {
198.128.111.111
}
worker-2 EX::known_clients = {
}
worker-3 EX::known_clients = {
}
worker-4 EX::known_clients = {
}
We made one connection and only one of the workers shows an address in the set proving that &synchronized is depreciated.
Wrap-Up
I hope this helps folks get started with Broker. There’s so much more to learn with Broker, but I felt it was a good idea to help ease the transition with the most common aspects.
Like this:
Like Loading...
RSS - Posts