Overview
Intelligence data, or feeds, are an important source of network security information. Many internet security research centers, non-profit organizations, and commercial organizations provide intellegence data sets freely
available to the public. (e.g. Emerging Threats, Shadow Server, etc.)
A solid solution for handling multiple intelligence feeds and acting upon them is to use Bro’s Intel Framework in conjunction with its Input Framework to log hits seen on your network.
Hits, or matches, are logged and stored in intel.log.
$ head /bro/logs/current/intel.log #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc seen.indicator seen.indicator_type seen.where sources #types time string addr port addr port string string string string enum enum table[string] 1389646914.469513 Cb1T1Y3r56rjSRr0ac 70.173.122.62 49379 93.184.215.163 80 - - - 93.184.215.163 Intel::ADDR Conn::IN_RESP snort 1389646925.095172 CeHuLr2IfATgjhQPUg 195.178.109.14 43471 227.146.143.88 5900 - - - 195.178.109.14 Intel::ADDR Conn::IN_ORIG CIF - need-to-know 1389646979.298695 Cj9bjq43KThmM8TOA6 194.160.95.196 60430 119.254.16.21 80 - - - www.dhgate.com Intel::DOMAIN HTTP::IN_HOST_HEADER CIF - need-to-know 1389636016.229788 CNXlzu1VdMhKVEDCPf 42.82.115.14 52518 184.72.106.52 443 - - - 184.72.106.52 Intel::ADDR Conn::IN_RESP tor 1389636187.690988 CsGYCjrhwI313rWob 72.49.141.144 41270 118.219.221.39 3389 - - - 72.49.141.144 Intel::ADDR Conn::IN_ORIG ciarmy ...
Each log entry consists of a number of fields describing the connection:
ts: Timestamp uid: Unique ID of connection id.orig_h: Connection originator’s endpoint IP address id.orig_p: Connection originator’s endpoint TCP/UDP or ICMP code id.resp_h: Connection responder’s endpoint IP address id.resp_p: Connection responder’s endpoint TCP/UDP or ICMP code fuid: File indentifier (if file was found in connection) file_mime_type: Libmagic file type (e.g. application/x-dosexec) file_desc: Optional file type description seen.indicator: The indicator that triggered the match (e.g. IP) seen.indicator_type: The type of indicator (e.g. ADDR, DOMAIN) seen.where: Location in Bro’s where the event triggered (e.g. DNS::REQUEST) sources: Intel data source description (e.g. emergingthreats.net)
Intel Framework
The Intel Framework provides the facilities to handle intelligence data in a meaningful manner (e.g. categorization, types, source, etc.). [1]
Supported types of intelligence data indicators are:
Intel::ADDR Intel::URL Intel::SOFTWARE Intel::EMAIL Intel::DOMAIN Intel::USER_NAME Intel::FILE_HASH Intel::FILE_NAME Intel::CERT_HASH
Before we can use the framework we must load it by adding the following lines to local.bro or equivalent:
@load frameworks/intel/seen @load frameworks/intel/do_notice
The next step is to properly format the data set. Each field must be separated by a single tab character.
#fields indicator indicator_type meta.source meta.url meta.do_notice meta.if_in 1.182.117.119 Intel::ADDR ciarmy http://www.ciarmy.com/list/ci-badguys.txt T - 2.187.28.215 Intel::ADDR ciarmy http://www.ciarmy.com/list/ci-badguys.txt T - 2.229.117.159 Intel::ADDR ciarmy http://www.ciarmy.com/list/ci-badguys.txt T - 4.35.96.216 Intel::ADDR ciarmy http://www.ciarmy.com/list/ci-badguys.txt T - 5.79.69.204 Intel::ADDR ciarmy http://www.ciarmy.com/list/ci-badguys.txt T - 5.135.146.0 Intel::ADDR ciarmy http://www.ciarmy.com/list/ci-badguys.txt T - 5.135.240.133 Intel::ADDR ciarmy http://www.ciarmy.com/list/ci-badguys.txt T - 5.153.54.130 Intel::ADDR ciarmy http://www.ciarmy.com/list/ci-badguys.txt T - 5.254.101.69 Intel::ADDR ciarmy http://www.ciarmy.com/list/ci-badguys.txt T -
Also, each field has a particular type of value. Two worthy of mention are meta.do_notice and meta.if_in because they work with the Notice Framework. meta.do_notice expects a boolean value which determines whether matches will be sent to the Notice Framework where entries will end up in notice.log, in addition to the normal intel.log.
$ grep Intel notice.log 1389636016.229788 CNXlzu1VdMhKVEDCPf 23.58.90.219 52518 184.72.106.52 443 - - - tcp Intel::Notice Intel hit on 184.72.106.52 at Conn::IN_RESP 184.72.106.52 23.58.90.219 184.72.106.52 443 1389636016.517027 Cr1u7g2pFxgSzHBsRf 234.136.242.173 52519 184.72.106.52 443 - - - tcp Intel::Notice Intel hit on 184.72.106.52 at Conn::IN_RESP 184.72.106.52 234.136.242.173 184.72.106.52 443 ...
meta.if_in concerns itself with where a particular match was found in network traffic (e.g. HTTP::IN_HOST_HEADER) and serves as a restriction for matches sent to the Notice Framework i.e. they must be found in the location of meta.if_in.
Note: meta.if_in takes a single location value. If you would to like to specify more than one location for an indicator you will need an entry for each location you desire to be seen. e.g.
#fields indicator indicator_type meta.source meta.url meta.do_notice meta.if_in aolon1ine.com Intel::DOMAIN mandiant - T DNS::IN_REQUEST aolon1ine.com Intel::DOMAIN mandiant - T HTTP::IN_HOST_HEADER aolon1ine.com Intel::DOMAIN mandiant - T SMTP::IN_FROM
The list of possible meta.if_in locations are: [2]
Conn::IN_ORIG Conn::IN_RESP Files::IN_HASH Files::IN_NAME DNS::IN_REQUEST DNS::IN_RESPONSE HTTP::IN_HOST_HEADER HTTP::IN_REFERRER_HEADER HTTP::IN_USER_AGENT_HEADER HTTP::IN_X_FORWARDED_FOR_HEADER HTTP::IN_URL SMTP::IN_MAIL_FROM SMTP::IN_RCPT_TO SMTP::IN_FROM SMTP::IN_TO SMTP::IN_RECEIVED_HEADER SMTP::IN_REPLY_TO SMTP::IN_X_ORIGINATING_IP_HEADER SMTP::IN_MESSAGE SSL::IN_SERVER_CERT SSL::IN_CLIENT_CERT SSL::IN_SERVER_NAME SMTP::IN_HEADER
Input Framework
The Input Framework [3] provides the facilities to read information from a text file, and in our case send it to the Intel Framework for processing.
Note: If you’re running a Bro cluster, the intelligence files only need to be stored and read on the manager.
An absolute file path is required to read from a file. Three ways to do this are:
-
Specify the file’s full path
-
Append the file name to @DIR which stores the directory path of the calling script. If the script with Intel::read_files is not in the same directory as the intel data sets this will not work.
-
Create a constant that’s value is the directory path in local.bro or equivalent.
const feed_directory = "/opt/bro/feeds";
Here’s an example using all three methods:
redef Intel::read_files += {
@DIR + "/botcc.intel",
@DIR + "/ciarmy.intel",
"/opt/bro/feeds/malhosts.intel",
"/opt/bro/feeds/malips.intel",
feed_directory + "/tor.intel",
feed_directory + "/snort.intel",
};
Obtaining Feeds
Two solutions for obtaining pre-formatted feeds for Bro to use are mal-dns2bro, a helper script for mal-dnssearch, or the Collective Intelligence Framework and its Bro output plug-in.
Mal-dns2bro
Mal-dnssearch [4] is a shell script I wrote that downloads, parses, and compares intelligence feeds against a number of popular application log files, reporting any matches. mal-dns2bro [5] is a helper script included with mal-dnssearch that formats feeds for Bro’s Intel Framework to extend the application of intelligence data directly against live network traffic. mal-dns2bro has the ability to customize Intel Framework fields like setting meta.source, meta.url, meta.do_notice, and meta.if_in.
mal-dnssearch supports a number of feeds, “mal-dnssearch -h”:
Malware List Options: -M <list> Name of list, e.g. ``-M snort'' List: | Description: snort - http://labs.snort.org/feeds/ip-filter.blf (IP) et_ips - http://rules.emergingthreats.net/open/suricata/rules/compromised-ips.txt (IP) alienvault - http://reputation.alienvault.com/reputation.generic (BIG file) (IP) botcc - http://rules.emergingthreats.net/open/suricata/rules/botcc.rules (IP) tor - http://rules.emergingthreats.net/open/suricata/rules/tor.rules (IP) rbn - http://rules.emergingthreats.net/blockrules/emerging-rbn.rules (IP) malhosts - http://www.malwaredomainlist.com/hostslist/hosts.txt (DNS) malips - http://www.malwaredomainlist.com/hostslist/ip.txt (IP) ciarmy - http://www.ciarmy.com/list/ci-badguys.txt (IP) mayhemic - http://secure.mayhemiclabs.com/malhosts/malhosts.txt (DNS) mandiant - https://raw.github.com/jonschipp/mal-dnssearch/master/mandiant_apt1.dns (DNS)
Download and install mal-dnssearch:
$ git clone https://github.com/jonschipp/mal-dnssearch $ cd mal-dnssearch $ sudo make install
Download Mandiant’s APT1 data set, parse, and pipe (“-p”) it to mal-dns2bro for formatting:
$ mal-dnssearch -M mandiant -p | mal-dns2bro -T dns -s mandiant > mandiant.intel
Sample file output:
#fields indicator indicator_type meta.source meta.url meta.do_notice meta.if_in advanbusiness.com Intel::DOMAIN mandiant - F - aoldaily.com Intel::DOMAIN mandiant - F - aolon1ine.com Intel::DOMAIN mandiant - F - applesoftupdate.com Intel::DOMAIN mandiant - F -
In the example above, mal-dns2bro reads in the mandiant list from stdin and sets the indicator type (“-T”) to DNS because the mandiant list consists of only DNS names. The source (“-s”) field is also set which is a short description of where the intelligence data came from.
mal-dns2bro will add the necessary tab separated columns for the Intel Framework. It accepts a list of a specific indicator type, but supports all of them, with one entry per line. It can read from stdin or from a file (“-f”). If you don’t want to use mal-dnssearch, you can create your own lists with a text editor or other program and have mal-dns2bro format them for Bro.
Another example, downloading a list of tor nodes and customizing all fields:
$ mal-dnssearch -M tor -p | mal-dns2bro -T ip -s tor -n true -u http://rules.emergingthreats.net/open/suricata/rules/tor.rules > tor.intel
Sample file output:
#fields indicator indicator_type meta.source meta.url meta.do_notice meta.if_in 103.10.197.50 Intel::ADDR tor http://rules.emergingthreats.net/open/suricata/rules/tor.rules T Conn::IN_RESP 103.3.188.169 Intel::ADDR tor http://rules.emergingthreats.net/open/suricata/rules/tor.rules T Conn::IN_RESP 105.237.43.166 Intel::ADDR tor http://rules.emergingthreats.net/open/suricata/rules/tor.rules T Conn::IN_RESP 106.186.21.31 Intel::ADDR tor http://rules.emergingthreats.net/open/suricata/rules/tor.rules T Conn::IN_RESP
In this last example, two new options are introduced: mal-dns2bro sets meta.do.notice to true (“-n”) which will send intel matches to the notice framework: the source URL, when applicable, is set (“-u”).
Collective Intelligence Framework
The Collective Intelligence Framework (CIF) is a cyber threat intelligence management system that pulls and stores feeds into a database for querying with CIF’s tools. [6] CIF comes with a number of output plug-ins including one for Bro.
Installing CIF is pretty involved so I will refer the reader to its documentation. [7]
The CIF plug-in for Bro outputs a few different fields than we’ve seen so far: meta.cif_impact, meta.cif_severity, and meta,cif_confidence. Because of this we will need to load a Bro script, in addition to those required for the Intel Framework, that can handle these.
Add the following line to local.bro or equivalent:
@load policy/integration/collective-intel
Now, lets output domains related to botnets with a confidence level of 85 or greater formatted for Bro.
$ cif -q domain/botnet -c 85 -p bro > domain_botnet.intel
Sample file output:
#fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence computo164.laweb.es Intel::DOMAIN CIF - need-to-know palevo https://palevotracker.abuse.ch/?host=computo164.laweb.es (public) - high 85 ms4all.twoplayers.net Intel::DOMAIN CIF - need-to-know palevo https://palevotracker.abuse.ch/?host=ms4all.twoplayers.net (public) - high 85 hcuewgbbnfdu1ew.com Intel::DOMAIN CIF - need-to-know palevo https://palevotracker.abuse.ch/?host=hcuewgbbnfdu1ew.com (public) - high 85 arta.romail3arnest.info Intel::DOMAIN CIF - need-to-know palevo https://palevotracker.abuse.ch/?host=arta.romail3arnest.info (public) - high 85
Create a list of machines known for scanning the internet:
$ cif -q infrastructure/scan -c 85 -p bro > intrastructure_scan.intel
Sample file output:
#fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence 192.74.251.115 Intel::ADDR CIF - need-to-know ssh http://danger.rulez.sk/projects/bruteforceblocker/blist.php (public) - medium 85 117.41.247.212 Intel::ADDR CIF - need-to-know ssh http://danger.rulez.sk/projects/bruteforceblocker/blist.php (public) - medium 85 133.242.171.75 Intel::ADDR CIF - need-to-know ssh http://danger.rulez.sk/projects/bruteforceblocker/blist.php (public) - medium 85
E-mailing Notices
This section presumes that Intel events are sent to the Notice Framework [8], described in the Intel section of this article.
Each time a Intel::Match event is generated, the intel data for that match is sent to the Notice Framework where a Notice is raised that has a type of Intel::Notice. To receive e-mail notifications upon a match add the following Notice type to emailed_types in local.bro or equivalent:
redef Notice::emailed_types += {
Intel::Notice,
};
Suppression can be used to reduce the number of events generated for a specific Intel match. To limit each particular match to once per day add the following lines to local.bro or equivalent:
redef Notice::type_suppression_intervals += {
[Intel::Notice] = 1day,
};
Log Queries
Bro-cut can be used to print fields in Bro logs. The following example will print a list of indicators only i.e. the host, hash, e-mail, etc. that triggered the match.
$ bro-cut seen.indicator < intel.log 61.36.24.57 193.107.16.206 192.0.72.2 ...
To print a unique list of all ADDR types and the number of times they were seen try the following query. If you count be the number of fields 11 is seen.indicator_type and field 10 is seen.indicator:
$ awk '$11 == "Intel::ADDR" { print $10 }' intel.log | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -k 4,4 | uniq -c
190 61.36.24.57
1 93.120.27.62
3 93.184.215.163
1 96.45.82.69
7 192.0.72.2
2 193.107.16.206
The next example illustrates an intel match found in the host field of the HTTP header.
1390425478.178039 CQzyg512Q8Mm2UrTc4 53.80.23.123 51220 65.52.128.33 80 - - - tipranks.com Intel::DOMAIN HTTP::IN_HOST_HEADER malhosts
Since the above connection is using HTTP there are bound to be more logs related to the connection such as those produced by the HTTP analyzer. We can search all Bro’s logs for the connection identifier, a 4-tuple flow hash, to find other logs related to that connection.
$ zgrep CQzyg512Q8Mm2UrTc4 *.log.gz 1390425478.076356 CQzyg512Q8Mm2UrTc4 53.80.23.123 51220 65.52.128.33 80 tcp http 123.242140 397 981 RSTR T 0 ShADadr 12 1302 12 2514 (empty) US US nids-31-2 1390425478.398364 FQSaaGSHQsLn11w8j 65.52.128.33 53.80.23.123 CQzyg512Q8Mm2UrTc4 HTTP 0 SHA1,MD5 text/plain - 0.000000 F F 595 - 0 0 F - d6923c591dfa3cb616327a0bb44375b3 aa5438c500083ee69297d8c1a2ab4f82655c4632 - - 1390425478.179990 CQzyg512Q8Mm2UrTc4 53.80.23.123 51220 65.52.128.33 80 1 GET tipranks.com /valuewalk/tipranks.js http://www.valuewalk.com/2014/01/advanced-micro-devices-inc-amd-shares-dive-after-earnings/ Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36 0 595 200 OK - - - (empty) - - - - - FQSaaGSHQsLn11w8j text/plain 1390425478.178039 CQzyg512Q8Mm2UrTc4 53.80.23.123 51220 65.52.128.33 80 - - - 65.52.128.33 Intel::ADDR Conn::IN_RESP malips 1390425478.178039 CQzyg512Q8Mm2UrTc4 53.80.23.123 51220 65.52.128.33 80 - - - tipranks.com Intel::DOMAIN HTTP::IN_HOST_HEADER malhosts
We found a number of events detailing the connection, these include the URL the user visited, the type of file that was transferred in the session, and another intel match from a different intel source. This is especially useful in cases where there are more than one HTTP request in a single TCP session.
Putting it all together
A powerful but simple solution is to write a daily cronjob that downloads and formats the latest version of each feed from which Bro continuously reads and then restarts Bro, if necessary. A restart is required if you want to purge entries that have been removed from the feeds, but not if you only want the new entries because Bro keeps the file open and will pick up any new additions.
If you come across any trouble be sure to check reporter.log.
References:
| [1] | http://www.bro.org/sphinx-git/frameworks/intel.html |
| [2] | http://www.bro.org/sphinx-git/_downloads/where-locations.bro |
| [3] | http://www.bro.org/sphinx/frameworks/input.html |
| [4] | https://github.com/jonschipp/mal-dnssearch |
| [5] | https://github.com/jonschipp/mal-dnssearch/blob/master/tools/mal-dns2bro.sh |
| [6] | https://code.google.com/p/collective-intelligence-framework/ |
| [7] | https://code.google.com/p/collective-intelligence-framework/wiki/ServerInstall_v1 |
| [8] | http://www.bro.org/sphinx-git/frameworks/notice.html |
RSS - Posts