Saturday, December 7, 2013

Network Forensics with Bro(s)

I recently had the opportunity to give a short presentation at the BayThreat security conference. I talked about using Bro (because that's all I talk about). The BayThreat CFP requested actionable talks and I thought it was a perfect opportunity to demonstrate Bro.
The presentation, similar to this blog, was geared towards Bro newbies. I presented a general high level methodology for analyzing network trace files and showed how Bro can be used to make steps of the methodology much faster and thorough. The presentation looked at a pcap from a live honeypot system which was part of an online challenge designed to force participants to analyze trace files containing common attacks.

The slides from my presentation can be found here.

First, I introduced Bro and described how it worked. I explained how Bro's core works together with its custom scripting language to handle events that occur on the network.

Next, a basic methodology for analyzing pcap files was presented. It included the following steps:

0. determine context
1. orient and inventory
2. investigate a single artifact from the inventory
3. categorize that single artifact
4. filter the categorized artifact from the inventory
5. repeat until every artifact in the inventory is categorized

Once Bro was explained and the analysis methodology was laid out, I walked through each of the steps in the methodology, applying Bro where possible.

Step 1.
Feeding the pcap to Bro, along with a custom policy script (a tweaked version of the local.bro), produced log files and a directory of files transferred within the pcap. These outputs were used as an inventory.

Step 2.
Searching through the log files was easily accomplished using bro-cut, grep, and other common CLI tools.

Step 3.
SSH bruteforcing, FTP bruteforcing, HTTP crawlers (looking for administrative pages e.g. for phpMyAdmin) were identified and categorized as suspect. Legitimate artifacts were also discovered such as a package update occurring on the honeypot system, an SSH session and anonymous FTP logins.

Step 4.
A Bro script (pseudo code) was presented to reduce the logs Bro creates when reading in a pcap file. By adjusting log output, an analyst would be able to filter the inventory generated in step 1.

Step 5.
Due to time constraints, the remainder of the pcap's inventory was left as an exercise for the audience.


Hopefully everyone who attended the presentation was able to learn and take something away from the material. I encourage you, as readers of this post, to attempt to use Bro to walk through the pcap.

Friday, November 22, 2013

An Interview with Bro(s)

In an effort to get additional perspectives on Bro, I recently contacted the contributors to Chris Sanders's and Jason Smith's upcoming book Applied Network Security Monitoring (NSM). Chris, Jason, Liam Randall, and David Bianco (who wrote the bulk of the book's material on Bro) were all kind enough to providing further insight into the Bro platform by answering a set of questions I proposed to them. 


I was first introduced to Bro as "the IDS you probably haven't heard of". How did you first learn about Bro?

Chris: I've been a long time Snort user and really believe in the power of signature-based IDS when used properly, but I also know that the future of NSM (or the now, some might say) is anomaly-based detection. I knew of Bro for quite a while as one of the better ways to perform anomaly-based detection for quite some time, but it wasn't until the past couple of years that I felt Bro was really beginning to gain the maturity that made it feasible for use in modern networks at scale. Once I really dug into Bro and was shown a few things by David Bianco and Liam Randall I really started to see the power of Bro and was blown away.

David: I used to work for Jefferson Lab, a US Department of Energy facility. Bro, of course, came out of UC Berkeley & Lawrence Berkeley National Lab, the latter of which is a DOE facility. Bro has been popular at DOE for quite a long time. My first real encounter with Bro was as a guest member of a DOE Red Team assigned to do a assessment at a sister site. It was pretty effective at detecting typical Red Team internal recon behavior. I didn't start actually using Bro myself until 2007, though. I've used in on and off sense then, and it's been a pretty valuable tool in my kit.

Jason: I learned about Bro several years back. Some guys much smarter than myself were in talks about creating a correlation engine of sorts. In reference to network logging, they had in mind what they wanted to look at, and in the end one of the gentlemen in that group mentioned that what they want was already being created. I'm not sure what happened with their correlation engine, but Bro has come a long way since then.

Liam:  I came to Bro out of desperation. Working in IR and auditing I was constantly finding fault in other open source and commercial tools. They had weak or no IPv6 support, they didn't work well with complex network provisioning, and they were just not designed to deal with the advanced threats I kept encountering in the field.


What advice would you give to those just starting out with Bro (besides reading the Bro chapter in Applied NSM)?

Chris: I think the best advice that I can give is the same advice I'd give anybody wanting to learn something new, and that is to just play around with it. It is so powerful and flexible and you can literally do almost anything with it. One good task might be to take a Snort rule that you understand, and try to achieve the same type of detection with Bro. This will help you learn the syntax and how to interact with Bro data types and generate the appropriate logs.

David: Learn all the flags to the "grep" command, because you'll be using it a lot to go through code samples! Seriously, the best advice is to take it slow and don't get discouraged. Bro is not a product, per se, so you don't just install, configure and turn it on. Come up with a few simple network oriented tasks you want to solve (not necessarily something that will detect Evil!) and use those to learn the language and the platform. After that, spend some time thinking about where you really need Bro. It can do almost anything, though it's up to you to use it wisely!

Jason: Start with the basic logging that Bro provides. Once you've managed to set up logging in your environment, set an analysis goal and approach it methodically (with Bro scripts or otherwise). With tools like Bro, it seems that the more you learn, the more you realize you can do. Suddenly, the one obtainable goal that you had set is now 10 obtainable goals, thanks to the knowledge you've obtained through approaching the original goal.

Liam: The best way to get into Bro is to use it. Start by replaying traffic samples through Bro and looking at the output. Looking at anomalous traffic will surprise you. Many common attacks and abuse just stand out like a sore thumb in Bro logs. Run Bro on your home network. It's informative seeing the output of traffic you are already familiar with on a smaller scale. You'll be surprised at what the devices on your home network are actually doing. Believe me, I put my TV on its own VLAN years ago!


What is the one thing Bro does that makes each of you say "Wow"?

Chris: The thing that makes me say WOW the most is that there isn't much that Bro can't do. I think the most useful feature for me is that every entry in every log file has a unique identifier that can be used to identify other aspects of communication between hosts in other log files. This takes a lot of the leg work out of correlation. I'm also generally amazed at how you can take every output from Bro and feed it back in as an input to refine detection or analysis.

David: I think that's the wrong question to ask. It's a lot like saying "What does Python do that makes you say 'Wow'?" Bro can do pretty much anything that involves network traffic (and even things that don't, so that's a "wow!" for you). The thing about Bro that really amazes me is the team behind it. They produce this fantastic platform, with tons of built-in features and frameworks to enable the community to do things even the Bro team hasn't yet thought of, and they give it all away for free! Wow!

Jason: Bro is one of very few tools that can make me feel like a beginner every time I use it, and I mean that in the best way. Calling it a "tool" isn't really fair, as it is more of a "capability". It is a wide open means of getting what you want from your data, which results in learning new ways to use Bro every time. Again, when I approach one goal with Bro, I generally end up with 10 more ideas.

Liam:  Bro continues to wow me on a regular basis. As Bro's tool chains and frameworks have evolved we keep discovering more and more use cases. Here's one that still can't believe: I had worked with a global retailer (a name brand everyone knows) to deploy Bro on their network. I get a call from their IR staff that they thought Bro had misfired with a "'credit card detected" alert. I thought that was unusual as Bro runs the Luhn checks on candidate hits and so I requested a pcap. What I found will continue to be passed along as Bro lore. A credit card processor had faxed over an 18 page credit card dispute form that included notes from a customer where the customer had printed their actual credit card number on paper. When it hit the fax server it was OCR'd and forwarded to Exchange for delivery. When Bro saw it on the wire, Bro popped the vlan, saw smtp, extracted the message, saw a .tif attachment, pulled the credit card number out of the OCR text in the TIF, verified it with Luhn, and then fired an alert. I shit you not. Part of the magic for this one was decent OCR, but still, I have broments like that all the time.


Do you use Bro at work, or if you're more comfortable, home, and how?

Chris: I use Bro at home on my network and lab, but I also deploy it to my clients in some instances. I try to use Bro in a lot of environments for my clients for specific tasks. For instance, deploying Bro to monitor a sensitive network segment with static IP addresses where all assets should be known. Using the same script that is discussed in the book in Chapter 10, Bro becomes a "dark net" detector, which will alert when communication is seen with any IP address that isn't on an approved list, existing in a dark part of the network.

David: I've used Bro at work for a number of things, not all of them detection oriented. I do a lot of packet analysis, and Bro is becoming my tool for quickly ripping apart PCAPs and extracting all kinds of useful info. I've also used it for a while inside malware sandboxes to try to detect/decode known families of malware, or even to just apply the Intel framework to see if the samples coming in are hitting any of the intel we already know about.

Jason: I use Bro as much as I am able to. My first and foremost use of Bro is just the basic out of the box logging. It logs everything I want out of the box, and on top of that it allows me to perform custom logging. On top of that, I use Bro for a lot of my detection experimentation. With Bro, I generally seem to start with narrowing down a custom logging format, and the next thing you know, it is more of a finely tuned detection mechanism. That feels good.

Liam: I use bro everywhere. I run Bro at home, it's my go to tool for IR, and professionally I deploy Bro for a living. As a team we are constantly discovering new use cases for the Bro Platform. I actually spend a lot of my time helping customers customize Bro for their specific use cases. It's exciting.


What are some commercial tools Bro could replace or compliment?

Chris: I am generally of the belief that most commercial NSM and detection tools on the market aren't entirely useful. They are basically "network antivirus" and try to automate detection and analysis. While you can automate detection to some extent, you can't automate analysis. The human has to be in the loop. I also think that most commercial tools abstract you from the raw data too much. These are reasons we only cover free/open source tools in Applied NSM. As for drawing comparisons between Bro and something else... I don't think there is anything commercially available that is like Bro.

David: Since it's really just a powerful programming language for network traffic, Bro really does have the potential to replace almost any network-oriented security tool out there. That said, I may be a little unusual in the Bro community, because I really also like traditional NIDS platforms like Snort. I think there's a place for both Bro and Snort (or similar IDS), and while one *could* implement traditional IDS signatures in Bro, it usually doesn't make as much sense because Snort is so good at that already. There's plenty of room in that space for both to co-exist, and I think they really complement each other when used appropriately.

Jason: Well, when you have Bro fans regularly give demos to vendors about what they should be doing instead of the other way around, you know that there seems to be a shift in things. A ton of organizations implement a ridiculous array of logging devices and detection mechanisms to perform tasks that are trivial for Bro. Without being too specific, it is more responsible to have centralized network intelligence, and Bro has that covered. Previously, many organizations had a huge reason to turn down Bro, and that was that it didn't meet the compliance check box for "professional support", but with groups like Broala, that is no longer a barrier for entry. I see a future with Bro replacing many current logging platforms, and down the road, complimenting other detection platforms.

Liam:  Bro both replaces and compliments entire suites of tools. Fundamentally it is a different approach to streams. A good example, people frequently ask, "Bro or Snort?". That's the wrong question. The tools approach problems with entirely different functions. Ultimately I believe in defense-in-depth, and that means layers.


What are some key takeaways you hope analysts will get from the Bro chapters?

Chris: The power and flexibility of Bro are unlimited. Applied NSM doesn't aim to be a thorough text on everything you can do with Bro, but rather, it contains some basic to advanced Bro concepts that are taught via example. We teach the reader how to build a dark net detector, how to extract specific files based upon MIME type, how to customize logging output, and more. With each practical example the reader should learn more concepts that can be applied to other scenarios. After all, this is APPLIED NSM, and we want the reader to be able to apply the knowledge. Between the concepts we present, the official Bro documentation, and scripts available from the Bro community, I think that leaves a lot that the reader can really work it.

David: The key points probably are the following: Bro is a platform, not an IDS, and getting started with Bro isn't so hard if you don't try to learn everything at once. I didn't have space to cover even 10% of what you really should know about Bro, though even what I have in the examples is pretty useful for solving real problems. Really, Bro needs it's own book, both for programming and for using it "cookbook" style.

Jason: It is an excellent tutorial for those getting started and it will no doubt teach the regular users a thing or two. I wish this material had existed when I first started with Bro!

Liam:  The biggest take away for people just getting started with Bro is this: Bro IDS is just the first great application written in the Bro Programming Language. There are hundreds of use cases for the Bro Platform.


Is there anything else you want people to know about Applied Network Security Monitoring or Bro?

Chris: I'm an analyst first and foremost. That goes for my co-author Jason Smith and my Contributing authors David Bianco and Liam Randall as well. This is really a book for analysts, by analysts. I've always wanted a book that I could give an entry to mid level analyst that would allow them to get up to speed on necessary topics for the job, or to help them further refine their methods and procedures. No book like that has ever existed, so that's why I decided to write Applied NSM. While it doesn't cover every topic thoroughly, I believe that it provides the necessary jump start for the individual who wants to be an NSM analyst. A lot of the concepts I've written into this book are new, and a lot of them are known techniques that have just never been codified and written down. I hope that this will be THE book for new NSM analyst, and that even the NSM veterans can gain something from it.

David: It's coming out on December 15th, and would look *great* under some security pro's tree! Seriously, I just want to thank Chris Sanders and Jason Smith for letting me help in my small way.

Jason: Applied Network Security Monitoring attempts to approach NSM in a way that will ensure that when you or your organization set an implementation goal, ANSM provides solid direction so you don't feel stranded. However, ANSM doesn't stop there. ANSM provides solid methods for collection, detection, and analysis, as well as an immense amount of reference material that should be at every analyst's desk. Regarding Bro, everyone is talking about Bro, but it is time to start implementing Bro. ANSM will get you started.

Liam: Working with Chris, Jason and David on the book was a real pleasure. It's great to work side by side with such talented people!


So there you have it - not only is Bro flexible enough to be used in a myriad of situations, it's also a continually-evolving platform that can only get better with time. To learn more about the book discussed, be sure to check out Applied Network Security Monitoring, available on December 15. Thank you to Chris, Jason, Liam, and David for taking time to participate in this interview.

Monday, October 7, 2013

SumStats Framework


I heard Seth talk about the SumStats framework (previously the measurements framework) at this past Bro Exchange. The presentation material was dense and at times difficult to follow. I had a previous interest in applying statistical measurements to Bro logs through R, but ran into memory issues (R is greedy), so I tried to understand as much about the framework as possible. The first time I read Seth and Aashish's udp_scan script, I had no clue what was going on. I dug into the SumStats framework's source and after banging my head on my keyboard long enough, it came to me.

SumStats isn't overly complex, it's just complex enough.

In this post I hope to shed some light on how the SumStats framework achieves scalable statistics you can use to determine what is "normal" on your network (or in your trace files). Below are some important terms SumStats relies on that you should be familar with upfront.
interval - a set amount of time
threshold - a maximum or minimum limit to something
calculation - calculations that can be applied to a set of observations (average, max, min, sample, std, etc)
key - a thing which is having observations collected about it (an IP address, an HTTP host header, port) (similar to an indicator in the intel framework)
observation - a piece of data associated with a key (fed to a reducer and ultimately a calculation)
reducer - a named set of operations to apply to keys and observations (including a set of calculations, a filtering function, and a key normalizer)
resultval - the value resulting from a calculation being applied to an observation (happens through a reducer)
result - a table of resultvals (multiple results can exist as multiple calculations can be applied to a single observation)
resultTable - a table of results (a table of tables)
SumStat - a named collection of reducers, intervals, thresholds and handling functions

For each key, a series of observations will occur over an interval of time. These observations are passed to a reducer that applies one or more calculations to each observation and produces a result. Result values are collected into tables. Upon a result exceeding a user defined threshold some single action can be taken, some action can be taken for each key, or both. Upon reaching the end of a user defined interval of time some single action can be taken, some action can be taken for each key, or both.

Because computer communications can happen at big and fast bursts, keeping a large amount of state is required for using the framework. To help reduce the memory footprint the SumStats framework works only with streaming algorithms (something I'd never been exposed to until working with Bro). This ensures if you've told Bro to measure something, for example the average value of all SYN sequence numbers, and all of a sudden that something happens big and fast (a SYN flood), Bro won't tip over.

Here is an example script that makes use of the SumStats framework. It measures the total amount of bytes sent by each host each day and the average number of bytes sent by each host every week. Keeping this historic data could help a network operator determine 'normal' transfer amounts per hosts. This script could easily be adapted to "summary statistic" the total bytes sent to a remote host per SSL certificate.

The script defines two intervals for my script. tx_summer_interval breaks every 1 "days" (I use minutes so you can see the script run in ~10 minutes instead of 1 week) and tx_aver_interval breaks every 1 "week". day_of_week maps day names to numbers. A day counter and a week counter are defined to be used as SumStat intervals (the actual value is a called $epoch in the SumStat record). Two reducers are defined. One is for summing, the other is for averaging. Two SumStat objects are declared; one is for averaging one is for summing.

Summing observations are sent to the summing SumStat object when a connection's state is removed. Averaging observations are sent to the averaging SumStat object when a summing result is calculated.

The SumStats framework took me a bit to wrap my head around but it's pretty neat.

Tuesday, September 24, 2013

Third Party Scripts and Example Code

Bro 2.2 beta was released today, providing new features to users. See the official announcement here. One new thing that came with the 2.2 beta release is an official guide to writing Bro scripts. Awesome! I'm especially excited about the SQLite writer, hooks, default function parameter values and everything SumStats.

Bro doesn't (yet) have a repository of modules. So, in an effort to help others learn how to program, below is a list of links to example Bro scripts others have written (that are not part of the Bro project) I have managed to locate. I cannot vouch for the quality of any of this code except my own. If your code or project is not listed below and you would like it to be, let me know and I'll update this page.

Bro Scripts
https://github.com/LiamRandall/bro-scripts
https://github.com/LiamRandall/BroMalware-Exercise
https://github.com/LiamRandall/novaha-bro-training
https://github.com/LiamRandall/bro-training

https://github.com/sethhall/bro-scripts
https://github.com/sethhall/bro-junk-drawer
https://github.com/sethhall/bro-securityonion
https://github.com/sethhall/ssn-exposure
https://github.com/sethhall/bro-domain-generation
https://github.com/sethhall/bro-apt1
https://github.com/sethhall/credit-card-exposure
https://github.com/sethhall/relog
https://github.com/sethhall/bro-scripts-2
https://github.com/sethhall/bro-dblogger

https://github.com/JustinAzoff/bro_scripts
https://github.com/JustinAzoff/bro_presentation

https://github.com/sheharbano/BotFlex
https://github.com/sheharbano/Correlation-Framework
https://github.com/sheharbano/scan
https://github.com/sheharbano/ipv6-security
https://github.com/sheharbano/icmp-security
https://github.com/sheharbano/dns-security

https://github.com/jsiwek/bro_vetting
https://github.com/jsiwek/bro_bitcoin

https://github.com/doppiosecurity/bro_scripts

https://github.com/mozilla/brozilla

https://github.com/kevinwilcox/bro-sccrew

https://github.com/srunnels/broselytize
https://github.com/srunnels/bro-scripts

https://github.com/grigorescu/bro-scripts

https://github.com/mavam/brospects
https://github.com/mavam/malbroro

https://github.com/anthonykasza/scratch_pad
https://github.com/anthonykasza/nxes
https://github.com/anthonykasza/users
https://github.com/anthonykasza/beginner_brogramming

https://github.com/punkrokk/bro-lessons

https://github.com/thorkill/bro-scripts

https://github.com/jonschipp/bro-scripts

https://github.com/visiblerisk/Bro-scripts

https://github.com/lruppert/bro-tor

https://github.com/stephane-chazelas/bro-skype-fake-https-detect

https://github.com/sooshie/bro-scripts

https://github.com/jshlbrd/bro_scripts

https://github.com/justbeck/bro-xorpe 

https://github.com/reservoirlabs/bro-scripts



Bro Syntax Helpers
https://github.com/sethhall/bro.tmbundle
https://github.com/mephux/bro.vim
https://github.com/srunnels/bro-mode
http://www.appliednsm.com/code/bro.nanorc




Projects Built On Bro and Auxiliary Projects
https://code.google.com/p/auditing-sshd/
https://github.com/jsiwek/sshd_audit_mux
https://github.com/JustinAzoff/bro-http-rest
https://github.com/grigorescu/Brownian
https://github.com/mavam/vast
https://github.com/bro/cheat-sheet
https://github.com/amannb/bro-utils
https://gist.github.com/anthonykasza/6326559
https://github.com/panaman/puppet-bro

Friday, August 9, 2013

Extending Bro's Core

If you've made it this far. Congrats, you now know more about Bro than almost everyone, ever. Bro is a great framework and has been going through tremendous growth in the past few years. New modules, new functionality, new features are being built into Bro every time I hear about it. If you use Bro enough you may eventually find that Bro doesn't have everything you need. Luckily, Bro is open source so you can add functionality to it yourself.

The Levenshtein distance algorithm is often used by spell checkers to determine if a word is a misspelling of another word in a dictionary. The algorithm takes two strings as input and calculates how different the strings are by counting the number of insertions, deletions and additions it takes to transform one string to the other string. Let's see how one would go about implementing this as a built in function for use in scriptland in Bro's C++ core.

Hopefully you still have the directory you created from cloning the Bro git repo from Building and Running Bro. If not reclone it with the following command.
    git clone --recursive git://git.bro-ids.org/bro

Open the file that defines built in functions that handle strings and have a look around.
    vi bro/src/strings.bif
This file contains specially crafted C++ that gets parsed and compiled into Bro when you run those ./configure, make, make install commands. Every string related bif available in scriptland is defined in this file. I essentially stumbled my way through this file as I don't truly know C++ or the extensions the Bro project has built into the core's C++ code.

The function here can be added to the src/strings.bif file and compiled into Bro. The way I came up with this new functionality was to
  1. Find a basic function and see how it works, I saw is_ascii as a good function to copy from
  2. Rip out what the function does and add your own logic. The core C++ requires you to use Bro defined functions such as Len() and Bytes()
  3. Be sure to wrap whatever you are returning to scriptland in a Val() or StringVal()
  4. Compile your added function into Bro while praying you did everything correctly
 Once Bro finishes recompiling, you should be able to call levenshtein_distance(s1: string, s2: string) from scriptland and receive a correct distance metric.

Intro to Brogramming - Frameworks

I have to be honest. I had to think for bit what the difference between a Bro module and a Bro framework was. I could be completely off base, but here's what I've concluded.

Frameworks in Bro provide global extensions to how Bro operates and what Bro can do. Frameworks can be thought of as shared libraries between parts of Bro. Frameworks are built on top of one or multiple modules. I like to think about modules in the context of what they expose to other scripts (they're exports) while I consider frameworks as something that change how Bro can be used from a high level/global perspective.

For example, the LogAscii, LogSQLite, and LogElasticSearch modules expose ways to log information Bro generates in different output formats. Together they help make up the logging framework which offers ways to log information in different ways to all of Bro. Bro's homepage has some frameworks listed. I have yet to write my own full featured framework and thus I cannot give an example of one. I again suggest turning to and reading the code. See the /usr/local/bro/share/bro/base/frameworks directory.

Bro frameworks include
  • Input - this framework provides a mechanism for Bro to read data (including Bro logs) into Bro for processing.
  • Intel - this framework provides a mechanism for Bro to monitor the entire network for a specific piece of intelligence. For example, a malicious domain could appear in a link within an email or chat, in an HTTP URL, or within a DNS request.
  • File Analysis - this framework allows for files seen on the network to be treated similarly to connections. Files can be carved from the wire, examined, hashed, or submitted to Team Cymru's Malware Hash Registry.
  • Logging - the logging framework supports other frameworks and modules by providing a mechanism to easily create a new log stream, filter an existing log stream, or define custom events that should occur when some piece of information is ready to be logged.
  • DPD (dynamic protocol detection) - DPD is very cool and probably the single reason I started to look at Bro. Bro uses a combination of standard ports, behavioral analysis and signatures to determine the protocol being used on the wire. If you've ever had to explicitly tell Wireshark which decoder to use, you have experienced the frustrations IRC running over port 8080 can create.
  • Notice - this framework provides a way for other Bro frameworks and modules to raise notices. Raised notices in scriptland are similar to raised events from the core.
  • Cluster - Networks get big and sometimes a single machine cannot handle all the traffic you may have to inspect. The cluster framework provides a mechanism for Bro to scale horizontally. Another nice thing about Bro is that it runs on commodity hardware. If a stand alone (a single Bro process on a single system. We've been running Bro in stand alone mode for these blog posts) can't handle all your packets, just through more cheap boxes at it.
  • Signatures - the signature framework provides a Snort style, packet by packet inspection mechanism. Signatures are mostly used by the DPD framework. As Bro focuses more on connections and stream than packets, signatures don't get used very often by brogrammers.

Intro to Brogramming - Modules

Bro modules are essentially C++ namespaces. Modules allow grouping of related functions, variables, and events under a single context.

The GLOBAL module is the default namespace for all Bro scripts. If no module is defined in a Bro script, it is dropped into GLOBAL. Modules are defined whenever a brogrammer is extending Bro at the scripting layer to do something beyond what Bro currently does. Modules are basic building blocks for extending Bro via scriptland. Examples of modules include the LogAscii, LogDataSeries, and LogSQLite. Each offers a different interface for logging information Bro generates.

Often times Bro modules are organized in the source tree in a somewhat logical manner and are given their own directory. A very basic module can be found here. To learn Bro, you have to read and understand its code. I highly recommend spending time exploring the /usr/local/bro/share/bro directory and files within it especially the base directory (which contains important scriptland modules and scripts) and the policy directory (which contains tweaks to the base scripts).

A module usually consists of the following
  • main.bro - The main script that defines the module and what it does.
  • __load__.bro - A special file Bro uses to to load all required dependency scripts for a module. Bro looks for this file in the working directory by default and loads it if the file is present.
  • helper scripts or data files - Other Bro scripts or data files that might be too long or specific to put into the main.bro file. Bro modules often work by chaining short script files (that do something specific) together.
From within a script file, the 'module' keyword registers a new module with Bro's core and the 'export' block defines global changes Bro should make to accommodate the new module being defined. Usually, the code in the rest of a module script is used to set values in the module's record (often for logging) or to accomplish other tasks the module intends to provide.

To find the names of all modules in your version of Bro, try running the following command in the directory %BRO_PREFIX%/share/bro
grep -R '^module' ./* | cut -f2 -d':' | sort | uniq

Intro to Brogramming - Events

Bro is event driven. When something happens on the network, Bro's core will raise an event and execute all the code blocks from scriptland associated with that event. Just as Bro has built in functions available for brogrammers to use, Bro has built in events. These built in events cover most of the things a network operator would want to know about (again, the magic of the BSD license allows you to extend Bro as you see fit).

I'm not going to write any code examples for built in events as Liam Randall already has done an excellent job of doing that. His Github page has Bro scripts that keep tallies of the number of events that fire while Bro runs. Read through the scripts and try to understand them. They are repetitive as they do the same thing for each event.

In reality, Laim's fire scripts are great for understanding how Bro works, but are too simple to give any great insight into what is happening on the network. Pick an event he included from a file, an easy one you can easily generate (like dns_request) and replace his code block with your own. Here's an example something I want Bro to do when it sees a DNS request.

event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count)
{
    print fmt("DUDE! I just saw some bro query the DNS for %s with a query type of %d and a query class of %d", query, qtype, qclass);
}
Try running this script and writing your own. Add some local variables and calls to built in functions to the event.

Intro to Brogramming - Built in Functions

Defining functions within a script is good and useful, but Bro has very useful functions already built in to C++ core for brogrammers to call from scriptland. These functions, available to Brogrammers from scriptland are called built-in functions.

Being an open source project Bro get contributions left and right from people who extend its functionality. The file located at
/usr/local/bro/share/bro/base/bro.bif.bro has a list of all the functions available for general programming in Bro. /usr/local/bro/share/bro/base/strings.bro.bif has a list of all the string related functions Bro has built in. In fact, you should familiarize yourself with the every .bif (built in function) file in that directory. Alternatively, you can browse the auto-generated documentation those files create by following the hyperlinks.

Bro has functions for doing math, string manipulation, address manipulation, file handling, type conversions, events, all sorts of stuff. Functions and events can be decorated with attributes too. The &priority attribute can be used to set an event's priority in the core's event handling queue. When an event occurs, the core of Bro collects all blocks of code associated with that event and executes them (you could have eight scripts loaded into the running Bro process and all eight scripts could have code blocks associated with a single event). Setting a priority on an event from one of those eight scripts ensures the code block is executed by the core with a higher priority.

Run this script and find the bif's it used in the built in function files (or their auto generated documentation equivalents). Try to find at least five new functions you think sound interesting or useful. Pay attention to the parameters' types the functions require and the type returned by the functions.

Intro to Brogramming - Logic, Flow Control and Functions

What would a programming language be without conditional statements and loops? Pretty, lame. Lucky for you, Bro isn't lame.

Code within Bro script files can be within functions, events (both of which we'll get to later), or neither. This is similar to code in a Perl/Ruby/Python script can be nicely placed within a subroutine/function or just written haphazardly in the file.
Code placed in functions and events is not run until the function gets called or the event is triggered. Code that is placed in a script but in neither a function nor an event is executed sequentially. To allow the brogrammer to control the flow of a script, scriptland has the following commands
  • if, else if, else  - Code following these statements execute given the defined condition is met.
  • ternary - Ternary statements are shorthand if statements and behave the same way.
  • when - Code blocks following these statements execute asynchronously. The code blocks are queued and are executed when the condition is met, indipendent of sequential execution happening in the rest of the script.
  • schedule - Schedule a block of code to execute after a user defined duration of time.
  • for - In Bro, for loops are used to iterate over something - a vector, table or set. For loops do NOT guarantee order of the structure they iterate over (random access). For loops require a local variable to be  assignment the current value of the complex data structure being iterated over.
  • break - A command to end execution of the nearest loop.
  • next - A command that begins the next iteration of the nearest loop, skipping any commands after the next.
  • return - Usually the last command in a function, it returns a value from a block of code to its caller

Run this script and be sure you understand everything that is happening internally.

This is what the script does
#scheduling and when
  • creates a user defined event 'e' that prints the count value, 'c' passed to it and sets the global count variable equal to 'c'
  • schedules the 'e' event to run 10 seconds after Bro initializes
  • schedules the 'e' event to run 15 seconds after Bro initializes
  • Asynchronously queues a print statement to execute when the global variable 'x' is equal to ten (this should occur 15 seconds after Bro initializes)
#conditionals
  • Define two boolean values, one to true and one to false
  • Test if 'b2' is true (it's not) and if so print something
  • Test if 'b2' is true (it's not) and if so print something
  • If the previous two tests both failed (which they did) print something
  • Declare an undefined string variable
  • Use the ternary operator to 
    • If 'b1' is true set 's' equal to "b1 is true"
    • Otherwise set 's' equal to "b1 was false"
  • print the value of 's' to STDOUT
# loops
  • Define a set of strings name 'ss'
  • Loop over each string within 'ss', setting 's' to one of the values
  • Print 's' to STDOUT
Chances are you see print output from the conditionals section of code before you see the output from the scheduled section of code. This is because Bro schedules the code to run in the schedules section and continues executing the script sequentially. That is the meaning of asynchronous.

Intro to Brogramming - Complex Data Types

The previous section covered variable conventions and basic atomic data structures in Bro. This section continues to build on the previous section
by introducing complex data structures. These complex types are composed of atomic types and behave differently. Below is a list of the complex data structures Bro has available for use.
  • enumerable - Enumerables are collections of dissimilar things and are used to bypass strong typing. Enumerable are strange, I think of them as sets of things with no type.
  • records - Records are collections of related things. records can be thought of as a single row within a table or a C-like structure. Each variable in a record has a type. Records are very important in Bro. Logging is done with records. Other data structures, such as the connection type, are built using records.
  • sets - Sets are two dimensional arrays of a single atomic type. Think of a set as an array of things that share a type. A set of strings could list user-agents.
  • tables - Tables are just associative arrays. Tables have keys which map to values.  Tables in Bro are similar to hashes in Perl/Ruby and dictionaries in Python. In Perl a common data structure is a hash of hashes, in Bro it is very possible to have a table of tables.
  • vectors - Vectors are tables which are always indexed by a count. Vectors may be on their way out of the Bro code (I heard something about that). Try not to get too attached to vectors.
  • functions - Functions are named blocks of code, surrounded by curly braces, that can be reused. Functions sometimes return a value and always require a type. If a function doesn't return a value it must a a type of void.
  • events - Events are raised by the C++ event engine within Bro. Code within event blocks is executed when Bro's core raises that event. Events often get passed parameters from the event engine to use within the user defined code block. For example, when the event engine raises a DNS request event it passes scriptland information about that request, like the query string, the IP address making the request, the IP address the request is being sent to, and a bunch of other information.
  • file - A file handle. Bro can write to files, and often does within the logging framwork, but the Input framework should be used to read from files. (We'll get to frameworks later on).
Other variable types exist within the Bro source, however they are built on combinations of simple and complex data types presented in this and the previous section and are mostly syntacitc sugar. The connection type I mentioned earlier is one such type. It is easy to image how these types can be combined to create many different data structures. For example, a database table could be represented by a table of records.

Because Bro deals with network traffic, which has a high volatility and often a high volume, data goes stale quickly and Bro needs a way to deal with this.
Attribute decorations are used to give a variable a special property, such as when the variable expires (variable timers), or if the variable is required to be defined (e.g. within a record). The Bro website details all the built in data types in great length, so I won't. It can be found here.

Run and read this script and be sure you understand what is going on.

Within the script the following occurs
  • A new variable type is built using a record type. 
  • Values are assigned to two instantiations of the NewType and printed to STDOUT. 
  • Sets of ports and strings are created, values are added and removed from them. 
  • The size of the set is calculated and printed to STOUT. 
  • A port table is created, accessed, and printed to STDOUT using a for loop (we'll get to control flow later).
    Run the script a few times and pay attention to the order the table values are printed to STDOUT each time. For loops in Bro are random access. Each value will get accessed once within the loop, but their order is NOT guaranteed. This is rather different from languages I was used to and its bitten me more than once. Be aware of the random access for loop.

Intro to Brogramming - Variables and Simple Data Types

The wordplay titles stop here. Sorry to disappoint.

Brolang is strictly typed which means a variable must be assigned a type and can only act in a specific way.  If you've written C code, you're used to this. If you come from the lazy land of Perl, you should expect your variables to rely more on how you define them.
Bro has been built to accommodate network programming which makes it very good at doing just that. Some of these accommodations can be seen when looking at the native data types Bro has to offer a brogrammer. For example, having variables of type addr is rather convenient when working with network data. MySQL users know all too well the pains of not having native IP address data types.
Bro includes the following atomic data types:
  • addr – an addr can contain a network address. IPv4 and IPv6 addresses can be held within an addr type.
  • subnet – a CIDR notation subnet (10.0.0.0/8). Even though a subnet is a collection of addresses, Bro still treats subnets as atomic data types.
  • bool – a bool can contain only two values, True or False. This data type behaves similarly to boolean variables in other languages. An example usage of this variable could be to determine if a condition is met or not.
  • count – a variable of type count contains an unsigned 64 integer (positive number). Bro has potential for looking at huge amounts of data off the wire. Sometimes you want to count things (you can’t have a negative number of SSH login attempts).
  • double – a variable of type double hold values of double-precision decimal numbers. Bro has the ability to do math for you and a situation could arise where simple integer math does not fulfill needs.
  • int – similar to counts but signed (can be positive or negative). An example usage of an integer is determining the change in number of connections seen between this hour and a previous hour.
  • interval – a range of time (3 sec/min/hr/day[s]). An interval can be used to measure time relative from something else. You may want to instruct Bro to set a timeout for something or to wait a specific amount of time after starting up to execute a function.
  • pattern – a regular express. Regular expression are fantastically powerful. Bro's dynamic protocol detection uses regular expressions to identify protocols on the wire instead of relying solely on standard port numbers.
  • port – a port number and associated protocol (they take the format of 53/udp or 80/tcp). What use is a network programming language without addresses and ports?
  • string – a string of bytes. Strings occur all over the place in Bro. Strings could be domain names, URLs, or even email messages.
  • time – an absolute epoch style time (this data type is global, compare to set which is local). This data type references a specific time (clock on the wall time). Sometimes a brogrammer might want to record the exact time a connection took place.
  • void - the absence of a type. This type is usually seen associate to function (we'll get to those later).
  • any - this type is used to bypass Bro's strong typing. This type is also associated mostly with functions.
   
Just like many other programming language, brolang has the concept of scope. A variable can be declared to have one of three different scopes.
  • const - the variable cannot be changed (this can be trumped with decorations)
  • global - the variable is available to all loaded scripts
  • local - the variable is local to the module, function or event
Declaring and defining a variable can be done by deciding a scope, name, type, and value for the variable. Let's say I want to print my name to the screen. I'll need a variable of type string and I'd like to call it 'myname'. The variable can be defined and declared the following way
    const global myname: string = "anthony";
   
Because myname probably won't change, I've set it to be a constant.
If I wanted to have a variable that held my age, I could define it in the following way
   
    global myage: count = 42;
   
Because myage will never be negative, I've assigned it a type of count and not int. Run this script and try to understand what happens with the myage variable and its scope. The bro_done event won't fire until you terminate the Bro process (ctrl+c).

Broverview

Bro is an asynchronous event driven platform. Take some time right now and be sure to understand what that means because it is key to how Bro operates and should be well understood to be a brogrammer.

Bro has an event engine that raises events when it sees something happen on the network (those things could be DNS requests, UDP packets, a specific email subject, or a file MD5). It's up to the brogrammer to define events that should be raised and what should happen (aka Bro policies) when those events are raised. Bro programmers sit at the policy script interpreter layer, waiting for events to be raised by the core event engine.

Bro reads raw data off the wire or from a pcap file (by default using libpcap), similar to tcpdump or Snort. The packets Bro reads are reassembled into streams. Bro has many events that are raised from the network streams it reassembles. A simple example is the new_connection event. The Bro core (where the event engine is located) raises this event and it 'occurs' within scriptland whenever Bro sees a new connection happen. Bro also raises events about itself. For example, Bro raises an event right as it begins running and right before it terminates. Bro has many more events which will be covered in a later post.

In addition to Bro's event engine, Bro has a scripting layer that allows brogrammers to define what happens when an event is triggered. For now, imagine Bro's event engine only has the three example events I mentioned above (new_connection when a new network connection occurs, bro_init when Bro first begins running, and bro_done just before Bro terminates).
With only these three events a brogrammer could instruct Bro to
  • print a nice welcoming message when Bro starts, perhaps "Sup bro, sup?"
  • print a different message, perhaps "Yo bro, a new connection happened", whenever a new connection happens on the wire
  • print a closing message when Bro is told to terminate, perhaps "I'm going down, bro"
This script does just that. Download and run the script with the following command. This tells Bro to begin monitoring traffic on the eth0 interface and to run with the policy script provided.

/usr/local/bro/bin/bro -i eth0 broverview.bro

Terminate the Bro process (with ctrl+c) and inspect the output that is printed to STDOUT. You should see the messages defined in bro_init and bro_done, but Bro might not have printed the message from the new_connection event. If it didn't, that's because Bro didn't see any new connections while it ran. Try running the script and generating a new connection across the interface.

Read through the broverview.bro script. Syntactically, Brolang, or the Bro scripting language, looks similar to many other C based languages. Brolang inherits semi-colon delimited lines, curly brace code blocks, and other semantics fromthe C++ core it is built on top of.

You might have noticed that the event handlers in the broverview.bro script aren't executed in the order they appear in the script. This is because they don't execute unless Bro see the event on the wire and 'handles' them. This non-sequential execution of events is where the event-driven programming reference I made at the beginning of this post comes into play. Don't worry if you don't understand all the particulars of the script just yet.

Our example is very simple and somewhat useless, but it shows how a brogrammer can specify actions (in Bro vernacular, 'policies') for Bro to take
when an event occurs. Bro is capable of doing much more than just printing to STDOUT. In useful Bro scripts, nothing is printed to STDOUT. Instead, things are logged to files or alerts, like emails, are generated.
Bro can send you an email, raise an alert, execute a system call, or just about anything else you can script. This is where Bro's power shines through - in extending and customizing event handling.

You might have noticed that Bro, by default, generates logs in the directory
you execute Bro in. This is because, by default, Bro is configure with policy scripts that tell it to log information. For example, when an HTTP request and response are seen and completed (events are raised for HTTP seen on the wire), Bro writes some information about the HTTP connection to disk. Try running Bro on an interface while browsing to this blog and see what log files get written to disk. Open the log files and try to determine what the fields in the file are.

The events Bro raises are neutral. The Bro event engine's job is to let scriptland know when something happened. It is up to the brogrammer to determine if the event is good, bad, actionable or uninteresting. Thus Bro isn't really an IDS, but a network investigation platform. You could certainly build an IDS from Bro's scriptland but you could also build Conway's game of life. The core of Bro simply passes summaries of the network up to scriptland for a brogrammer to script against. If I don't care about SMTP traffic, I can tell Bro (in scriptland) not to do anything when the Bro core reassembles SMTP traffic. If I do care about SSL traffic, I can write a script to extract all SSL certificates Bro sees. In fact, that's what the ICSI Certificate Notary does.

Hopefully now you understand how Bro inspects live/stored traffic, generates and passes events to scriptland based on the traffic, which are handled by user defined scripts or policies.

If you still have questions, feel free to ask in the comment section. A complete summary of the Bro project can be found here. I recommend having the Bro cheat sheet on hand through out the rest of these posts.

Building and Running Bro

In this article I will introduce how to get Bro up and running on Debian based system as quickly as possible with the most features as possible. The rest of the articles will assume you followed this guide. You should know that the Bro website has a very useful quickstart guide of its own, but I've decided to write down the procedure I often use. If you have questions or want to learn more about the different ways to install Bro, you should visit their guide.

The first thing to do is to set a system up. You can use a VM or build a physical machine, it doesn't matter. We'll be grabbing the source code and compiling it to make sure we get all the latest features and frameworks.
Bro depends on some packages for compilation to succeed. You'll need to install these before you can build Bro. Run the following command to do so:

sudo apt-get install git cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev

Now, download the latest Bro source code via git by running

git clone --recursive git://git.bro-ids.org/bro

Make sure you include the recursive option or you won't grab everything you need to compile Bro. When the clone is finished you should see a 'bro/' directory sitting in your working directory. Change into that directory. The default directory Bro is installed to is /usr/local/bro (that's what I'll be using for this and other blog posts) but that can be changed with the '--prefix=/desired/path' option. Configure the installation by running:

./configure

Compile and install the Bro software by running:

make
make install

NOTE: you might need root privileges install Bro

You can add bro to your path as a convenience by altering your PATH environment variable and exporting it, but I usually don't bother. Once Bro is installed, run it by calling it with the i option and specifying an interface name to begin sniffing on.

/usr/local/bro/bin/bro -i eth0

If Bro starts successful you should get some output similar to "listening on eth0, capture length 8192 bytes". Terminate Bro the same way you would with any other long running terminal process (ctrl + c). Depending on the packets Bro saw on that interface you might have some new files in your working directory with a .log extension.

Congrats, you just installed and ran Bro!

Bropening Statements

Welcome! If you are looking to learn more about Bro as a programming language, you have come to the right place. Bro is a network monitoring platform that can provide insight into what is happening on a network or what has happened within a trace file. Bro has the capabilities to collect traffic to a trace file, identify specific patterns within traffic, and summarize flows. Bro can provide the same capabilities provided by tcpdump, Snort, and netflow combined. And more!

Bro is also programmable, meanin an operator is bale to script between the previous mentioned capabilities. It comes with a large set of default and tunable scripts. These included scripts are used to build Bro modules and frameworks which extend what Bro can do. As Bro is extremely extensible, mastering its domain specific language unlocks the potential of Bro.

This site is dedicated to programming with the Bro language. The Bro documentation can be intimidating for beginners and the constant developement that makes Bro so useful means keeping up can be a challenge. For these reasons, I intended to post a series of 'getting started; articles for understanding the fundamentals of programming with Bro.

I hope you find this blog useful. Feel free to leave comments if you have any questions or suggestions.