The Bro Network Security Monitor
bro.org
bro.org
Funny, I was just telling my girlfriend this morning how the US doesn't have many free-standing "research institutes" a la Germany's Max Planck institute, but bro, having been produced by ICSI, came out of one of the best academic CS labs in the US, if not the world -- the International Computer Science Institute.
This is a very rich area of research and one I'd like to apply some of my recent high-scale stuff I've done at work on. bro was doing event-based processing in pure C++ long before node.js existed (some 10 years ago) -- these guys are really, really smart.
The large number of "think tanks" which are often funded through a diverse set of grants (RAND, MITRE)
ISI? (the one at USC which invented a lot of the Internet, not the one funding Al Qaeda)
Cheap access to grad students and assistant professors, as well as access to USG funding (and easy visas), probably pushes a lot of this into university affiliated or university departments.
It's just a mess. The tech looks interesting though, but it's really hard to take them seriously because of the name they chose.
Edit: Why is this getting downvoted? I pointed out my reaction to the title. I had never heard of it before so had no connection to their writings from 15 years ago. P.S. I didn't flag it.
> "Our monitoring system is called Bro (an Orwellian reminder that monitoring comes hand in hand with the potential for privacy violations)"
So Big Brother = Bro
When I was in the position to scan and choose between many hundreds of job applications and even on a mid-sized start-up, where only a few applications went in. No I am NOT a HR Person, absolutely not, but they asked me to do it, so I did that too.
EDIT: Not sure why this earned a downvote, but to your information there is no law forcing companies to let a % of women in for example (in the country I live).
For a project named "Bro", I believe that we have a very open and inclusive community. We do play with the word a lot but it's all in the name of fun and very convenient since the name lends itself toward so much word play. There have been times where people have started to use it in the more modern pejorative sense and we've actively yet quietly made it stop. None of us on the core team like that behavior and with the name "Bro" we have to be extra careful with how it's used within the community.
There has been a number of women that have passed through our community and several that are actively involved at the moment (I could say the same about men, our team isn't very large).
I really like that people on this thread have been digging quote out of Vern's original paper about the lineage for the name. That's easily the best response to all of the "Why Bro?" questions.
Fortunately, there was at least some good discussion of the substance here!
They use Bro, Snort, Suricata, Argus and other tools to record metadata about every IP connection that comes into or leaves their networks. Some of them terminate SSL connections and forge certificates. A few of them even drop encrypted protocols that they are not able to decrypt and inspect.
They use taps and/or SPAN ports to do the spying.
Most of them try to keep this activity quiet. This mentality is pervasive and it is everywhere (especially in USA based organizations). Everyone should be aware.
No one is safe from this spying, even senior management and tenured faculty connections are being inspected and recorded for later use if needed. They just don't know it.
They also emailed and temporarily disconnected all students who were running servers vulnerable to heartbleed, so presumably they do some form of more intensive inspection and logging as well.
And if they were very concerned with exposure of sensitive data, they wouldn't be logging it.
That's not the same thing as "monitoring for illegal/commercial/malware activity".
This is anecdotal of course but I can't imagine what were were doing back then was unique or isolated. It's trivial.
My boss barely understood what I was doing, his superiors certainly didn't know anything about it. We were not part of what would be considered the university IT department either -- we were just some random organization within the university. Who knows how many different people on hops above us were doing the same thing. And I was capturing the full traffic (not metadata) of people who would normally take even extra offense at this sort of thing going on. Not because the traffic contained SSNs or credit card details or something like that, but because the traffic was sensitive in a more private, personal way (I can't go into the particular details any more).
Unfortunately, this brief story doesn't have a juicy ending. I didn't do anything nefarious with the data. I didn't use it to spy on anyone. I simply used it to watch out for attacks against services on our network -- I thought I was doing something positive for the users of our services. But reading the parent post made me pause for a moment and consider that all these things I'm reading about and taking issue with in the news today (the NSA, eavesdropping, etc), that I did something similar, albeit on a much, much, smaller scale myself, many years ago when I was younger and more naive.
Full packet capture for how much disk space you want to allocate to it (many like 48 hours) then longer term storage of flow records, DNS and http metadata, etc.
The majority of the use cases are watching the internal uses of the network as well - not generally being used to detect intruders.
Your average IT Security Group is focused on their own Internal network. This includes all Internal and External Traffic/Communication going to/From the Internal Network. The reality is, most Security Threats come from an internal source [1]. So yes, your average IT Security Group is interested in monitoring, analyzing, and sometimes dropping internal Traffic. This allows the Organization to track and respond to Data Breaches and Security Incidents. The overall insinuation of this comment seems to be that this is Evil and a Violation of your Privacy (Spying!). But if you've ever worked with (or used Services provided by) any Organization that has a handle on Security, you've likely signed a User Agreement Form (or similar), which clearly states what is going on. So nothing is hidden, and when you think about it, this is a logical reaction to the realities of Security in today's Digital Age. If you can't trust people, then it makes sense to implement checks and balances. Instead of thinking about it from the perspective is a User, think about it from the perspective of a Service Provider, and it makes a lot more sense. If you think this is Unjust, then the solution is simple. Provide your own services and control your own Destiny.
A 'miniature NSA group' is (presumably) focused on External Networks and External Data Sources. And I say presumably because it is not really clear what you mean by 'miniature NSA group', but the insinuation is clear. So this is very different from your average IT Security Group, and it is not correct to insinuate that they are one and the same.
[1] http://www.itproportal.com/2013/10/15/security-experts-no-su...
Bro, on the other hand, is less of a product and more of a framework (think RoR) for studying protocol design and parsing. It's written in pure C++ so it's fast, and a lot of research papers have been written exploring things like programming paradigms for processing network data (functional vs. stream vs. imperative vs. OO), the best way to express exploits and vulnerabilities, efficient ways of tracking and storing protocol state, etc.
I think they're trying to commercialize it into something usable without tons of tuning. A good goal, as it's not that usable for out-of-box IDS/IPS ca. 5 yrs ago.
In practice Snort (Suricata, etc) can read, understand and react to individual streams on the wire very quickly. This is especially important for intrusion prevention (IPS) inline.
BroIDS (prelude, etc) generate detailed logs and highlight interesting traffic (as configured) and are excellent for gathering intelligence. One of the recently popular features of BroIDS is to decode and save to disk all files in traffic it sees, checking the hashes of those files against blacklists as it goes.
If you are at all interested in these systems you should try out Security Onion at www.securityonion.net, an awesome pre-configured Linux with many network security monitoring (NSM) tools already installed including Snort, BroIDs and many many others.
First, Bro is a Turing-complete scripting language ("the Python for the network") and Snort/Suricata a system centered around regular-expression matching [1]. These two paradigms have fundamentally different levels of expressiveness.
Second, Bro's core is policy-neutral. That means has no preconceived notion of good or bad, it simply provides information about activity. On top of that, it ships with numerous policy scripts to detect actual attacks. For example, there exists an SSH analyzer simply reporting the banner for each connection and byte-heuristic for detecting successful logins. On top, there exists another script that attempts to detect brute-forces by simply counting connection attempts per unit time. On the contrary, operators feed Snort/Suricata with rules which feed the system with malicious data. Your analysis is only as good as your rule set. As such, the systems spit out mostly attacks, which are useful iff calibrated to not emit a ton of false positivies.
[1] Snort features numerous enhancements for state tracking, but these are one-offs and hard-wired. For example, Snort supports "pre-processers" written C, which integrate at a much lower level of abstraction.