Security Log Management: Identifying Patterns in the Chaos

We have already shown you how you can use the Bro IDS tool to enforce policy, but how about helping to track things like the top sites your users visit, or which user is your top Web surfer? When you run Bro with the HTTP module enabled, tracking each connection is quite easy the connection details are harder.
To track the top site using this module, you need to search in the first line in the next example:
1126121446.333380 %1580 start 10.0.4.100 > 64.233.161.104 (cut for brevity)1126121446.333380 %1580 > USER-AGENT: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT5.0; .NET CLR 1.0.3705)1126121446.333380 %1580 > HOST: www.google.com1126121446.333380 %1580 > CONNECTION: Keep-Alive1126121446.351828 %1580 GET /ig/f/lPb00AJmja0/ig.js (200 "OK" [3732]) (cut for brevity)
Several versions of BRO have a limitation: if you let BRO run for long periods of time (more than three hours) without a restart, several of the policies will start to leak larger and larger sections of memory. Another hangup of BRO is when you restart it, it will delete and recreate the empty files for logging! To correct both of these problems a simple script is needed when to schedule BRO restarts and log moves. In all of my examples, BRO is restarted every 10 minutes. While this may seem excessively short to some, this actually will help make data searches easier and faster. For example at a recent client site with only a T3 connection the daily HTTP...