Security Log Management: Identifying Patterns in the Chaos

Using an open-source tool called Bro (www.bro-ids.org), the logging and capture of application-level protocols is now possible. For example, when using the Bro HTTP module and its submodule for http-header, you can capture several useful pieces of Web traffic for every connection. Such information as the server name, user-agent, HTTP commands issued, and other information is captured for each connection, as shown in the following example:
1126121446.333380 %1580 start x.x.x.22 > x.x.x.1041126121446.333380 %1580 > ACCEPT: */*1126121446.333380 %1580 > REFERER: www.google.com/ig1126121446.333380 %1580 > ACCEPT-LANGUAGE: en-us1126121446.333380 %1580 > ACCEPT-ENCODING: gzip, deflate1126121446.333380 %1580 > IF-MODIFIED-SINCE: Tue, 06 Sep 2005 06:37:45 GMT; length=107041126121446.333380 %1580 > USER-AGENT: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.0.3705)1126121446.333380 %1580 > HOST: www.google.com1126121446.333380 %1580 > CONNECTION: Keep-Alive1126121446.333380 %1580 > COOKIE: IGPC=ET=hjlqEWZI3AM; IGAT=PI=0; IGTP=H4sIAAAAAAAAAG2PQQ6DIBBFr2JYNwqiCydN1z1AF12QNBOg1UQBAWO8fdHWuumCZH4y782HA82q9Bo4MyjrjAPbcg2EARO0oiUI0sbooCjmec7HyUaMnTXB4Uvn0g6FwojFaKPKfQiCkMvXViVbudvMNDzQx072OsDq5cIELVfTFqkwO9gkju9cbyX-lgS5onPLaZ1vd0EOZqu-Qux_ddVtp9AvuddP7bWRn_Kz9co-Y6sVLikcn3gDNp28shoBAAA; PREF=ID=2911b04d733ad670:TB=2:TM=1109377476:LM=1112645061:S=t0bvd40u7M-OORUS; testcookie=; IGDND=11126121446.351660 %1580 < CONTENT-TYPE: text/javascript1126121446.351660 %1580 < LAST-MODIFIED: Tue, 06 Sep 2005 06:43:46 GMT
The rest is kept out for brevity.
Now you re probably wondering how to enable this tool and what other information you can obtain by using it. As mentioned before BRO is an application-level IDS that at its lowest levels is a series of protocol decoders. These decoders then have a policy built around them; for example, in the case of SMTP, the decoder knows the order and SMTP commands to expect in a typical mail session. If the connection uses different commands such as use of authentication between active...