Security Log Management: Identifying Patterns in the Chaos

If you follow the defense in depth methodology of network defense, arguably one of the first methods to discover if attacks worked is to use flow or session information about the network traffic. Flow- or session-based logging that can be done even at high speed, such as multigigabit Ethernet, is a type of IDS that can come from different types of tools such as Argus for text-based logging or in graphical format from such companies as Lancope, Arbor Network, Q1 labs, or Mazu Networks. This type of IDS is meant to capture not the packet payload or the single hostile packet but rather record each connection in an auditable form. For example, when you use a tool such as Argus or Cisco s Netflow, a logging record is kept for HTTP, FTP, and other connections. However, several key pieces of that connection are recorded when dealing with flow or session logging. This auditable portion of the log includes the following pieces:
Duration How long did this connection last?
Source and destination IPs and ports involved Who was talking to whom? And over what ports?
Protocol Was this a TCP? UDP? ICMP? Or other protocol?
Number of packets sent/received Was this a small connection or a large one?
Amount of packet payload size How much data was involved in this connection?
TCP flags (optional) What TCP flags were in use by both client and server in...