Security Log Management: Identifying Patterns in the Chaos

Can you determine the top .com/.net/.edu/etc. site that your users are requesting? How much of that is malware? Can you use that information to create a blocklist of bad domains to kill malware before it has a chance to communicate outside your network(s)?
Since your organization is using Bro, what other cool information can we gather with this tool? Since Bro is logging all this information about a DNS and Web traffic, can we generate statistics about this data? How about showing how many sites are being accessed by domain (.com/.net/.xxx)? The short answer to these questions is a definitive yes. Again, using the DNS requests or A record queries, we can determine and count the breakdown of DNS domains through scripts. In the following example, we are simply gathering a total number of DNS queries per 24-hour period:
#!/bin/bash total_dns=`grep "?A " /path/to/bro/dns/dns.log.*cat n tail n 1 awk '{ print $1 }' `
This count alone can be used as leverage in swaying an organization into using their own caching servers, as a caching DNS server can cut down on the amount of bandwidth being used by an organization s Internet pipe. Another reason to use a caching DNS server is that it is easier to poison the server with bad domains for known malware hosts or policy violating hosts (discriminating, sexual,...