Since june 2002, Lire™ supports the merging of reports: one can combine two reports into one bigger report. This can be used to generate e.g. a weekly report from 7 daily reports, or generate a site-wide report from reports about the behaviour of each server on a site.
We describe some of the issues involved here. More in-depth information can be found in the lr_xml_merge manpage.
We give an example.
Example 3.5. Merging Reports
To process two BIND v9 logfiles, and merge the reports, one would run:
$ zcat /var/log/named.2.gz | lr_run lr_log2report \ -o xml bind9_query > $XMLDIR/20020622.xml $ zcat /var/log/named.1.gz | lr_run lr_log2report \ -o xml bind9_query > $XMLDIR/20020623.xml $ lr_run lr_xml2report -U dns -S bind9_query \ $XMLDIR/20020622.xml $XMLDIR/20020623.xml > \ $ASCIIDIR/20020622-20020623.txt
Of course, refer to the lr_log2report and lr_xml2report manpages for more detailed information.
Suppose you're running a daily cronjob which feeds your log to lr_log2report (possibly via lr_cron). If you'd like to generate weekly reports, you should first make sure LR_ARCHIVE is set to '1'. This way, each daily XML report will get saved in files like e.g. $XMLDIR/hostname/20020602085048-20020603221309.xml.gz where XMLDIR is /var/lib/lire/data/report/xml/dns/bind9 (the raw log files, as well as the intermediate DLF files, will get saved too). Now you can add a weekly cronjob, which unzips the last 7 files from this directory to 7 temporary files, and runs:
lr_run lr_xml2mail -U yoursuperservice -S yourservice \ uncompressed-xml-file-nr-1 uncompressed-xml-file-nr-2 ... \ uncompressed-xml-file-nr-7 joe@example.com
Be sure to clean up the 7 uncompressed files afterwards. Refer to the lr_xml2mail manpage for more details.
The merging functionality is very powerful, and allows you to shoot yourself in the foot. We document some pitfalls.
When merging XML report files xml.3 (2002-06-02 08:50:48 CEST - 2002-06-09 08:05:06 CEST) and xml.1 (2002-06-16 08:18:40 CEST - 2002-06-21 22:13:09 CEST) , the generated report will gladly display "Reporting on period: 2002-06-02 08:50:48 CEST - 2002-06-21 22:13:09 CEST": There is no saveguard against forgetting in-between report files.
In some cases, changing the report configuration file (superservice.cfg) just before merging might lead to bogus data in your report.
Consider this case: Our firewall.cfg file looks like:
=section Denied Packets Reports |select-action action_match="denied" top-pkt-by-src ips_to_show=10
We process some firewall logs, and archive the XML reports. Now we set ips_to_show to 1000, and merge the XML reports. This could incorrectly omit some IPs! You've got no guarantee the exact top 1000 IPs are shown. This is due to the fact the XML reports do not contain all information from the log: they're reports, after all.
Due to these issues, the merging is implemented with some heuristics: we keep more data than what's requested by the user in the XML report, to be able to handle most after-the-fact merging requests. We've tested the algorithm with a pretty broad range of real-life log files, and found out generally, the merged reports do give a good reflection of what actually has happened on the network: the heuristic is pretty well choosen. However, if you really need guaranteed 100% accurate data, generate your report directly from the raw logs. If you just want a quick overview, the merging is more suitable. Just make sure you're not cranking the limit parameters up too high in this case.
See also the Report Generating chapter in the Lire Architecture part of the Lire™ Developer's Manual.