Lire User's Manual

Joost van Baal

Wessel Dankers

Francis J. Lacoste

Egon L. Willighagen

This manual is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this manual (see COPYING); if not, check with http://www.gnu.org/copyleft/gpl.html or write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111, USA.

Revision History
Revision 1.4rc1$Date: 2003/08/27 16:05:06 $
$Id: user-manual.dbx,v 1.74 2003/08/27 16:05:06 vanbaal Exp $

Table of Contents

Preface
What This Book Contains
How Is This Book Organized?
Conventions Used
If You Don't Find Something In This Manual
I. Lire Overview
1. Introducing Lire
What Is Lire?
Supported Systems
Supported Applications
Supported Output Format
What Lire Can't Do
2. Installing Lire
Client Installation
Requirements
Installing
Standalone Installation
Requirements
Installing
Anonymized Client Installation
Requirements
Installing
Responder Installation
Requirements
Installation
Installing Under MTA's using procmail as their MDA
Installing Under Exim
Installing Under Postfix
Installing Under qmail
Making The Responder Run At Boot
3. Running Lire
Lire's configuration system
Getting Started
Using A Responder
Generating A Report From A Log File
Selecting Output Format
Including Charts in the Report
Merging Reports
Manual Merging
Automating Merging using cron: some ideas
Gotchas
Sending Anonymized Log Files To A Responder
Processing The Responder's Results
Running Lire In A Server Cluster
Using Mail
Using Syslog
4. Automating Lire
Automatically Processing Log Files Using Cron
Configuring lr_cron
Installing the Cron Job
Automatically Processing Log Files Through A Responder
Automatically Processing Log Files In A Server Farm
5. Customizing Lire's Reports
The Report's Configuration File
Selecting Subreports
Reordering The Subreports
Changing Parameters
Using Subreports On Filtered Input
II. Reports Reference
6. Database Reports
Supported Log Format
MySQL's Log
Report Descriptions and Configuration
Actions By Period Database Report
Most Active Users Database Report
Most Accessed Databases Database Report
Queries By Type
Filter Descriptions and Configuration
7. Dialup Reports
Supported Log Format
Linux kernel 2.4.x isdnlog
Report Descriptions and Configuration
8. DNS Reports
Supported Log Format
Bind8 Query Log
Bind9 Query Log
Report Descriptions and Configuration
Top Requesting Hosts Report
Top Requesting Hosts Report
Top Requested Names Report
Distribution of Request Types by Method DNS Report
Distribution of Request Types Report
Requests By Period DNS Report
Requests By Timeslot DNS Report
Requests by Period by Method DNS Report
Requests by Timeslot by Method DNS Report
Filter Descriptions and Configuration
Select Resolver Filter
9. DNS Zone Reports
Supported Log Format
Report Descriptions and Configuration
Top Requesting AXFR Hosts Report
Top Requesting AXFR Hosts Report
Top Requesting AXFR Hosts Report
Top Denied AXFR Requests
Top Denied Dynamic DNS Updates Report
Top Newly Loaded Zones Report
Filter Descriptions and Configuration
AXFR denied only
AXFR denied only
AXFR denied only
Loaded zones only
Loaded zones only
10. Email Reports
Supported Log Format
ArGoSoft Mail Server
Exim
Netscape Messaging Server
Postfix
Qmail
Sendmail
Reports' Descriptions and Configuration
Deliveries Attempts By Period By Status Email Report
Deliveries Attempts By Period Email Report
Deliveries Attempts By Delay Email Report
Deliveries Attempts By Size Email Report
Failed Deliveries By Relay Email Report
Different From Domain by Period Email Report
Different From Email for Selected Domains by Period Email Report
Different To Domain by Period Email Report
Different To Email for Selected Domains by Period Email Report
Highest Average Delay By To Relay And To Domain Email Report
Most Deliveries Between Relays Email Report
Most Deliveries From Domain Email Report
Most Deliveries From User By Domain Email Report
Most Deliveries From Relay Email Report
Largest Email Exchange Email Report
Largest Email Exchange Per Relay Pair Email Report
Status Summary For Most Common Domains
Most Deliveries To Domain Email Report
Most Deliveries To User By Domain Email Report
Most Deliveries From Relay Email Report
Largest Volume Received From Domain Email Report
Largest Volume Sent To Domain Email Report
Tracked Recipients Email Report
Tracked Senders Email Report
Volume Delivered By Period Email Report
Email Summary
Filters' Descriptions and Configuration
Select Client IP Filter
11. Firewall Reports
Supported Log Format
Cisco ACL
IPChains
IP Filter
IPTables
WebTrends Enhanced Log Format
Report Descriptions and Configuration
Bytes by Period Firewall Report
Traffic's Volume by Rule Firewall Report
Bytes by Timeslot Firewall Report
Top Bytes per From-IP Report
Top Bytes per From-IP per Port Report
Top Bytes per To-ip Report
Top Bytes per destination IP per Port Report
Top blocked tcp packets per source IP per destination port Report
Packets by Period Firewall Report
Packets by Rule Firewall Report
Packets by Timeslot Firewall Report
Top Volume to Destination by Source Firewall Report
Top Volume to Destination by Source Firewall Report
Top Messages Firewall Report
Top Messages Firewall Report
Top Messages Firewall Report
Top Packets by Source IP Report
Top Packets by Destination IP Report
Top Packets by Destination by Port Firewall Report
Top Packets Destination by Source Firewall Report
Top Packets Source by Destination Firewall Report
Filter Descriptions and Configuration
Select Action Filter
Select Source IP Filter
Select Destination IP Filter
12. FTP Reports
Supported Log Format
Microsoft Internet Information Server
Xferlog
Report Descriptions and Configuration
Top Remote Host FTP Report
Bytes by Period FTP Report
Bytes by User by Period FTP Report
Bytes by Period by User FTP Report
Bytes by Direction by User with count by Period FTP Report
Top Files FTP Report
Top Uploaded Files FTP Report
Top Downloaded Files FTP Report
Top Users FTP Report
Top by User (Bytes Transferred) FTP Report
Tracked Users FTP Report
Tracked Files FTP Report
Number of Transfers by Direction FTP Report
Number of Transfers by Transfer Type FTP Report
Each Transfer by Filename Report
Filter Descriptions and Configuration
13. Message Store Reports
Supported Log Formats
Report Descriptions and Configuration
Failed Logins by Period Message Store Report
Closed Event by Period Message Store Report
Events by Protocol Message Store Report
Successful Login by Period Message Store Report
Top User Logins Message Store Report
Top User Most Message Leftover in Store Report
Top User Most Message Leftover Store Report
Top User Most Message Store Report
Top User Most Message Store Report
Top Users doing Select Message Store Report
Unique Users by Period Message Store Report
Filter Descriptions and Configuration
Select Client Host Filter
14. Print Reports
Supported Log Format
CUPS page_log
LPRng Account Log File
Report Descriptions and Configuration
Jobs per Printer Print Report
Top Users Print Report
Number Of Jobs For Each Number Of Sheets
Jobs per Period Print Report
Jobs per User per Period Print Report
Jobs per Printer per Period Print Report
Sheets per User Print Report
Sheets per Period Print Report
Sheets per User per Period Print Report
Billing Report
Billing per Printer Report
Filter Descriptions and Configuration
15. Proxy Reports
Supported Log Formats
Microsoft Internet Security and Acceleration Server
Squid
WebTrends Enhanced Format
Report Descriptions and Configuration
Bytes by Cache Result
Bytes by Object's Source
Bytes Transferred By Period Proxy Report
Bytes Transferred By Timeslot Proxy Report
Requests by Cache Result
Requests By Period Proxy Report
Requests By Size Proxy Report
Number of Requests By Timeslot Proxy Report
Requests By Request's Time Proxy Report
Top Clients by Destinations Proxy Report
Top Destinations by Number of Requests
Top Destinations by Bytes Downloaded
Top Destinations by Clients
Top Destinations by Users Proxy Report
Top Users by Destinations Proxy Report
Top MIME types by Transferred Size
Top Users by Bytes Proxy Report
Top URLs by Users Proxy Report
Filter Descriptions and Configuration
Select Cache Result Filter
16. Syslog Reports
Supported Log Formats
Report Descriptions and Configuration
Messages by Facility Syslog Report
Messages by Level Syslog Report
Messages by Period Syslog Report
Top Hosts Syslog Report
Top Messages by Period Syslog Report
Top Messages by Process Syslog Report
Top Messages Syslog Report
Top Processes by Period Syslog Report
Top Processes Syslog Report
Filter Descriptions and Configuration
Exclude Message Filter
Exclude Priority Filter
Exclude Process Filter
Select Host Filter
Select Message Filter
Select Priority Filter
Select process Filter
17. WWW Reports
Supported Log Format
Common Log Format
Combined Log Format
CLF With mod_gzip Extensions
Referer Log Format
Logs With Virtual Host Information
W3C Extended Log Format
Report Descriptions and Configuration
Bytes By Period WWW Report
Bytes Per Directory WWW Report
Bytes By HTTP Result By Period WWW Report
Bytes By HTTP Result WWW Report
Bytes Per Request WWW Report
Client Hosts By Period WWW Report
Search Engines with Keywords Report
Requests By Browser WWW Report
Number of Requests By Period WWW Report
Requests By Browser Language WWW Report
Requests By HTTP Method WWW Report
Requests By OS WWW Report
Requests By Result By Period WWW Report
Requests By HTTP Result WWW Report
Requests By Gzip Result WWW Report
Requests By Robot Report
Requests By Top Level Domain Report
Requests By Attack Report
Requests By Keywords Report
Requests By User Agent WWW Report
Requests By Search Engines Report
Number of Requests By Size WWW Report
Number of Requests By Timeslot WWW Report
Requests By HTTP Protocol Version WWW Report
Average Compression By File Type WWW Report
Most Averaged Compressed Requested File WWW Report
Top Client By HTTP Result WWW Report
Top Client by Size WWW Report
Top Client WWW Report
Last Pages By Session WWW Report
First Pages By Session WWW Report
Most Travelled Referer -> Page Connections WWW Report
Top Referring Pages WWW Report
Top Referring Pages By Requested Page WWW Report
Top Referring Sites WWW Report
Most Requested Pages WWW Report
Top Traversals WWW Report
Top URLs By HTTP Result WWW Report
Most Requested URLs By Client Host WWW Report
User Sessions By Period WWW Report
Recurring Visitors WWW Report
Visit times User Session WWW Report
Page Counts User Session WWW Report
Filter Descriptions and Configuration
Select URL Filter
Select Sessions by Page Filter
Select Client Host Filter
Exclude URL Filter
Exclude Sessions by Page Filter
Exclude Client Host Filter
Exclude Referer Filter
III. Lire Reference
18. Installation Parameters
./configure parameters
Installation Environment Variables
19. Lire Logging and Error Messages
Logging
Log Messages
20. Lire Installation Layout

List of Examples

3.1. Sending a Log File For Processing To A Responder
3.2. Generating a Report With lr_log2report
3.3. Generating A HTML Report
3.4. Generating A HTML Report With Charts
3.5. Merging Reports
3.6. Sending An Anonymized Postfix Log File To A Responder
3.7. Deanonymizing and Generating A HTML Report
5.1. Commented Report Configuration File
5.2. FTP Report Configuration File
6.1. Sample MySQL Log File
8.1. Enabling Query Log In Bind
8.2. Sample Bind 8 Query Log
8.3. Sample Bind 9 Query Log
10.1. ArGoSoft Mail Server Log Sample
10.2. Exim Log Sample
10.3. Netscape Messaging Server Log Sample
10.4. Postfix Log Sample
10.5. Qmail Log Sample
10.6. Sendmail Log Sample
11.1. IOS Log Sample
11.2. IPChains Log Sample
11.3. IP Filter Log Sample
11.4. IPTables Log Sample
11.5. WELF Log Sample
11.6. SonicWall Log Sample
12.1. Microsoft Internet Information Server FTP Log Sample
12.2. Xferlog Log Sample
14.1. CUPS page_log Log Sample
14.2. LPRng Log Sample
15.1. Microsoft Internet Security and Acceleration Server Log Sample
15.2. Squid Log Sample
15.3. WELF Log Sample