]> a modular logfile analyser Jan Kneschke $Date: 2003/05/17 18:53:51 $ This is the manual for ModLogAn, a flexible logfile analyser written in C. Covering all topics of modlogan is not an easy task. The average user expaects a explaination for the generated output, admins want to know how configure modlogan in the way they want to and developers need a reference which helps them to understand how the code behaves/should behave. This manual try to reach these aim. To accomplish them the manual is sepearted into 4 parts. The green book for the user of modlogan who needs support in reading the output The yellow book for administrators which gives an overview how modlogan is intented to be used and provides several scenarios which can be used as a starting point for a initial setup The blue book is for developers and provides the documentation of the internals of modlogan. The black book is a reference and covers all configfile options and there behaviour. As it is generated from the source-code it is always fresh. ( If I could write a introduction I would write a real book ) The green give the user of modlogan an overview over the generated output. It explains every report in depth and provides extensive informations in the used words, descriptions and numbers. Before the output can be explained you have to know how and which input can be processed. As modlogan is a very flexible it supports many different input format from various programs. It starts with the standard webserver logfiles from a Apache and doesn't really stop with logfiles from mailservers like qmail or a sendmail. modlogan is based on three types of plugins: input (the parser) processor (take the data from the parser and pre-process them) output (the the pre-processed data and transform them into a viewable output) In the standard configuration modlogan parses webservers logs (clf), processes them (web) and generates a eye-friendly HTML output (modlogan) which also includes several graphics which help to get a overview what those number represent. It's the job of the parser to take the input, cut it in to nice pieces and to validate eash field if it contains valid data. We take the parser from the above mentioned standard configuration for a short looks into the internals of a parser. The input for the so called `clf' parser are records in the follow format: 212.126.17.130 - - [11/Jun/2000:07:23:15 +0200] ``GET /'' 200 2695 ``-'' ``-'' First we have an IP-address (212.126.17.130) of the client which has queryed this webserver at a given time (11/Jun/2000:07:23:15 +0200) for the index-page (GET /). The webserver fullfilled this request and returned with status code 200 which just means OK. The webserver transfered 2695 bytes to the client. The parser is exactly doing this job. It seperates the field, extacts the data and transforms them into the machine-readable format. If the parser is doing it job well the processor can take the parsed records and make calculations an them. The processor which is selected by the standard configuration for this job is the `web' processor which can process most of the logfiles which are generated by the servers which (in common sense) form `the web'. This include HTTP-server (apache, netscape, ms iis, ...), FTP-servers (wuftpd, proftpd, pureftpd, the ftpd which is shipped with FreeBSD) and PROXY-servers like Squid. All these servers have one thing in common. They are IP-based and are transfering files. Other processors are handling mail-traffic which has a different characteristic which can't be handled by the web processor. The web processor generates several statistics like: , , , and transfered bytes per day and hour generates the so called (user sessions) from the sessionless stream of input data. removes and combines several fields based on rules provided by the configfiles extracts some extra information from the records like searchengines, bookmarked pages und pages which were indexed by robots. These statistics are just containing raw data. You can see them into XML-files which are generated as the so called `state-files' which contains the internal state after the last record has been processed. As the XML which is generated by the processor is not readable for all humans it is transformed into a HTML with additional graphics which illustrate the huge amount of numbers. blas Hits Hits are the only physical data a webserver log provides. Each request that is set to the server is logged int eh logfile. These requests are called hits. Pages If you browse through a website your browser sends a request to the webserver which requests the initial HTML-Source and each in the HTML source inserted images. `Pages' are only those hits which deliver the HTML-source to the browser. Files Files classify the htis which resulted in a data transfer. Sometime the server doesn't have to transmit a reuested file as the browser already has the latest version in its cache. Only a transfer (hits) is finished with a status code 200 it is counted as File. Visits Foreword: Visits are a pure statistical value. Don't compare them between with logfile analyzer. They will vary. HTTP doesn't provide the data to do real visit counting. modlogan uses a very simple and easy to understand configfile format. The configuration can be done with a texteditor like emacs, vi or joe. Even notepad (for out windows uses) should be sufficent as the configfile is plain ASCII. Each configfile is divided into several sections. Each section starts with a sectionname. [section] The configvalue are following a very simple `key = value' rule. key = value whitespaces before and after the equal sign are ignored. Comments are only a starting with `#' as first character in a line. # comment A small example will show how to write dummy config-file: # this is a dummy configfile for the readers of the manual # let's start with a section named `dummy' [dummy] input = /etc/passwd outputfile = /tmp/secret The value of each `key-value-pair' can follow of these rules a plain string inputfile = /home/www/logs/access.log a colortripple col_circle = #E8F174 a regular exprecssion hideurl = "\.gif$" a group rule groupextension = "\.gif",GIF images Most logfile contain IP addresses which have to transformed into human-readable domainnames for the can really be processed. This is the job of a resolver. modlogan has an internal resolver which intented for all those users which don't want to install a `professional', external resolver. This internal is not tuned into any direction. Please don't complain about speed. Use a external resolver instead. The internal resolver is controlled by two configvariables: [global] = 1 [input_clf] = 1000 As already said, external resolvers are preferred. The following resolver are verified to work with modlogan: adnslogres jdresolve As most resolvers expect the input at stdin and write the resolved records to stdout they very easy to integrate into the logfile crunching process. $ gzip -cd /home/www/logs/access.log.gz | \ jdresolve <options> | \ modlogan -c <config-file> the is set to `-' in this case to read from stdin. In clustered server environments every webserver is generating its own logfile. modlogan is prepared process these logs without any external help. A external loadbalancer is dispatching request to two Netscape Enterprise Webservers. Both servers are generating their own logfiles. [global] = /path/to/modlogan/etc/ = modlogan.def.conf, global = input_netscape,input_lb_1 = input_netscape,input_lb_2 = output_null = processor_null = /path/to/output/ = 1 = 2 [processor_null] [output_null] [input_lb_1] = modlogan.def.conf, input_netscape = /path/to/logs/log.netscape.lb-1 [input_lb_2] = modlogan.def.conf, input_netscape = /path/to/logs/log.netscape.lb-2 The directive loads the plugin input_netscape with two different configurations. One is using the first logfile the other the logfile from the second webserver. Both input stream are merged based on the timestamps. As always the logfiles are expected to be chornological. price_per_unit subprocessor groupreceipient groupsender hidereceipient hidesender ignorereceipient ignoresender debug_resolver debug_searchengines debug_visits groupbrokenlinks groupextension grouphosts groupos groupreferrer grouprobots groupsearchstrings groupua groupurl hidebrokenlinks hideextension hidehost hidereferrer hiderobots hideurl ignorehost ignoreua ignoreurl pagetype to distiguish between pages and files you have to specify which urls should be counted as pages. searchengines splitby splits a stream of record into different output streams. the value is in the format: field,"regex",unique_name where field is the field which shall be checked, regex is the regular expression which shall be applied on the field and unique_name is the name the output stream. the unique_name is also used for the name of the directory which will be used to write the output into. (<outputdir>/<unique_name>/index.html) visit_timeout inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. matchos matchua format inputfile name of the file which will be used as the input source. use '-' for stdin as input source. matchos matchua readaheadlimit inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. split_provider in germany you have to dial the call-by-call provider id before the actual phone number. set split_provider to '1' if you want to split the provider from the dialed phone number. '0' disables this feature. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. serial_baud serial_databits serial_flow_hard serial_parity split_provider in germany you have to dial the call-by-call provider id before the actual phone number. set split_provider to '1' if you want to split the provider from the dialed phone number. '0' disables this feature. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. serial_baud serial_databits serial_flow_hard serial_parity split_provider in germany you have to dial the call-by-call provider id before the actual phone number. set split_provider to '1' if you want to split the provider from the dialed phone number. '0' disables this feature. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. matchos matchua inputfile name of the file which will be used as the input source. use '-' for stdin as input source. matchos matchua inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. inputfile name of the file which will be used as the input source. use '-' for stdin as input source. assumedprotocol background border content_charset content_language country_circle cssfile dont_cut_urls dont_escape_entities files foreground hits hostname htmlfooter htmlfooter specifies the file which should be included into the html-file to replace the original html-footer. NOTE: you have to take care that the html-page is closed. only the closing TABLE tag for the report is written to the outputfile. htmlheader if you want to replace the default page header, you can provide a filename here which will be included instead. NOTE: only the starting TABLE tag for the report is written. the rest is in the hand of the user -> you !! don't forget the DOCTYPE, HTML tags and the LINK to the STYLESHEETS. maxbookmarks maximum entries in the report named "Bookmarked Pages". a value of '0' disables this report. maxbrokenlinks maximum entries in the report named "Broken Links". a value of '0' disables this report. maxcountries maximum entries in the report named "Countries". a value of '0' disables this report. maxentrypages maximum entries in the report named "Entry Pages". a value of '0' disables this report. maxexitpages maximum entries in the report named "Exit Pages". a value of '0' disables this report. maxextensions maxhosts maximum entries in the report named "Hosts". a value of '0' disables this report. maxindexedpages maximum entries in the report named "Indexed Pages". a value of '0' disables this report. maxinteralerrors maximum entries in the report named "Internal Errors". a value of '0' disables this report. maxos maximum entries in the report named "Operating system". a value of '0' disables this report. maxrefurls maximum entries in the report named "Referrers". a value of '0' disables this report. maxreqmeth maximum entries in the report named "Request Method". a value of '0' disables this report. maxreqprot maximum entries in the report named "Request Protocol". a value of '0' disables this report. maxrequrls maximum entries in the report named "Requested URL's". a value of '0' disables this report. maxrobots maximum entries in the report named "Robots". a value of '0' disables this report. maxsearchengines maximum entries in the report named "SearchEngines". a value of '0' disables this report. maxsearchstrings maximum entries in the report named "SearchStrings". a value of '0' disables this report. maxstatuscodes maximum entries in the report named "Status Code". a value of '0' disables this report. maxua maximum entries in the report named "Browsers". a value of '0' disables this report. maxvhosts maxviewdurations maxvisitdurations maxvisitpathlengths maxvisitpaths pages pagestyle shadow showcountrygraph showdailygraph showhourlygraph showmonthlygraph showvalidationlinks showvhostgraph vhost_circle visits xfer hostname background body files grouping hits hostname hosts maxbookmarks maximum entries in the report named "Bookmarked Pages". a value of '0' disables this report. maxbrokenlinks maximum entries in the report named "Broken Links". a value of '0' disables this report. maxcountries maximum entries in the report named "Countries". a value of '0' disables this report. maxentrypages maximum entries in the report named "Entry Pages". a value of '0' disables this report. maxexitpages maximum entries in the report named "Exit Pages". a value of '0' disables this report. maxhosts maximum entries in the report named "Hosts". a value of '0' disables this report. maxindexedpages maximum entries in the report named "Indexed Pages". a value of '0' disables this report. maxinteralerrors maximum entries in the report named "Internal Errors". a value of '0' disables this report. maxos maximum entries in the report named "Operating system". a value of '0' disables this report. maxrefurls maximum entries in the report named "Referrers". a value of '0' disables this report. maxreqmeth maximum entries in the report named "Request Method". a value of '0' disables this report. maxreqprot maximum entries in the report named "Request Protocol". a value of '0' disables this report. maxrequrls maximum entries in the report named "Requested URL's". a value of '0' disables this report. maxrobots maximum entries in the report named "Robots". a value of '0' disables this report. maxsearchengines maximum entries in the report named "SearchEngines". a value of '0' disables this report. maxsearchstrings maximum entries in the report named "SearchStrings". a value of '0' disables this report. maxstatuscodes maximum entries in the report named "Status Code". a value of '0' disables this report. maxua maximum entries in the report named "Browsers". a value of '0' disables this report. pages shadow visits xfer background calls grouping hostname incoming maxcallednumbers maxcallingnumbers outgoing pagestyle shadow xfer assumedprotocol content_charset content_language country_circle dont_cut_urls dont_escape_entities hostname compress_mode set the compression level of the XML-files debug_level enable_resolver gen_report_threshold include include a section of a external configfile includepath incremental loadplugin load a plugin loadplugin = pluginname,[configsection] pluginname is the name of the plugin like input_clf, output_modlogan or processor_web. The plugin is configured by the options specified by the options in the specified configsection. If the field `configsection' is omitted the pluginname is used as name of the configsection. [global] loadplugin = input_clf loadplugin = processor_web, proc [input_clf] [proc] the plugin `input_clf' is using the options specified in the section `[input_clf]', `processor_web' is using the options from `[proc]'. outputdir directory where the generate files should be written to The output directory is used to save multiple files: the regular output like the HTML files the statefiles some meta-info (modlogan.statefiles) If you are using the rule the meaning of the outputdirectory changes a little bit.