SYNOPSIS
rklogd [ -c n ] [ -d ] [ -f fname ] [ -iI ] [ -n ] [ -o ] [ -p ] [ -s ]
[ -k fname ] [ -v ] [ -x ] [ -2 ]
DESCRIPTION
rklogd is a system daemon which intercepts and logs Linux kernel mes-
sages.
OPTIONS
-c n Sets the default log level of console messages to n.
-d Enable debugging mode. This will generate LOTS of output to
stderr.
-f file
Log messages to the specified filename rather than to the syslog
facility.
-i -I Signal the currently executing rklogd daemon. Both of these
switches control the loading/reloading of symbol information.
The -i switch signals the daemon to reload the kernel module
symbols. The -I switch signals for a reload of both the static
kernel symbols and the kernel module symbols.
-n Avoid auto-backgrounding. This is needed especially if the
rklogd is started and controlled by init(8).
-o Execute in 'one-shot' mode. This causes rklogd to read and log
all the messages that are found in the kernel message buffers.
After a single read and log cycle the daemon exits.
-p Enable paranoia. This option controls when rklogd loads kernel
module symbol information. Setting this switch causes rklogd to
load the kernel module symbol information whenever an Oops
string is detected in the kernel message stream.
-s Force rklogd to use the system call interface to the kernel mes-
sage buffers.
-k file
Use the specified file as the source of kernel symbol informa-
tion.
-v Print version and exit.
-x Omits EIP translation and therefore doesn't read the System.map
file.
-2 When symbols are expanded, print the line twice. Once with
/proc file system and the syscall (sys_syslog) interface, although
ultimately they are one and the same. Klogd is designed to choose
whichever source of information is the most appropriate. It does this
by first checking for the presence of a mounted /proc file system. If
this is found the /proc/kmsg file is used as the source of kernel log
information. If the proc file system is not mounted rklogd uses a sys-
tem call to obtain kernel messages. The command line switch (-s) can
be used to force rklogd to use the system call interface as its messag-
ing source.
If kernel messages are directed through the syslogd daemon the rklogd
daemon, as of version 1.1, has the ability to properly prioritize ker-
nel messages. Prioritization of the kernel messages was added to it at
approximately version 0.99pl13 of the kernel. The raw kernel messages
are of the form:
<[0-7]>Something said by the kernel.
The priority of the kernel message is encoded as a single numeric digit
enclosed inside the <> pair. The definitions of these values is given
in the kernel include file kernel.h. When a message is received from
the kernel the rklogd daemon reads this priority level and assigns the
appropriate priority level to the syslog message. If file output (-f)
is used the prioritization sequence is left pre-pended to the kernel
message.
The rklogd daemon also allows the ability to alter the presentation of
kernel messages to the system console. Consequent with the prioritiza-
tion of kernel messages was the inclusion of default messaging levels
for the kernel. In a stock kernel the the default console log level is
set to 7. Any messages with a priority level numerically lower than 7
(higher priority) appear on the console.
Messages of priority level 7 are considered to be 'debug' messages and
will thus not appear on the console. Many administrators, particularly
in a multi-user environment, prefer that all kernel messages be handled
by rklogd and either directed to a file or to the syslogd daemon. This
prevents 'nuisance' messages such as line printer out of paper or disk
change detected from cluttering the console.
When -c is given on the commandline the rklogd daemon will execute a
system call to inhibit all kernel messages from being displayed on the
console. Former versions always issued this system call and defaulted
to all kernel messages except for panics. This is handled differently
nowardays so rklogd doesn't need to set this value anymore. The argu-
ment given to the -c switch specifies the priority level of messages
which will be directed to the console. Note that messages of a prior-
ity value LOWER than the indicated number will be directed to the con-
sole.
For example, to have the kernel display all messages with a pri-
ority level of 3 (KERN_ERR) or more severe the following command
For example, to read all the kernel messages after a system boot
and record them in a file called krnl.msg the following command
would be given.
rklogd -o -f ./krnl.msg
KERNEL ADDRESS RESOLUTION
If the kernel detects an internal error condition a general protection
fault will be triggered. As part of the GPF handling procedure the
kernel prints out a status report indicating the state of the processor
at the time of the fault. Included in this display are the contents of
the microprocessor's registers, the contents of the kernel stack and a
tracing of what functions were being executed at the time of the fault.
This information is EXTREMELY IMPORTANT in determining what caused the
internal error condition. The difficulty comes when a kernel developer
attempts to analyze this information. The raw numeric information
present in the protection fault printout is of very little use to the
developers. This is due to the fact that kernels are not identical and
the addresses of variable locations or functions will not be the same
in all kernels. In order to correctly diagnose the cause of failure a
kernel developer needs to know what specific kernel functions or vari-
able locations were involved in the error.
As part of the kernel compilation process a listing is created which
specified the address locations of important variables and function in
the kernel being compiled. This listing is saved in a file called Sys-
tem.map in the top of the kernel directory source tree. Using this
listing a kernel developer can determine exactly what the kernel was
doing when the error condition occurred.
The process of resolving the numeric addresses from the protection
fault printout can be done manually or by using the ksymoops program
which is included in the kernel sources.
As a convenience rklogd will attempt to resolve kernel numeric
addresses to their symbolic forms if a kernel symbol table is available
at execution time. If you require the original address of the symbol,
use the -2 switch to preserve the numeric address. A symbol table may
be specified by using the -k switch on the command line. If a symbol
file is not explicitly specified the following filenames will be tried:
/boot/System.map
/System.map
/usr/src/linux/System.map
Version information is supplied in the system maps as of kernel 1.3.43.
This version information is used to direct an intelligent search of the
list of symbol tables. This feature is useful since it provides sup-
port for both production and experimental kernels.
a static kernel. Additional difficulties are encountered when attempt-
ing to diagnose errors which occur in loadable kernel modules. Load-
able kernel modules are used to implement kernel functionality in a
form which can be loaded or unloaded at will. The use of loadable mod-
ules is useful from a debugging standpoint and can also be useful in
decreasing the amount of memory required by a kernel.
The difficulty with diagnosing errors in loadable modules is due to the
dynamic nature of the kernel modules. When a module is loaded the ker-
nel will allocate memory to hold the module, when the module is
unloaded this memory will be returned back to the kernel. This dynamic
memory allocation makes it impossible to produce a map file which
details the addresses of the variable and functions in a kernel load-
able module. Without this location map it is not possible for a kernel
developer to determine what went wrong if a protection fault involves a
kernel module.
rklogd has support for dealing with the problem of diagnosing protec-
tion faults in kernel loadable modules. At program start time or in
response to a signal the daemon will interrogate the kernel for a list-
ing of all modules loaded and the addresses in memory they are loaded
at. Individual modules can also register the locations of important
functions when the module is loaded. The addresses of these exported
symbols are also determined during this interrogation process.
When a protection fault occurs an attempt will be made to resolve ker-
nel addresses from the static symbol table. If this fails the symbols
from the currently loaded modules are examined in an attempt to resolve
the addresses. At the very minimum this allows rklogd to indicate
which loadable module was responsible for generating the protection
fault. Additional information may be available if the module developer
chose to export symbol information from the module.
Proper and accurate resolution of addresses in kernel modules requires
that rklogd be informed whenever the kernel module status changes. The
-i and -I switches can be used to signal the currently executing daemon
that symbol information be reloaded. Of most importance to proper res-
olution of module symbols is the -i switch. Each time a kernel module
is loaded or removed from the kernel the following command should be
executed:
rklogd -i
The -p switch can also be used to insure that module symbol information
is up to date. This switch instructs rklogd to reload the module sym-
bol information whenever a protection fault is detected. Caution
should be used before invoking the program in 'paranoid' mode. The
stability of the kernel and the operating environment is always under
question when a protection fault occurs. Since the rklogd daemon must
execute system calls in order to read the module symbol information
there is the possibility that the system may be too unstable to capture
useful information. A much better policy is to insure that rklogd is
SIGTERM, SIGTSTP, SIGUSR1, SIGUSR2 and SIGCONT. The SIGINT, SIGKILL,
SIGTERM and SIGHUP signals will cause the daemon to close its kernel
log sources and terminate gracefully.
The SIGTSTP and SIGCONT signals are used to start and stop kernel log-
ging. Upon receipt of a SIGTSTP signal the daemon will close its log
sources and spin in an idle loop. Subsequent receipt of a SIGCONT sig-
nal will cause the daemon to go through its initialization sequence and
re-choose an input source. Using SIGSTOP and SIGCONT in combination
the kernel log input can be re-chosen without stopping and restarting
the daemon. For example if the /proc file system is to be un-mounted
the following command sequence should be used:
# kill -TSTP pid
# umount /proc
# kill -CONT pid
Notations will be made in the system logs with LOG_INFO priority docu-
menting the start/stop of logging.
The SIGUSR1 and SIGUSR2 signals are used to initiate loading/reloading
of kernel symbol information. Receipt of the SIGUSR1 signal will cause
the kernel module symbols to be reloaded. Signaling the daemon with
SIGUSR2 will cause both the static kernel symbols and the kernel module
symbols to be reloaded.
Provided that the System.map file is placed in an appropriate location
the signal of generally greatest usefulness is the SIGUSR1 signal.
This signal is designed to be used to signal the daemon when kernel
modules are loaded/unloaded. Sending this signal to the daemon after a
kernel module state change will insure that proper resolution of sym-
bols will occur if a protection fault occurs in the address space occu-
pied by a kernel module.
FILES
/proc/kmsg
One Source for kernel messages rklogd
/var/run/rklogd.pid
The file containing the process id of rklogd
/boot/System.map, /System.map, /usr/src/linux/System.map
Default locations for kernel system maps.
BUGS
Probably numerous. Well formed context diffs appreciated.
AUTHOR
The rklogd was originally written by Steve Lord (lord@cray.com), Greg
Wettstein made major improvements.
Dr. Greg Wettstein (greg@wind.enjellic.com)
Man(1) output converted with
man2html