=pod

=head1 NAME

B<rwbagtool> - Perform high-level operations on binary Bag files

=head1 SYNOPSIS

  rwbagtool [BAGFILE[,BAGFILE...]] 
        { --add | --intersect=SETFILE | 
          --complement-intersect=SETFILE |
          --divide | --subtract | --coverset | --invert }
        [--mincounter=VALUE] [--maxcounter=VALUE]
        [--minkey=VALUE] [--maxkey=VALUE]
        [--compression-method=COMP_METHOD] [--output=OUTPUTFILE]

=head1 DESCRIPTION

B<rwbagtool> performs various operations on Bags.  It can add Bags
together, subtract a subset of data from a Bag, perform key
intersection of a Bag with an IP set, extract the key list of a Bag as
an IP set, or filter Bag records based on their counter value.

I<BAGFILE> is a filename, named pipe, or the keyword C<stdin>.  If no
Bag file names are given on the command line, B<rwbagtool> attempts to
read a Bag from the standard input.  If I<BAGFILE> does not contain a
Bag, B<rwbagtool> prints an error to stderr and exits abnormally.

=head1 OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an
exact match for an option.  A parameter to an option may be specified
as B<--arg>=I<param> or S<B<--arg> I<param>>, though the first form is
required for options that take optional parameters.

=over 4

=item B<--add>

Adds all Bag files given on the command line.  If any file cannot be
parsed as a Bag, B<rwbagtool> prints an error to stderr and exits
abnormally.  Key values are unioned and counter values are summed.  If
a key does not exist, it has a counter of zero.

=item B<--intersect>=I<SETFILE>

Adds Bag files as described for B<--add>, then masks the keys in the
resulting Bag using the set in I<SETFILE>.  I<SETFILE> is a filename,
named pipe, or the keyword C<stdin> containing an IP set.  If
I<SETFILE> does not contain an IP set, B<rwbagtool> prints an error to
stderr and exits abnormally.  Only keys which match an entry in
I<SETFILE> are output.

=item B<--complement-intersect>=I<SETFILE>

As B<--intersect>, but only keys which do B<not> match an entry in
I<SETFILE> are output.

=item B<--coverset>

Adds Bag files as described for B<--add>, then outputs an IPset which
contains the keys contained in the resulting Bag.

=item B<--invert>

Adds Bag files as described for B<--add>, then generates a new Bag
whose keys are the counters in the intermediate Bag and whose counter
is the number of times the counter was seen.  For example, this turns
the Bag <sip:flow> into the Bag <flow:count(sip)>.  Any counter in the
intermediate Bag that is larger than the maximum possible key will be
attributed to the maximum key; to prevent this, specify
C<--maxcounter=4294967295>.

=item B<--subtract>

Subtracts from the first Bag file all subsequent Bag files.  If any
inputs cannot be parsed as a Bag, B<rwbagtool> prints an error to
stderr and exits abnormally.  If any counter subtraction results in a
negative number, B<rwbagtool> prints an error to stderr and exits
abnormally.

=item B<--divide>

Divides the first Bag file by all subsequent Bag files.  For example,

  ( ( BAGFILE1 / BAGFILE2 ) / BAGFILE3 ) ...

All Bags must have the same keys (i.e., all Bags must have the same
keys with non-zero counters), or else B<rwbagtool> prints an error and
exits abnormally.  Likewise, Bags do not support fractional values.
To work around this limitation, all counters in the first Bag must be
at least as large as the counter for the matching key in subsequent
Bags.

Fractional remainders after division are truncated.

=item B<--mincounter>=I<VALUE>

Causes the output to contain only those records whose counter value is
I<VALUE> or higher.  Default is 1.

=item B<--maxcounter>=I<VALUE>

Causes the output to contain only those records whose counter value is
I<VALUE> or lower.  Default is the maximum counter value.

=item B<--minkey>=I<VALUE>

Causes the output to contain only those records whose key value is
I<VALUE> or higher.  Default is 0.0.0.0.  Accepts input as an integer
or as an IP address in dotted decimal notation.

=item B<--maxkey>=I<VALUE>

Causes the output to contain only those records whose key value is
I<VALUE> or higher.  Default is 255.255.255.255.  Accepts input as an
integer or as an IP address in dotted decimal notation.

=item B<--output>=I<OUTPUTFILE>

Redirect output to I<OUTPUTFILE>.  I<OUTPUTFILE> is a filename,
named pipe, or the keyword C<stdout>.

=item B<--compression-method>=I<COMP_METHOD>

Set the compression method of the output to I<COMP_METHOD>.  Some SiLK
tools can use an external library to compress their binary output.
The list of available compression methods and the default method are
set when SiLK is compiled (the B<--help> and B<--version> switches
print the available and default compression methods) and depend on
which supported libraries are found.  SiLK can support:

=over 4

=item none

Do not compress the output using an external library

=item zlib

Use the B<zlib(3)> library for compressing the output

=item lzo1x

Use the I<lzo1x> algorithm from the LZO real time compression library
for compression

=item best

Use whichever available method gives the C<best> compression in
general, though not necessarily the C<best> for this particular
output.

=back

=back

=head1 EXAMPLES

To add F<bag1.bag> and F<bag2.bag> together:

  rwbagtool --add bag1.bag bag2.bag > newbag.bag

To only store entries with a counter of 3:

  rwbagtool bag1.bag --mincounter=3 --maxcounter=3 > count3.bag

To build the coverset for that Bag:

  rwbagtool count3.bag --coverset > count3.set

=head1 BUGS

Prior to SiLK-0.9.9, B<rwbagtool> always attempted to read its first
Bag file from the standard input for the B<--add>, B<--divide>,
B<--subtract>, and B<--coverset> operations.  However, this prevented
B<rwbagtool> from being used in non-interactive situations (such as by
B<cron(8)> or behind an HTTP server), since B<rwbagtool> would exit
when it failed to read a (nonexistent) Bag file from the standard
input.  To return to the former functionality, explicitly give
C<stdin> as the name of the first Bag file.

=head1 SEE ALSO

B<rwbag(1)>, B<rwbagbuild(1)>, B<rwbagcat(1)>, B<rwset(1)>

=cut

$SiLK: rwbagtool.pod 7514 2007-06-14 12:20:54Z mthomas $

Local Variables:
mode:text
indent-tabs-mode:nil
End: