=pod

=head1 NAME

B<rwstats> - Print interval counts or top-N or bottom-N lists

=head1 SYNOPSIS

  rwstats [--overall-stats] [--detail-proto-stats=PROTO[,PROTO]]
        [{--sip | --sip=CIDR}] [{--dip | --dip=CIDR}]
        [--sport] [--dport] [--protocol] [--icmp]
        [--flows] [--packets] [--bytes]
        [--count=N] [--threshold=N] [--percentage=N] [--top]
        [--bottom] [--integer-ips] [--no-titles] [--no-columns]
        [--column-separator=CHAR] [{--delimited | --delimited=CHAR}]
        [--print-filenames] [--copy-input=PATH] [--output-path=PATH]
        [--pager=PAGER_PROG] [--site-config-file=FILENAME]
        [--legacy-help] [FILES...]

=head1 DESCRIPTION

B<rwstats> reads SiLK Flow records from files named on the command
line or from the standard input, bins those records by the
user-specified specified key, computes the volume (record count
or sum of packets or bytes) per bin, sorts the bins by volume,
and prints the bins with the greatest or least volume.
Alternatively, B<rwstats> can provide statistics for each of bytes,
packets, and bytes-per-packet giving minima, maxima, quartile,
and interval flow-counts across all flows or across a list of
protocols specified by the user.

The number of bins printed can be specified as a fixed value
(e.g., print 10 bins), as a threshold (print bins whose byte
count is less than 400), or as a percentage of the total volume
across all bins (print bins who that contain at least 10% of all
the packets).

RWSTATS WILL POTENTIALLY RUN OUT OF MEMORY.  See the BUGS
section for details.

=head1 OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an
exact match for an option.  A parameter to an option may be specified
as B<--arg>=I<param> or B<--arg> I<param>, though the first form is
required for options that take optional parameters.

The following switches will compute and print, for each of bytes,
packets, and bytes per packet, the minimum value, the maximum value,
quartiles, and a count of the number of flows that fall into each of
one of ten intervals statistics.  These switches cannot be combined
with the switches that produce Top-N or Bottom-N lists.

=over 4

=item B<--overall-stats>

Print intervals and quartiles across all flows that were read by
B<rwstats>.

=item B<--detail-proto-stats>=I<PROTO[,PROTO...]>

Print intervals and quartiles for each individual protocol listed as
an argument.  The argument should be a comma separated list of
protocols or ranges of protocols: C<1-6,17>.  Specifying this option
implies B<--overall-stats>.

=back

To compute a Top-N or Bottom-N list, one or two key fields must be
specified.  Currently, B<rwstats> only supports pairing B<--sip> with
B<--dip> or pairing B<--sport> with B<--dport>.

=over 4

=item B<--sip>

=item B<--sip>=I<CIDR>

Use the source address as the key.  When I<CIDR> is given, only the
most significant I<CIDR> bits of the source address are considered;
i.e., the least significant 32-I<CIDR> bits of the source address are
set to 0.  May be used with B<--dip> to key by IP-pairs.

=item B<--dip>

=item B<--dip>=I<CIDR>

Use the destination address as the key.  When I<CIDR> is given, only
the most significant I<CIDR> bits of the destination address are
considered; i.e., the least significant 32-I<CIDR> bits of the
destination address are set to 0.  May be used with B<--sip> to key by
IP-pairs.

=item B<--sport>

Use the source port as the key.  May be used with B<--dport> to key by
port-pairs.

=item B<--dport>

Use the destination port as the key.  May be used with B<--sport> to key by
port-pairs.

=item B<--protocol>

Use the protocol as the key.

=item B<--icmp>

Use the ICMP type and code as the key.  This switch assumes that
I<all> the records fed to B<rwstats> are ICMP.  It is not possible to
key solely on the ICMP type or the ICMP code.

=back

When computing a Top-N or Bottom-N, all flows that have the same key
field(s) will be binned together.  The C<value> of each bin may be
based on the number of flows or on the sum of packets or bytes in the
flows.  If none of the following value field switches is specified,
B<--flows> will be used.

=over 4

=item B<--flows>

Use the flow count as the value.

=item B<--packets>

Use the packet count as the value.

=item B<--bytes>

Use the byte count as the value.

=back

To determine the value of N for a Top-N (or Bottom-N) list, one
of the following switches must be specified:

=over 4

=item B<--count>=I<N>

Print the specified number of key-value pairs with the
largest (or smallest) values.

=item B<--threshold>=I<N>

Print key-value pairs where the value is greater-than (or
less-than) this threshold.

=item B<--percentage>=I<N>

Print key-value pairs where the value is greater-than (or
less-than) this percentage of the total value.

=back

To determine whether to compute the Top-N or the Bottom-N,
specify one of the following switches.  If neither switch is
given, B<--top> is assumed:

=over 4

=item B<--top>

Print the top N keys and their values.

=item B<--bottom>

Print the bottom N keys and their values.

=back

Miscellaneous switches:

=over 4

=item B<--integer-ips>

Print IP addresses as integers.  If not specified, IP addresses are
printed in dotted decimal form.

=item B<--no-titles>

Disable section and column titles.  By default, titles are
printed.

=item B<--no-columns>

Disable fixed-width columnar output.

=item B<--column-separator>=I<CHAR>

Use specified character between columns.  If not specified, the
default of 'B<|>' is used.

=item B<--delimited>

=item B<--delimited>=I<CHAR>

Run as if B<--no-columns> B<--column-sep>=I<CHAR> had been specified.
That is, disable fixed-width column output; if character I<CHAR> is
provided, it is used as the delimiter between columns instead of the
default 'B<|>'.

=item B<--print-filenames>

Prints to the standard error the names of input files as they are
opened.

=item B<--copy-input>=I<PATH>

Copy all binary input to the specified file or named pipe.  I<PATH>
can be C<stdout> to print flows to the standard output as long as the
B<--output-path> switch has been used to redirect B<rwaddrcount>'s
ASCII output.

=item B<--output-path>=I<PATH>

Determines where the output of B<rwaddrcount> (ASCII text) is written.
If this option is not given, output is written to the standard output.

=item B<--pager>=I<PAGER_PROG>

When output is to a terminal, invokes the program I<PAGER_PROG> to
view the output one screen full at a time.  This switch overrides the
SILK_PAGER environment variable, which in turn overrides the PAGER
variable.  If the value of the pager is determined to be the empty
string, no paging will be performed and all output will be printed to
the terminal.

=item B<--site-config-file>=I<FILENAME>

Read the SiLK site configuration from the named file I<FILENAME>.
When this switch is not provided, the location specified by the
C<$SILK_CONFIG_FILE> environment variable is used if that variable is
not empty.  The value of C<$SILK_CONFIG_FILE> should include the name
of the file.  Otherwise, the application looks for a file named
F<silk.conf> in the following directories: the directory specified in
the F<$SILK_DATA_ROOTDIR> environment variable; the data root
directory that is compliled into SiLK (use the B<--version> switch to
view this value); the directories F<$SILK_PATH/share/silk/>,
F<$SILK_PATH/share/>, or F<$SILK_PATH>; or in the F<share/silk/> and
F<share/> directories parallel to the application's directory.

=item B<--legacy-help>

Print help, including legacy switches.  See the LEGACY SWITCHES
section below for these switches.

=back

=head2 LEGACY SWITCHES

Use of the following switches is discouraged; instead, use the
replacement switches as indicated.

=over 4

=item B<--sip-topn>=I<N>

Use: C<--sip [--top] [--flows] --count=I<N>>

=item B<--sip-top-threshold>=I<N>

Use: C<--sip [--top] [--flows] --threshold=I<N>>

=item B<--sip-top-pct>=I<N>

Use: C<--sip [--top] [--flows] --percentage=I<N>>

=item B<--sip-btmn>=I<N>

Use: C<--sip --bottom [--flows] --count=I<N>>

=item B<--sip-btm-threshold>=I<N>

Use: C<--sip --bottom [--flows] --threshold=I<N>>

=item B<--sip-btm-pct>=I<N>

Use: C<--sip --bottom [--flows] --percentage=I<N>>

=item B<--dip-topn>=I<N>

Use: C<--dip [--top] [--flows] --count=I<N>>

=item B<--dip-top-threshold>=I<N>

Use: C<--dip [--top] [--flows] --threshold=I<N>>

=item B<--dip-top-pct>=I<N>

Use: C<--dip [--top] [--flows] --percentage=I<N>>

=item B<--dip-btmn>=I<N>

Use: C<--dip --bottom [--flows] --count=I<N>>

=item B<--dip-btm-threshold>=I<N>

Use: C<--dip --bottom [--flows] --threshold=I<N>>

=item B<--dip-btm-pct>=I<N>

Use: C<--dip --bottom [--flows] --percentage=I<N>>

=item B<--pair-topn>=I<N>

Use: C<--sip --dip [--top] [--flows] --count=I<N>>

=item B<--pair-top-threshold>=I<N>

Use: C<--sip --dip [--top] [--flows] --threshold=I<N>>

=item B<--pair-top-pct>=I<N>

Use: C<--sip --dip [--top] [--flows] --percentage=I<N>>

=item B<--pair-btmn>=I<N>

Use: C<--sip --dip --bottom [--flows] --count=I<N>>

=item B<--pair-btm-threshold>=I<N>

Use: C<--sip --dip --bottom [--flows] --threshold=I<N>>

=item B<--pair-btm-pct>=I<N>

Use: C<--sip --dip --bottom [--flows] --percentage=I<N>>

=item B<--sport-topn>=I<N>

Use: C<--sport [--top] [--flows] --count=I<N>>

=item B<--sport-top-threshold>=I<N>

Use: C<--sport [--top] [--flows] --threshold=I<N>>

=item B<--sport-top-pct>=I<N>

Use: C<--sport [--top] [--flows] --percentage=I<N>>

=item B<--sport-btmn>=I<N>

Use: C<--sport --bottom [--flows] --count=I<N>>

=item B<--sport-btm-threshold>=I<N>

Use: C<--sport --bottom [--flows] --threshold=I<N>>

=item B<--sport-btm-pct>=I<N>

Use: C<--sport --bottom [--flows] --percentage=I<N>>

=item B<--dport-topn>=I<N>

Use: C<--dport [--top] [--flows] --count=I<N>>

=item B<--dport-top-threshold>=I<N>

Use: C<--dport [--top] [--flows] --threshold=I<N>>

=item B<--dport-top-pct>=I<N>

Use: C<--dport [--top] [--flows] --percentage=I<N>>

=item B<--dport-btmn>=I<N>

Use: C<--dport --bottom [--flows] --count=I<N>>

=item B<--dport-btm-threshold>=I<N>

Use: C<--dport --bottom [--flows] --threshold=I<N>>

=item B<--dport-btm-pct>=I<N>

Use: C<--dport --bottom [--flows] --percentage=I<N>>

=item B<--portpair-topn>=I<N>

Use: C<--sport --dport [--top] [--flows] --count=I<N>>

=item B<--portpair-top-threshold>=I<N>

Use: C<--sport --dport [--top] [--flows] --threshold=I<N>>

=item B<--portpair-top-pct>=I<N>

Use: C<--sport --dport [--top] [--flows] --percentage=I<N>>

=item B<--portpair-btmn>=I<N>

Use: C<--sport --dport --bottom [--flows] --count=I<N>>

=item B<--portpair-btm-threshold>=I<N>

Use: C<--sport --dport --bottom [--flows] --threshold=I<N>>

=item B<--portpair-btm-pct>=I<N>

Use: C<--sport --dport --bottom [--flows] --percentage=I<N>>

=item B<--proto-topn>=I<N>

Use: C<--protocol [--top] [--flows] --count=I<N>>

=item B<--proto-top-threshold>=I<N>

Use: C<--protocol [--top] [--flows] --threshold=I<N>>

=item B<--proto-top-pct>=I<N>

Use: C<--protocol [--top] [--flows] --percentage=I<N>>

=item B<--proto-btmn>=I<N>

Use: C<--protocol --bottom [--flows] --count=I<N>>

=item B<--proto-btm-threshold>=I<N>

Use: C<--protocol --bottom [--flows] --threshold=I<N>>

=item B<--proto-btm-pct>=I<N>

Use: C<--protocol --bottom [--flows] --percentage=I<N>>

=item B<--cidr-src>=I<N>

Use: C<--sip=I<N>>

=item B<--cidr-dest>=I<N>

Use: C<--dip=I<N>>

=back

=head1 EXAMPLES

  $ rwfilter ...| rwstats --sip --count=4
  INPUT SIZE: 549092 records for 12990 unique keys
  SOURCE IP Key: Top 4 flow counts
          sIP|     Records|%_of_total|   cumul_%|
     10.1.1.1|       36604|  6.666278|  6.666278|
     10.1.1.2|       13897|  2.530906|  9.197184|
     10.1.1.3|       12739|  2.320012| 11.517196|
     10.1.1.4|       11807|  2.150277| 13.667473|

  $ rwfilter ...| rwstats --dip --packet --count=7
  INPUT SIZE: 549092 records for 44654 unique keys
  DESTINATION IP Key: Top 7 packet counts
          dIP|     Packets|%_of_total|   cumul_%|
     10.1.1.1|      217574|  3.286325|  3.286325|
     10.1.1.2|      138177|  2.087081|  5.373407|
     10.1.1.3|      121892|  1.841106|  7.214512|
     10.1.1.4|       97073|  1.466230|  8.680742|
     10.1.1.5|       82284|  1.242851|  9.923593|
     10.1.1.6|       80051|  1.209123| 11.132715|
     10.1.1.7|       73602|  1.111714| 12.244430|

  $ rwfilter ...| rwstats --sip --dip --byte --threshold=100000000
  INPUT SIZE: 549092 records for 107136 unique keys
  SIP/DIP PAIR Key: Top 5 byte counts (threshold 100000000)
          sIP|            dIP|       Bytes|%_of_total|   cumul_%|
     10.1.1.1|       10.1.1.2|   307478707|  9.016177|  9.016177|
     10.1.1.3|       10.1.1.4|   172164463|  5.048367| 14.064544|
     10.1.1.5|       10.1.1.6|   142059589|  4.165604| 18.230147|
     10.1.1.7|       10.1.1.8|   119388394|  3.500818| 21.730965|
     10.1.1.9|      10.1.1.10|   108268824|  3.174759| 24.905725|

  $ rwfilter ...| rwstats --sport --sport --percentage=5
  INPUT SIZE: 549092 records for 56799 unique keys
  SOURCE PORT Key: Top 3 flow counts (5% == 27454)
        sPort|     Records|%_of_total|   cumul_%|
           80|       86677| 15.785515| 15.785515|
           53|       64681| 11.779629| 27.565144|
            0|       47760|  8.697996| 36.263140|

  $ rwfilter ...| rwstats --dport --bottom --count=8
  INPUT SIZE: 549092 records for 44772 unique keys
  DESTINATION PORT Key: Bottom 8 flow counts
        dPort|     Records|%_of_total|   cumul_%|
        19417|           1|  0.000182|  0.000182|
        12110|           1|  0.000182|  0.000364|
        34777|           1|  0.000182|  0.000546|
         8999|           1|  0.000182|  0.000728|
        36404|           1|  0.000182|  0.000911|
        16682|           1|  0.000182|  0.001093|
        27420|           1|  0.000182|  0.001275|
        14162|           1|  0.000182|  0.001457|

  $ rwfilter ... | \
        rwstats --sport --dport --packet --top --threshold=500000
  INPUT SIZE: 366309 records for 130307 unique keys
  SPORT/DPORT PAIR Key: No packet counts above threshold of 500000

  $ rwfilter ... | \
        rwstats --sport --dport --packet --top --threshold=50000
  INPUT SIZE: 366309 records for 130307 unique keys
  SPORT/DPORT PAIR Key: Top 3 packet counts (threshold 50000)
        sPort|          dPort|      Packets|%_of_total|   cumul_%|
         6699|           3607|       138177|  2.468531|  2.468531|
           80|           1179|        59774|  1.067862|  3.536393|
           80|           9659|        50319|  0.898949|  4.435342|

  $ rwfilter ...| rwstats --protocol --bottom --count=10
  INPUT SIZE: 545262 records for 3 unique keys
  PROTOCOL Key: Bottom 10 flow counts
     protocol|     Records|%_of_total|   cumul_%|
            1|       46319|  8.494815|  8.494815|
           17|      132634| 24.324820| 32.819635|
            6|      366309| 67.180365|100.000000|

  $ rwfilter ...| rwstats --detail-proto-stats=6,17
  FLOW STATISTICS--ALL PROTOCOLS:  549092 records
  *BYTES min 28; max 88906238
    quartiles LQ 122.06478 Med 420.30930 UQ 876.21920 UQ-LQ 754.15442
     interval_max|count<=max|%_of_input|   cumul_%|
               40|     35107|  6.393646|  6.393646|
               60|     35008|  6.375616| 12.769263|
              100|     49500|  9.014883| 21.784145|
              150|     40014|  7.287303| 29.071449|
              256|     65444| 11.918586| 40.990034|
             1000|    224016| 40.797535| 81.787569|
            10000|     75708| 13.787853| 95.575423|
           100000|     21981|  4.003154| 99.578577|
          1000000|      1901|  0.346208| 99.924785|
       4294967295|       413|  0.075215|100.000000|
  *PACKETS min 1; max 70023
    quartiles LQ 1.76962 Med 3.68119 UQ 7.61567 UQ-LQ 5.84605
     interval_max|count<=max|%_of_input|   cumul_%|
                3|    232716| 42.381969| 42.381969|
                4|     61407| 11.183372| 53.565341|
               10|    195310| 35.569631| 89.134972|
               20|     33310|  6.066379| 95.201351|
               50|     17686|  3.220954| 98.422304|
              100|      4854|  0.884005| 99.306309|
              500|      2760|  0.502648| 99.808957|
             1000|       373|  0.067930| 99.876888|
            10000|       637|  0.116010| 99.992897|
       4294967295|        39|  0.007103|100.000000|
  *BYTES/PACKET min 28; max 1500
    quartiles LQ 57.98319 Med 90.71150 UQ 164.77250 UQ-LQ 106.78932
     interval_max|count<=max|%_of_input|   cumul_%|
               40|     42568|  7.752435|  7.752435|
               44|     15173|  2.763289| 10.515724|
               60|     91003| 16.573361| 27.089085|
              100|    163850| 29.840173| 56.929258|
              200|    153190| 27.898786| 84.828043|
              400|     39761|  7.241227| 92.069271|
              600|     12810|  2.332942| 94.402213|
              800|      7954|  1.448573| 95.850786|
             1500|     22783|  4.149214|100.000000|
       4294967295|         0|  0.000000|100.000000|

  FLOW STATISTICS--PROTOCOL 6:  366309/549092 records
  *BYTES min 40; max 88906238
    quartiles LQ 310.47331 Med 656.53661 UQ 1089.75344 UQ-LQ 779.28013
     interval_max|count<=max|%_of_proto|   cumul_%|
               40|     29774|  8.128110|  8.128110|
               60|     11453|  3.126595| 11.254706|
              100|      6915|  1.887751| 13.142456|
              150|     16369|  4.468632| 17.611088|
              256|     12651|  3.453642| 21.064730|
             1000|    196881| 53.747246| 74.811976|
            10000|     68989| 18.833553| 93.645529|
           100000|     21099|  5.759891| 99.405420|
          1000000|      1784|  0.487021| 99.892441|
       4294967295|       394|  0.107559|100.000000|
  *PACKETS min 1; max 70023
    quartiles LQ 3.39682 Med 5.85903 UQ 8.80427 UQ-LQ 5.40745
     interval_max|count<=max|%_of_proto|   cumul_%|
                3|     69358| 18.934288| 18.934288|
                4|     55993| 15.285729| 34.220016|
               10|    186559| 50.929407| 85.149423|
               20|     30947|  8.448332| 93.597755|
               50|     16186|  4.418674| 98.016429|
              100|      4204|  1.147665| 99.164094|
              500|      2178|  0.594580| 99.758674|
             1000|       315|  0.085993| 99.844667|
            10000|       537|  0.146598| 99.991264|
       4294967295|        32|  0.008736|100.000000|
  *BYTES/PACKET min 40; max 1500
    quartiles LQ 60.19817 Med 96.78616 UQ 175.08044 UQ-LQ 114.88228
     interval_max|count<=max|%_of_proto|   cumul_%|
               40|     36559|  9.980372|  9.980372|
               44|     14929|  4.075521| 14.055893|
               60|     39593| 10.808634| 24.864527|
              100|    100117| 27.331297| 52.195824|
              200|    111258| 30.372718| 82.568542|
              400|     26020|  7.103293| 89.671834|
              600|      8600|  2.347745| 92.019579|
              800|      7726|  2.109148| 94.128727|
             1500|     21507|  5.871273|100.000000|
       4294967295|         0|  0.000000|100.000000|

  FLOW STATISTICS--PROTOCOL 17:  132634/549092 records
  *BYTES min 32; max 2115559
    quartiles LQ 66.53665 Med 150.61551 UQ 242.44095 UQ-LQ 175.90430
     interval_max|count<=max|%_of_proto|   cumul_%|
               20|         0|  0.000000|  0.000000|
               40|      5195|  3.916794|  3.916794|
               80|     42150| 31.779182| 35.695975|
              130|     11528|  8.691587| 44.387563|
              256|     45497| 34.302667| 78.690230|
             1000|     23401| 17.643289| 96.333519|
            10000|      4447|  3.352836| 99.686355|
           100000|       389|  0.293288| 99.979643|
          1000000|        23|  0.017341| 99.996984|
       4294967295|         4|  0.003016|100.000000|
  *PACKETS min 1; max 8839
    quartiles LQ 0.84383 Med 1.68768 UQ 2.53149 UQ-LQ 1.68766
     interval_max|count<=max|%_of_proto|   cumul_%|
                3|    117884| 88.879171| 88.879171|
                4|      4452|  3.356605| 92.235777|
               10|      6678|  5.034908| 97.270685|
               20|      1766|  1.331484| 98.602168|
               50|      1055|  0.795422| 99.397590|
              100|       368|  0.277455| 99.675046|
              500|       353|  0.266146| 99.941192|
             1000|        33|  0.024880| 99.966072|
            10000|        45|  0.033928|100.000000|
       4294967295|         0|  0.000000|100.000000|
  *BYTES/PACKET min 32; max 1415
    quartiles LQ 63.23827 Med 91.27180 UQ 158.10219 UQ-LQ 94.86392
     interval_max|count<=max|%_of_proto|   cumul_%|
               20|         0|  0.000000|  0.000000|
               24|         0|  0.000000|  0.000000|
               40|      5671|  4.275676|  4.275676|
              100|     70970| 53.508150| 57.783826|
              200|     39298| 29.628904| 87.412730|
              400|     12175|  9.179396| 96.592126|
              600|      4130|  3.113832| 99.705958|
              800|       160|  0.120633| 99.826590|
             1500|       230|  0.173410|100.000000|
       4294967295|         0|  0.000000|100.000000|

=head1 NOTES

The output of B<rwstats> is similar to that of B<rwaddrcount(1)>,
B<rwtotal(1)>, and B<rwuniq(1)>.

To compute Top-N lists for other key combinations or to see values for
Records, Packets, and Bytes in a single view, consider using another
SiLK tool and passing the output through sort and head.  For example,
to see the Top-10 lists for sip,sport combinations, counting by Bytes:

  $ rwfilter ...| rwuniq --fields=sip,sport --all --no-titles \
        | sort -r -t '|' -k 3 | head -10

=head1 BUGS

B<rwstats> uses an hash table internally when computing Top-N and
Bottom-N lists.  B<rwstats> may run of memory when processing IP
addresses, especially IP-pairs.  If B<rwstats>'s hash table does run
out of memory, B<rwstats> will stop processing input, print a warning
to the standard error, output the entries it has computed to that
point, and exit with code 16.

=head1 SEE ALSO

B<rwfilter(1)> B<rwaddrcount(1)> B<rwtotal(1)> B<rwuniq(1)>

=cut

$SiLK: rwstats.pod 8269 2007-08-03 18:54:48Z mthomas $

Local Variables:
mode:text
indent-tabs-mode:nil
End: