.\" Automatically generated by Pod::Man v1.37, Pod::Parser v1.32 .\" .\" Standard preamble: .\" ======================================================================== .de Sh \" Subsection heading .br .if t .Sp .ne 5 .PP \fB\\$1\fR .PP .. .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. | will give a .\" real vertical bar. \*(C+ will give a nicer C++. Capital omega is used to .\" do unbreakable dashes and therefore won't be available. \*(C` and \*(C' .\" expand to `' in nroff, nothing in troff, for use with C<>. .tr \(*W-|\(bv\*(Tr .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' 'br\} .\" .\" If the F register is turned on, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.Sh), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . nr % 0 . rr F .\} .\" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .hy 0 .if n .na .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "Xango 3" .TH Xango 3 "2008-01-18" "perl v5.8.8" "User Contributed Perl Documentation" .SH "NAME" Xango \- Event Based High Performance Web Crawler Framework .SH "SYNOPSIS" .IX Header "SYNOPSIS" .Vb 1 \& use Xango; .Ve .SH "DESCRIPTION" .IX Header "DESCRIPTION" .Vb 4 \& ****************** WARNING! WARNING! WARNING! **************** \& * This new release is NOT compatible with the old Xango 0.0x! * \& * Beware and use at your own risk. * \& *************************************************************** .Ve .PP Xango is a framework for writing web crawlers. As such, it doesn't do a whole lot by itself \- you need to create custom handlers to do the grunt work. See the documentation for Xango::Manual::Intro for more details on how to write your own crawler. .PP Please note that Xango is still in beta. Some behavior may change as we keep on developing it. .SH "COMPONENTS" .IX Header "COMPONENTS" The main component that comes with Xango is the Broker component. The Broker handles the basic flow of a web crawler \- it accepts data to fetch, applies some policies to it, fetches it, and then sends it for final processing. .PP The only concrete implementation provided by the Broker of the above flow is the part where the requested \s-1URI\s0 is fetched. You must provide the rest of the logic. See Xango::Broker for details .PP The Handler is the component in which the Broker delegates the above processing to. .SH "CONSTANTS" .IX Header "CONSTANTS" There are a few constants that may change the behavior of Xango. You can either set these by pre-declaring the constant subroutine of the same name or you can set them by populating the corresponding environment variable, which is prefixed with '\s-1XANGO_\s0' .PP For example, .PP .Vb 2 \& BEGIN { sub Xango::DEBUG { 1 } } \& use Xango; .Ve .PP or .PP .Vb 2 \& $ENV{XANGO_DEBUG} = 1; \& use Xango; .Ve .PP will both toggle the \s-1DEBUG\s0 switch. .PP See blow for the list of possible flags and switches. .Sh "\s-1DEBUG\s0" .IX Subsection "DEBUG" Set this to 1 if you want to enable debugging messages. .SH "FUNCTIONS" .IX Header "FUNCTIONS" .Sh "debug" .IX Subsection "debug" .Sh "info" .IX Subsection "info" These are simple interface to the underlying logging mechanism .SH "DEBUGGING" .IX Header "DEBUGGING" To turn debugging on, you need to pre-declare some constants (which, in Perl, are subroutines). For example, to turn debugging on, you need to say something like this: .PP .Vb 2 \& sub Xango::DEBUG { 1 } \& use Xango; .Ve .PP Xango will recognize that the \s-1DEBUG\s0 flag is already set, and turn debugging on. .SH "BUGS" .IX Header "BUGS" Plenty, I'm sure. Please report bugs to \s-1RT\s0 http://rt.cpan.org/NoAuth/Bugs.html?Dist=Xango .SH "TODO" .IX Header "TODO" .IP "Documentation." 4 .IX Item "Documentation." Documentation for this distribution is half-baked at best. It needs a lot of work, including a tutorial. .SH "CAVEATS" .IX Header "CAVEATS" This is an alpha version release. Expect a lot of things to be still either in broken or experimental stage. .SH "SEE ALSO" .IX Header "SEE ALSO" \&\s-1POE\s0 Xango::Manual::Intro Xango::Broker::Pull Xango::Broker::Push .SH "AUTHOR" .IX Header "AUTHOR" Copyright 2005 Daisuke Maki . All rights reserved. Development funded by Brazil, Ltd.