.\" Automatically generated by Pod::Man v1.37, Pod::Parser v1.32 .\" .\" Standard preamble: .\" ======================================================================== .de Sh \" Subsection heading .br .if t .Sp .ne 5 .PP \fB\\$1\fR .PP .. .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. | will give a .\" real vertical bar. \*(C+ will give a nicer C++. Capital omega is used to .\" do unbreakable dashes and therefore won't be available. \*(C` and \*(C' .\" expand to `' in nroff, nothing in troff, for use with C<>. .tr \(*W-|\(bv\*(Tr .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' 'br\} .\" .\" If the F register is turned on, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.Sh), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . nr % 0 . rr F .\} .\" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .hy 0 .if n .na .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "Xango::Manual::Intro 3" .TH Xango::Manual::Intro 3 "2008-01-18" "perl v5.8.8" "User Contributed Perl Documentation" .SH "NAME" Xango::Manual::Intro \- Learn How To Write Crawlers With Xango .SH "DECIDE ON BROKER TYPE" .IX Header "DECIDE ON BROKER TYPE" There are two types of Xango::Broker \*(-- the \*(L"pull\*(R" and \*(L"push\*(R" types. The \*(L"pull\*(R" crawler is implemented as Xango::Broker::Pull, and the \*(L"push\*(R" is implemented as Xango::Broker::Push. .PP Xango::Broker::Pull crawler periodically \*(L"pulls\*(R" the jobs that needs to be processed. Xango::Broker::Push crawler waits for jobs to be pushed to the queue. .PP You should decide on the type of crawler to use depending on the application that you want to build. If you have a constant pool of jobs that need to be processed, then usually the pull model is better. However, if your crawler's processing is triggered by an event (for example, a user submits a form that contains the \s-1URI\s0 to crawl) then the push crawler is the way to go. .SH "IMPLEMENTING THE PULL CRAWLER" .IX Header "IMPLEMENTING THE PULL CRAWLER" A \*(L"pull\*(R" crawler goes into a loop that periodically checks for new jobs to process from a source. If you have a crawler that should be crawling a possibly infinite list of jobs, then this is probably the type of crawler that you want to use. .PP .Vb 6 \& MyHandler->spawn(...); \& Xango::Broker::Pull->spawn( \& Alias => 'broker', \& HandlerAlias => 'handler' \& ... \& ); .Ve .PP .Vb 1 \& POE::Kernel->run; .Ve .PP In this scenario, the handler component needs to implement the following states: .IP "apply_policy" 4 .IX Item "apply_policy" apply_policy is called synchronously via POE::Kernel\->\fIcall()\fR method, and accepts the Xango::Job object that is currently being handled. You should use this state in your handler to decide if the job is really suitable for crawling. .Sp For example, if you want to filter out URLs that contain the word 'credit', you can do something like this: .Sp .Vb 5 \& sub apply_policy \& { \& my ($job) = @_[ARG0]; \& return $job->uri !~ /credit/; \& } .Ve .Sp And Xango will stop processing this job. .IP "retrieve_jobs" 4 .IX Item "retrieve_jobs" The retrieve_jobs state is called periodically to fetch new jobs to be processed. This is probably where you would be accessing a database or such. .Sp In this state, just return a list of jobs: .Sp .Vb 6 \& package MyHandler; \& sub retrieve_jobs \& { \& # of course, you probably wouldn't be connecting to a database every \& # time retrieve_jobs is called in reality (too much overhead) \& my $dbh = DBI->connect(...); .Ve .Sp .Vb 8 \& my @jobs; \& my $sth = $dbh->prepare("SELECT ...."); \& while ($sth->fetchrow_arrayref) { \& my $job = Xango::Job->new(uri => $uri, ...); \& push @jobs, $job; \& } \& return @jobs; \& } .Ve .IP "handle_response" 4 .IX Item "handle_response" \&\fIhandle_response()\fR is a state where you should do the actual processing of the data that got fetched by Xango. You can do whatever you want in here, for example parsing the \s-1HTML\s0, saving the fetched content, etc. The job is passed as \s-1ARG0\s0. .Sp .Vb 5 \& sub handle_response \& { \& my($response) = @_[ARG0]; \& my $file = '...'; \& open(my $fh, ">$file") or die $!; .Ve .Sp .Vb 3 \& print $fh $response->as_string; \& close($fh); \& } .Ve .IP "finalize_job" 4 .IX Item "finalize_job" This state is called at the end of the processing chain. You should perform any cleanup that may be required. .SH "IMPLEMENTING THE PUSH CRAWLER" .IX Header "IMPLEMENTING THE PUSH CRAWLER" You should use the push crawlers for crawlers that crawl a \s-1URL\s0 in response to an event. For example if you want to make a \s-1POE\s0 server that accepts lists of \s-1URL\s0 sent in by a user, and only want to crawl the specified \s-1URL\s0 (as opposed to fetching the URLs from a data source yourself), this is the crawler type to go. .PP You define a broker and handler like the pull crawler: .PP .Vb 6 \& MyHandler->spawn(...); \& Xango::Broker::Push->spawn( \& Alias => 'broker', \& HandlerAlias => 'handler' \& ... \& ); .Ve .PP .Vb 1 \& POE::Kernel->run(); .Ve .PP And then you just post to the 'enqueue_job' state from somewhere else: .PP .Vb 2 \& # In your code elsewhere... \& POE::Kernel->post('broker', 'enqueue_job', $job); .Ve .IP "apply_policy" 4 .IX Item "apply_policy" .PD 0 .IP "handle_response" 4 .IX Item "handle_response" .IP "finalize_job" 4 .IX Item "finalize_job" .PD These states are all the same as the pull\-parser. .SH "PERFORMANCE" .IX Header "PERFORMANCE" .Sh "\s-1CUSTOM\s0 \s-1HTTP\s0 \s-1COMPONENT\s0" .IX Subsection "CUSTOM HTTP COMPONENT" For maximum performance, you will most likely need to re-write your \s-1HTTP\s0 fetcher component (by default POE::Component::Client::HTTP is used). This is due to the fact that, most crawler are daemon that perpetually retrieves data, and this requires maximum tuning from the code-writer to reduce the memory impact from downloading arbitrary content. .PP For example, http://xango.razil.jp uses a disk-based variation of POE::Component::Client::HTTP that writes data as soon as it can to minimize the memory usage. .PP For casual users, though, this may not be a problem. .SH "AUTHOR" .IX Header "AUTHOR" Copyright (c) 2005\-2006 Daisuke Maki