;ò 5Ec@s¿dZdkZdkZdkZdkZeadkZdkZei ddfjo da nda dfd„ƒYZ dfd „ƒYZ d e ei fd „ƒYZ d e eifd „ƒYZdeifd„ƒYZdeifd„ƒYZdeifd„ƒYZd„Zd„Zd„Zdd„Zd„Zdd„ZedjoadkZdkZy$eeidƒZeidZWndeidGHq»XeeeƒndS(s An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive. >>> import urllib2 >>> from keepalive import HTTPHandler >>> keepalive_handler = HTTPHandler() >>> opener = urllib2.build_opener(keepalive_handler) >>> urllib2.install_opener(opener) >>> >>> fo = urllib2.urlopen('http://www.python.org') If a connection to a given host is requested, and all of the existing connections are still in use, another connection will be opened. If the handler tries to use an existing connection but it fails in some way, it will be closed and removed from the pool. To remove the handler, simply re-run build_opener with no arguments, and install that opener. You can explicitly close connections by using the close_connection() method of the returned file-like object (described below) or you can use the handler methods: close_connection(host) close_all() open_connections() NOTE: using the close_connection and close_all methods of the handler should be done with care when using multiple threads. * there is nothing that prevents another thread from creating new connections immediately after connections are closed * no checks are done to prevent in-use connections from being closed >>> keepalive_handler.close_all() EXTRA ATTRIBUTES AND METHODS Upon a status of 200, the object returned has a few additional attributes and methods, which should not be used if you want to remain consistent with the normal urllib2-returned objects: close_connection() - close the connection to the host readlines() - you know, readlines() status - the return status (ie 404) reason - english translation of status (ie 'File not found') If you want the best of both worlds, use this inside an AttributeError-catching try: >>> try: status = fo.status >>> except AttributeError: status = None Unfortunately, these are ONLY there if status == 200, so it's not easy to distinguish between non-200 responses. The reason is that urllib2 tries to do clever things with error codes 301, 302, 401, and 407, and it wraps the object upon return. For python versions earlier than 2.4, you can avoid this fancy error handling by setting the module-level global HANDLE_ERRORS to zero. You see, prior to 2.4, it's the HTTP Handler's job to determine what to handle specially, and what to just pass up. HANDLE_ERRORS == 0 means "pass everything up". In python 2.4, however, this job no longer belongs to the HTTP Handler and is now done by a NEW handler, HTTPErrorProcessor. Here's the bottom line: python version < 2.4 HANDLE_ERRORS == 1 (default) pass up 200, treat the rest as errors HANDLE_ERRORS == 0 pass everything up, error processing is left to the calling code python version >= 2.4 HANDLE_ERRORS == 1 pass up 200, treat the rest as errors HANDLE_ERRORS == 0 (default) pass everything up, let the other handlers (specifically, HTTPErrorProcessor) decide what to do In practice, setting the variable either way makes little difference in python 2.4, so for the most consistent behavior across versions, you probably just want to use the defaults, which will give you exceptions on errors. NiiiisConnectionManagercBsGtZdZd„Zd„Zd„Zd„Zd„Zed„Z RS(sV The connection manager must be able to: * keep track of all existing cCs.tiƒ|_h|_h|_h|_dS(N(sthreads allocate_locksselfs_locks_hostmaps_connmaps _readymap(sself((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys__init__zs  cCsy|iiƒzW|ii|ƒ og|i| host is the host:port spec, as in 'www.cnn.com:8080' as passed in. no error occurs if there is no connection to that host.N(sselfs_cmsget_allshostshsremovesclose(sselfshostsh((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysclose_connection»s cCsUxN|iiƒiƒD]7\}}x(|D] }|ii|ƒ|iƒq)WqWdS(sclose all open connectionsN( sselfs_cmsget_allsitemsshostsconnsshsremovesclose(sselfsconnsshshost((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys close_allÃs cCs|ii|dƒdS(sdtells us that this request is now closed and the the connection is ready for another requestiN(sselfs_cms set_readys connection(sselfsrequestshosts connection((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys_request_closedÊsicCs)|o|iƒn|ii|ƒdS(N(scloses connectionsselfs_cmsremove(sselfshosts connectionsclose((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys_remove_connectionÏscCs |iƒ}| otidƒ‚ny×|ii|ƒ}x¾|oQ|i |||ƒ}|oPn|i ƒ|ii |ƒ|ii|ƒ}q?W|i |ƒ}totid|t|ƒƒn|ii||dƒ|i||ƒ|iƒ}Wn0titifj o}ti|ƒ‚nX|io|ii |ƒntotid|i|iƒn||_||_|iƒ|_||_ |i|_!|i"|_#|i|_"|idjpt$ o|Sn)|i%id|||i|i"|i#ƒSdS(Ns no host givens"creating new connection to %s (%d)isSTATUS: %s, %siÈshttp(&sreqsget_hostshostsurllib2sURLErrorsselfs_cmsget_ready_connshs_reuse_connectionsrsclosesremoves_get_connectionsDEBUGsinfosidsadds_start_transactions getresponsessocketserrorshttplibs HTTPExceptionserrs will_closesstatussreasons_handlers_hosts get_full_urls_urls _connectionscodesmsgsheaderss HANDLE_ERRORSsparent(sselfsreqserrshshostsr((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysdo_openÔsJ          cCs y |i||ƒ|iƒ}Wnptiti fj o t }nMt o!t idd|t |ƒƒn|ii|ƒ|iƒ‚nX|t jp |idjo.t ot id|t |ƒƒnt }n%t ot id|t |ƒƒn|SdS(sGstart the transaction with a re-used connection return a response object (r) upon success or None on failure. This DOES not close or remove bad connections in cases where it returns. However, if an unexpected exception occurs, it will close and remove the connection before re-raising. sunexpected exception - closing sconnection to %s (%d)i s&failed to re-use connection to %s (%d)sre-using connection to %s (%d)N(sselfs_start_transactionshsreqs getresponsesrssocketserrorshttplibs HTTPExceptionsNonesDEBUGshostsids_cmsremovesclosesversionsinfo(sselfshsreqshostsr((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys_reuse_connections*      cCsZy§|iƒo€|iƒ}|id|iƒƒ|iidƒ o|i ddƒn|iidƒ o|i ddt |ƒƒq¦n|id|iƒƒWn0t i t ifj o}ti|ƒ‚nXx!|iiD]}|i |ŒqçWx-|iiƒD]\}}|i ||ƒqW|iƒ|iƒo|i|ƒndS(NsPOSTs Content-types!application/x-www-form-urlencodedsContent-lengths%dsGET(sreqshas_datasget_datasdatashs putrequests get_selectorsheadersshas_keys putheaderslenssocketserrorshttplibs HTTPExceptionserrsurllib2sURLErrorsselfsparents addheaderssargssitemssksvs endheadersssend(sselfshsreqsargsserrsvsdatask((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys_start_transaction/s*    "    cCstSdS(N(sNotImplementedError(sselfshost((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys_get_connectionFs( s__name__s __module__s__init__sopen_connectionssclose_connections close_alls_request_closeds_remove_connectionsdo_opens_reuse_connections_start_transactions_get_connection(((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysKeepAliveHandler±s       0 + s HTTPHandlercBs#tZd„Zd„Zd„ZRS(NcCsti|ƒdS(N(sKeepAliveHandlers__init__sself(sself((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys__init__JscCs|i|ƒSdS(N(sselfsdo_opensreq(sselfsreq((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys http_openMscCst|ƒSdS(N(sHTTPConnectionshost(sselfshost((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys_get_connectionPs(s__name__s __module__s__init__s http_opens_get_connection(((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys HTTPHandlerIs  s HTTPSHandlercBs&tZed„Zd„Zd„ZRS(NcCs2ti|ƒ| otiƒ}n||_dS(N(sKeepAliveHandlers__init__sselfs ssl_factorys sslfactorys get_factorys _ssl_factory(sselfs ssl_factory((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys__init__Ts cCs|i|ƒSdS(N(sselfsdo_opensreq(sselfsreq((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys https_openZscCs|ii|ƒSdS(N(sselfs _ssl_factorysget_https_connectionshost(sselfshost((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys_get_connection]s(s__name__s __module__sNones__init__s https_opens_get_connection(((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys HTTPSHandlerSs  s HTTPResponsecBsntZdded„ZeiiZd„Zd„Z d„Z d„Z ed„Zdd „Z dd „Z RS( NicCs‰|otii||||ƒntii|||ƒ|i|_t|_ d|_ d|_ t|_ t|_ t|_t|_dS(Nsi (smethodshttplibs HTTPResponses__init__sselfssocks debuglevelsfilenosNonescodes_rbufs _rbufsizes_handlers_hosts_urls _connection(sselfssocks debuglevelsstrictsmethod((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys__init__ts       cCsR|ioD|iiƒt|_|io |ii||i|iƒqNndS(N(sselfsfpsclosesNones_handlers_request_closeds_hosts _connection(sself((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysclose„s     cCs-|ii|i|iddƒ|iƒdS(Nsclosei(sselfs_handlers_remove_connections_hosts _connectionsclose(sself((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysclose_connectionŒscCs |iSdS(N(sselfsheaders(sself((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysinfoscCs |iSdS(N(sselfs_url(sself((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysgeturl“scCsŽ|io |tj oOt|iƒ}||jo||8}qg|i| }|i||_|Sn|i|i|ƒ}d|_|SdS(Ns(sselfs_rbufsamtsNoneslensLsss _raw_read(sselfsamtsLss((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysread–s   iÿÿÿÿcCs=d}|iidƒ}xœ|djo%d|jot|iƒjn of|i|iƒ}| oPn|idƒ}|djo|t|iƒ}n|i||_qW|djot|iƒ}n |d}d|jot|iƒjno |}n|i| |i|f\}|_|SdS(Nss ii( sdatasselfs_rbufsfindsislimitslens _raw_reads _rbufsizesnew(sselfslimitsisnewsdata((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysreadline¦s$5   ' #cCsrd}g}x[noS|iƒ}| oPn|i|ƒ|t|ƒ7}|o ||joPqqW|SdS(Nii(stotalslistsselfsreadlineslinesappendslenssizehint(sselfssizehintslistslinestotal((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys readlinesµs   (s__name__s __module__sNones__init__shttplibs HTTPResponsesreads _raw_readsclosesclose_connectionsinfosgeturlsreadlines readlines(((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys HTTPResponse`s       sHTTPConnectioncBstZeZRS(N(s__name__s __module__s HTTPResponsesresponse_class(((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysHTTPConnectionÂssHTTPSConnectioncBstZeZRS(N(s__name__s __module__s HTTPResponsesresponse_class(((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysHTTPSConnectionÆsc Cs<t} tƒ} ti| ƒ}ti|ƒhdd<dd<}xÍddfD]¿}d|||fGH|aymti |ƒ} | i ƒ}| iƒy| i| if\}}Wn%tj ottf\}}nXWn!tj o}d|GH‚qPXd||fGHqPW| a| iƒ}dG|GH| iƒdS( Nisoffisons. fancy error handling %s (HANDLE_ERRORS = %i)s EXCEPTION: %ss status = %s, reason = %ssopen connections:(s HANDLE_ERRORSsorigs HTTPHandlerskeepalive_handlersurllib2s build_openersopenersinstall_openerspossisurlopensurlsfosreadsfoosclosesstatussreasonsAttributeErrorsNonesIOErrorsesopen_connectionsshostss close_all( surlsstatussfoosesisreasonsopenersposshostssfoskeepalive_handlersorig((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys error_handlerÍs4        cCsQdk}d}tiƒ}ti|ƒti|ƒ}|i ƒ}|i ƒ|i |ƒ}|d|iƒfGHtitƒƒ}ti|ƒti|ƒ}|i ƒ}|i ƒ|i |ƒ}|d|iƒfGHti|ƒ}d}x.no&|iƒ}|o||}qñPqøW|i ƒ|i |ƒ}|d|iƒfGHdS(Ns%25s: %ss normal urllibskeepalive readsiskeepalive readline(smd5sformatsurllib2s build_openersopenersinstall_openersurlopensurlsfosreadsfoosclosesnewsms hexdigests HTTPHandlersreadlinesf(surlsfoosformatsfsmsopenersmd5sfo((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys continuityçs6          cCs«d||fGHtiidƒtiƒ}ti|ƒt ||ƒ}d|GHtiidƒtit ƒƒ}ti|ƒt ||ƒ}d|GHd||fGHdS(Ns making %i connections to: %ss( first using the normal urllib handlerss TIME: %.3f ss( now using the keepalive handler s improvement factor: %.2f( sNsurlssyssstdoutswritesurllib2s build_openersopenersinstall_openersfetchst1s HTTPHandlerst2(sNsurlst2st1sopener((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pyscomps     c Csñdk}g}|iƒ}xqt|ƒD]c}|o |djo|i|ƒnti |ƒ} | i ƒ} | iƒ|it| ƒƒq(W|iƒ|}d}xA|dD]5}|d}||dj od||fGHq°q°W|SdS(Niis+WARNING: inconsistent length on read %i: %i(stimeslenss starttimesrangesNsisdelayssleepsurllib2surlopensurlsfosreadsfoosclosesappendslensdiffsj( sNsurlsdelayslenssjsis starttimestimesdiffsfoosfo((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysfetchs&       cCst}dfd„ƒY}|ƒadGHti|ƒ}|iƒ}|i ƒd}d|GHxJ|djo<t i i d|ƒt i iƒtidƒ|d8}q^Wt ii d ƒd GHti|ƒ}|iƒ}|i ƒ||jo d GHnd GH|adS( Ns FakeLoggercBstZd„ZeZZZRS(NcGs ||GHdS(N(smsgsargs(sselfsmsgsargs((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pysdebug2s(s__name__s __module__sdebugsinfoswarningserror(((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys FakeLogger1s s- fetching the file to establish a connectionis; waiting %i seconds for the server to close the connectionis %2iis s! fetching the file a second times data are identicals ERROR: DATA DIFFER(sDEBUGsdbbackups FakeLoggersurllib2surlopensurlsfosreadsdata1sclosesissyssstdoutswritesflushstimessleepsstderrsdata2(surlsisdata1s FakeLoggersdata2sdbbackupsfo((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys test_timeout.s2           i cCsqdGHyt|ƒWn$tj o}dGHtiƒnXHdGHt|ƒHdGHt||ƒHdGHt |ƒdS(Ns,checking error hander (do this on a non-200)s.exiting - exception will prevent further testss>performing continuity test (making sure stuff isn't corrupted)sperforming speed comparisons#performing dropped-connection check( s error_handlersurlsIOErrorsessyssexits continuityscompsNs test_timeout(surlsNse((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pystestPs  s__main__s%s (s__doc__surllib2shttplibssocketsthreadsNonesDEBUGs sslfactoryssyss version_infos HANDLE_ERRORSsConnectionManagersKeepAliveHandlers HTTPHandlers HTTPSHandlers HTTPResponsesHTTPConnectionsHTTPSConnections error_handlers continuityscompsfetchs test_timeoutstests__name__stimesintsargvsNsurl(s test_timeoutsConnectionManagers error_handlers HTTPHandlerscompstests HTTPResponses continuitys HTTPSHandlersurllib2ssyssHTTPConnectionsNsKeepAliveHandlershttplibssocketsthreadsurls sslfactorystimesHTTPSConnectionsfetch((sR/mnt/gmirror/ports/www/py-urlgrabber/work/urlgrabber-3.1.0/urlgrabber/keepalive.pys?ds@       <˜  b  !   "