;ò )äHEc@sódZdkTdkZdkZdkZdklZdklZl Z dk l Z dk l Z lZlZlZdklZdklZeeeƒd Zd Zd Zd Zd eeefZeZdZdfd„ƒYZd„ZdS(sD Zwiki's mailin external method - posts an incoming email message to a wiki This external method receives a message in RFC2822 format and takes appropriate action in the wiki where it has been called (eg, adding a comment to one of the pages). This is usually invoked by a mail alias like:: thewiki: |curl -n -F 'msg=<-' http://mysite/mywikifolder/mailin See http://zwiki.org/HowToSetUpMailin for more help. For the delivery algorithm, see the MailIn class docs below. TODO: friendly non-subscriber bounce messages keep refactoring size limit html mail ? file attachments ? (s*N(sMessage(s parseaddrs getaddresses(styped_subpart_iterator(s wikiname1s wikiname2s bracketedexprsurlchars(sBLATHER(sISSUE_SEVERITIESis(wiki|mailin)@s(tracker|bugs|issues)@s(spam)@s (%s|%s|%s)iôsMailIncBsËtZdZeZeZeZeZeZeZ dZ dZ eZ d„Z d„Zd„Zd„Zd„Zd„Zd„Zd „Zd „Zd „Zd „Zd „Zd„Zd„Zd„ZRS(s¶ I represent an incoming mail message being posted to a wiki. I parse the incoming rfc2822 message string and expose the parts of interest, and (given a wiki context) figure out how to deliver myself. Here are the delivery rules: - if the message appears to be a zwiki mailout or from an auto-responder or junk, or it does't have a plain text part, DISCARD. - if we have been called in a page context (../SOMEPAGE/mailin), POST message as a comment on that page. - (DISABLED: select virtual host: Intended to allow a single mailin alias to serve many vhosts. If there is a recipient of the form .*MAILINADDREXP@vhost where vhost matches a virtual host monster entry on this server, use that vhost's folder. ) - if there is not at least one zwiki page in the current folder, DISCARD. - check sender's subscription status: unless the folder's mailin_policy property, possibly acquired, is 'open', check that the sender is either subscribed somewhere in the wiki or listed in the mail_accept_nonmembers property, and if not BOUNCE the message. - if the recipient looks like a spam reporting address (.*SPAMADDREXP), UPDATE SPAM BLOCKING RULES and DISCARD (see updateSpamBlocks). - if the recipient looks like a tracker mailin address (.*TRACKERADDREXP), CREATE AN ISSUE page. Otherwise, - identify destination page name: the first [bracketed page name] in the message subject, which may be a fuzzy/partial name; or the folder's default_mailin_page property, possibly acquired; or the first zwiki page in the folder - unless default_mailin_page was blank. - if no destination page could be found (default_mailin_page is blank/false), DISCARD. - if no wiki page by that name exists (fuzzy match allowed), CREATE it - POST message as a comment to that page icCs:td|ƒ||_||_ti|iƒ|_|id|_t i dd|ii ddƒƒ|_ t i dd|i ƒ|_ |ii ddƒ|_|ii d dƒ|_|ii d ƒ|_t|iƒd |_t|iƒd |_|ipt i d d|iƒ|_|ii dƒ|_|iot|iƒd pt|_|iidgƒ}|iidgƒ}|iidgƒ}|iidgƒ}t||||ƒ|_|ii dƒ|_y+t |iddƒi!ƒi"dd ƒ}Wnt$j o d}nX|i%|ƒ|_&dS(s´ Extract the bits of interest from an RFC2822 message string. This perhaps should do the isJunk test up front to avoid unnecessary resource usage. s)mailin.py processing incoming message: %ssDates\nssSubjects.*?\[.*?\] ?(.*)s\1s Message-ids In-reply-tosFromiis@.*$sSenderstosccs resent-tos resent-ccs X-BeenTherestextsplainsdecodeN('sBLATHERsmessagescontextsselfsoriginalsemailsmessage_from_stringsmsgsdatesressubsgetssubjects realSubjects messageids inreplytosFroms parseaddrs FromRealNames FromEmails FromUserNamessendersNones senderEmailsget_allstossccss resent_toss resent_ccss getaddressess recipientss xbeentherestyped_subpart_iteratorsnexts get_payloads plaintextparts StopIterations cleanupBodysbody(sselfscontextsmessagestoss resent_tossccss resent_ccss plaintextpart((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pys__init__os:   '%'  cCsZ|i}tid|ƒodSntid|ƒodSn|i odSndSdS(s Return true if this mail message should be silently ignored. Ideally, this should block mail loops, auto-responders and spam, but allow mailing list messages and mailouts from other zwikis. Currently it flags everything that looks like it came from a bot. Actually I don't think works as a way to filter autoresponders. We should make sure to include mailing headers in our mailouts and then most autoresponders should ignore us. qmail-autoresponder's bot-filtering procedure is reportedly good - see http://untroubled.org/qmail-autoresponder/procedure.txt . TMDA and spamassassin are two good spam filters - see http://software.libertine.org/tmda , http://spamassassin.taint.org . s(?mi)^X-Zwiki-Version:is(?mi)^Subject:.*out of officeiN(sselfsoriginalsmsgtextsressearchsbody(sselfsmsgtext((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pysisJunk¡s  cCs!tit|iƒƒodSdS(sJ Return true if this message appears to be a spam report. iN(sressearchs SPAMADDREXPsselfsrecipientAddress(sself((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pys isSpamReportÊsc Csft|iƒdƒoLt|iƒiƒ}tidt|id|i ƒ}g}x9|D]1}||jo|i |ƒ|i |ƒq\q\W|oÆ|iƒ}x"t|idƒ o |i}q§W|id|ƒd|ti|iƒdƒf}t|ƒt|iƒdtƒ}|iƒ}|o|o#|ig|tddd |ƒq^qbnd S( s Update the wiki's spam-blocking rules based on this message. 1. add any urls in the message body and/or attached message body to the folder's banned_links property, if it exists (can acquire) 2. add the first ip address found in the message body, if any, to the folder's banned_ips property, if it exists (can acquire). This means that any subscriber can forward a spam mailout to the wiki's spam address, and/or submit spammer urls or one spammer's IP address manually, and the banned_links/banned_ips properties will be updated. It's probably best to create these on the root folder; they will not be created automatically. XXX how to prevent frivolous reports ? XXX add support for banned_ips, see http://zwiki.org/BlockList. Better to use apache ? should we append to some filesystem config file instead ? make this a separate external method ? call add_spammer_url() and add_spammer_ip() for each ? s banned_linkss%((?:http|https|ftp|mailto|file):/*%s)s s&mailin.py: added %s to %s/banned_linkss/s mail_adminssubjects(banned_links updated)stoN(shasattrsselfsfolderslists banned_linkssresfindallsurlcharsssubjectsbodys spam_linkss added_linksslsappendsaq_bases aq_parentsmanage_changePropertiessstringsjoinsgetPhysicalPathslogsBLATHERsgetattrsNonesadmins workingPagespages sendMailTo( sselfs spam_linksslogsadminslspages added_linkss banned_linkssfolder((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pysupdateSpamBlocksÐs6    "   cCs,tidd|ƒ}|i|ƒ}|SdS(sR Clean up/remove uninteresting parts of an incoming message body. s(?s)\n+$s\nN(sressubsbodysselfsstripSignature(sselfsbody((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pys cleanupBodyscCsitid|ƒ}|o |iƒ}|ot|ƒtjo"titi|ƒd|ƒ}n|SdS(s‹ Strip a signature after -- . Can't really do this safely; we'll strip only if it's below a certain size. s(?s)\n--\n.*?$sN( sressearchsbodys signaturesgroupslensMAX_SIGNATURE_STRIP_SIZEssubsescape(sselfsbodys signature((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pysstripSignatures "cCs|iidjSdS(Ns ZWiki Page(sselfscontexts meta_type(sself((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pys contextIsPage#scCs|iƒ SdS(N(sselfs contextIsPage(sself((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pyscontextIsFolder&scCs&|iƒo|iiSn|iSdS(s= The wiki folder to which we are delivering. N(sselfs contextIsPagescontexts aq_parent(sself((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pysfolder)s cCs·t|iƒdjo|idSnt|iƒdtƒ}|o1x.|iD]}|d|jo|SqNqNWnx0|iD]%}ti t |dƒo|SqqW|idSdS(s¢ The recipient that was used to deliver here (an email tuple). This may be needed to determine the mailin action. If the message has multiple recipients, decide which one refers to us as follows: - the first recipient matching the folder's mail_from property, - or the first one looking like a typical zwiki mailin alias (.*MAILINADDREXP), - or the first one. iis mail_fromN( slensselfs recipientssgetattrsfoldersNonesfolder_mail_fromsrsressearchs MAILINADDREXP(sselfsfolder_mail_fromsr((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pys recipient0s    cCs|iƒdSdS(sT Just the email address part of the recipient used to deliver here. iN(sselfs recipient(sself((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pysrecipientAddressFscCs™|iƒiddƒ}|or|d}|i|iƒiƒƒ}|idƒ}|it |iƒdt ƒƒ}|p|p |p|Snt SdS(s  Try to get a wiki page object which we can use for further operations. We'll try to ensure that new pages are parented somewhere sensible: the default mailin page if specified, or FrontPage, or the first page in the hierarchy. sspecs ZWiki Pageis FrontPagesdefault_mailin_pageN( sselfsfolders objectValuessallpagess alphafirsts pageWithNames wikiOutlinesfirsts outlinefirsts frontpagesgetattrsNonesdefaultmailinpage(sselfs outlinefirstsallpagess alphafirstsdefaultmailinpages frontpage((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pys workingPageLs !cCs:t|iƒd|iƒo|iƒiƒptƒSdS(sW The name of the wiki's default destination page for mailins, or None. sdefault_mailin_pageN(sgetattrsselfsfolders workingPagespageNamesNone(sself((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pysdefaultMailinPage`scCs|iƒod|_dSn|iƒo%tdƒ|iƒd|_dSn|iƒoG|i|_|_ |ii ƒ|_ d|_ |_ |iƒdSn|iƒ|_ |i otdƒd|_dSn|iƒ|iodSntit|iƒƒod|_ dSn|i oLtit|iƒ}|o,|d |_ titd |i ƒ|_ qvn|i o|iƒ|_ n|i o|i i|i d dƒ}|o&||_|_ |ii ƒ|_ n+|i o d|_ nd |_td ƒdS(sJ Figure out what we should do with this mail-in. Fairly involved calculations trying to figure out what to do in a robust way (see the module docstring for more). Sets self.destpage, self.destpagename, self.newpage, self.trackerissue, or sets a message in self.error if it can't. s Discarding junk mailin. Ns!mailin.py: processing spam reports Processed spam report. is<mailin.py: could not find a working page, discarding messages@ Could not find a wiki page to work from, discarding message. iiÿÿÿÿs\1s allow_partials- Message had no destination page, ignored. s3mailin.py: message had no destination page, ignored(sselfsisJunkserrors isSpamReportsBLATHERsupdateSpamBlockss contextIsPagescontextsdestpages workingpagespageNames destpagenamesnewpages trackerissuescheckMailinAlloweds workingPagesressearchsTRACKERADDREXPsrecipientAddresssfindallsPAGEINSUBJECTEXPssubjectsmatchesssubs bracketedexprsdefaultMailinPagespageWithFuzzyNamespage(sselfsmatchesspage((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pysdecideMailinActionhsV                      cCs®t|iƒdtƒ}t|iƒdgƒ}|djpI|i|jp9|i|jp)|ii |iƒp|ii |iƒodSnd|_ t d|iƒdS(sÏ Check if the mailin determined by decideMailinAction() is permitted. Is the sender allowed to mail in here ? - if open posting is allowed, we'll accept anybody. - otherwise they must be a subscriber somewhere in the wiki - or be listed in the mail_accept_nonmembers property (useful to allow mailing lists, etc) Otherwise, set self.error. Requires self.workingpage to be already set up. s mailin_policysmail_accept_nonmemberssopenNs> Sorry, you must be a subscriber to send mail to this wiki. s+mailin.py: bounced mail from non-subscriber( sgetattrsselfsfoldersNones postingpolicysaccepts FromEmails senderEmails workingpagesallSubscriptionsForserrorsBLATHER(sselfs postingpolicysaccept((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pyscheckMailinAllowed¶s Y (s__name__s __module__s__doc__sNonesmsgsfolders recipientsdestpages destpagenames workingpages trackerissuesnewpageserrors__init__sisJunks isSpamReportsupdateSpamBlockss cleanupBodysstripSignatures contextIsPagescontextIsFoldersrecipientAddresss workingPagesdefaultMailinPagesdecideMailinActionscheckMailinAllowed(((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pysMailIn5s2 . 2 )  7         NcCsCt||ƒ}|iƒ|io |iSn|iid|iƒ|i }d}|i oA|i i|iddddƒ|i i|iƒ|_d}nQ|ioF|i i|dtd|iddƒ}|i i|ƒ|_d}n|iid|id |id|id ||d |id |id |iƒtSdS(s See module docstring. sMAILIN_USERNAMEsstextssendmailis(new) sseveritysREQUESTsusernamessubject_headingsexclude_addresss message_ids in_reply_toN(sMailInsselfsmsgsmsdecideMailinActionserrorsREQUESTssets FromUserNames realSubjectssubjects subjectPrefixsnewpages workingpagescreates destpagenames pageWithNamesdestpages trackerissuescreateNextIssuesDEFAULT_SEVERITYspagenamescommentsbodys xbeentheres messageids inreplytosNone(sselfsmsgs subjectPrefixsmspagenamessubject((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pysmailinós6               (s__doc__stypessstringsresemails email.MessagesMessages email.Utilss parseaddrs getaddressessemail.Iteratorsstyped_subpart_iteratorsProducts.ZWiki.Regexpss wikiname1s wikiname2s bracketedexprsurlcharssProducts.ZWiki.UtilssBLATHERs&Products.ZWiki.plugins.tracker.trackersISSUE_SEVERITIESslensDEFAULT_SEVERITYs WIKIADDREXPsTRACKERADDREXPs SPAMADDREXPs MAILINADDREXPsPAGEINSUBJECTEXPsMAX_SIGNATURE_STRIP_SIZEsMailInsmailin(surlcharssTRACKERADDREXPs bracketedexprsDEFAULT_SEVERITYstyped_subpart_iteratorsPAGEINSUBJECTEXPsresmailinsemailsISSUE_SEVERITIESs getaddressessstringsBLATHERs wikiname2s wikiname1s parseaddrs SPAMADDREXPsMAX_SIGNATURE_STRIP_SIZEs MAILINADDREXPsMailIns WIKIADDREXPsMessage((sA/mnt/gmirror/ports/www/zope-zwiki/work/ZWiki/Extensions/mailin.pys?s&     ÿ¿