IMSP Implementation Updates Joseph Jackson 22-Feb-2000 These notes supplement the file "imsp.implementation" with updates and corrections that bring it up to date with the latest changes to the server. There are several sections found below. - LDAP-based address books - IMAP integration removed - SASL authentication added - Implementation status - Organization of code, description of modules LDAP-BASED ADDRESS BOOKS ------------------------ The IMSP server now supports a special type of read-only address book that is implemented using LDAP lookups rather than the built-in "syncdb" database. A section has been added to the "Setup-instructions" document to explain how to use it. The "Changes-Todo" file also has some comments on how this feature might be improved in the future. The LDAP lookup code was written against the old Netscape LDAPv2 library. If you need any changes in order to make it work with your preferred LDAP library, I'd appreciate getting those patches back from you. Why was this added? For several reasons. At Carnegie Mellon, we find it useful to have an address book available that lists all users on the system. This is especially helpful in the context of the Mulberry 2.0 mail client, where it is very easy to initiate IMSP address book lookups but not as convenient to perform LDAP lookups. Before this feature was added, the "All Users" address book was generated from the password file every morning, getting munged into the IMSP database format. It seemed silly to regenerate a database that's already available elsewhere. Plus, the source data violates the IMSP spec in that there are several instances where more than one user shares the same full name. Since IMSP address books are keyed on full name, they are supposed to be unique. This caused all sorts of bugs and bad behavior. The mondo address book also caused performance problems. The IMSP server can only keep one address book in memory at a time, but people are likely to want to use a personal address book at the same time as the "All Users" address book. There are noticable delays when the server re-reads the large address book database file, making automatic name lookups painfully slow. By eliminating the really huge database file, the server feels a lot more responsive to users. Changing between multiple personal address books is okay because they tend to be relatively small. In fact, the LDAP lookups are really quick. It seems like the overhead of sending a request to the LDAP server is small enough that it's indexed lookups still return faster than a local syncdb lookup. I mentioned earlier a problem where multiple users have the same full name (ex: John Anderson). The LDAP code has a somewhat heavy handed approach to solving that problem, so it's worth describing. IMSP address books are keyed on the full name. When you perform a search on some address book field, the results are returned as a list of full names (database keys). To get the complete details on the entries associated with those keys, a fetch command is sent with one or more full names. If your search returns "John Anderson" as a match and you later fetch it, how will the server know which one you want? The IMSP server normally won't let you create duplicate keys, but a manually generated database file might violate that rule. If so, it's unpredicatable which entry will be returned. To solve this problem in the case of LDAP address books, some extra LDAP searches are performed during the SEARCHADDRESS command. Before the server returns the full name of a search result, an extra search is done to see if the full name is unique on the LDAP server. If it isn't, the full name is augmented with the name and value of some other LDAP attribute which is known to be unique. The "uniquifing" information is enclosed by square brackets and tacked onto the end of the full name. If the extra search finds that only one entry has the full name, it is returned directly with nothing extra added on. When a FETCHADDRESS command is processed for an LDAP address book, the server looks for the special square bracket delimiters. If found, the value of the unique attribute is used to find the entry instead of the full name. We find that users of Mulberry, our preferred e-mail client, don't notice the extra details appended to the name. Mulberry never displays the full names as returned by the initial SEARCHADDRESS command. It returns the full name from the individual FETCHADDRESS commands instead, and those are not modified by the server. Applications that don't hide this information will result in the users seeing something a little odd, but still readable. It might look like this, assuming you use the "uid" (user-id) attribute as the uniquifier: John Anderson[uid:anderson2] The "Setup-instructions" file explains how to specify the name of the LDAP attribute which is known to be unique at your site. See that file for more details. IMAP INTEGRATION REMOVED ------------------------ In the original design of the IMSP server, many mailbox-related commands were provided. The intention was to make the IMSP server the centralized source of mailbox details such as the name of the IMAP server housing the folder. Those features were never completely implemented and never used by any client that we are aware of. (However, the concept continues on in the ACAP protocol's Mailboxes dataset.) To simplify the process of compiling the IMSP server, the IMAP integration features have been removed. See the "Implementation status" section for an exact list of protocol commands that have been removed. The primary side-effect of this change is that the IMSP server no longer obtains Kerberos credentials to be used while connecting to the IMAP server. Sites using Kerberos will no longer need to take precautions to ensure that the credentials of the invoker are not overwritten. Read on for details on how the rest of the authentication infrastructure has been affected by the integration of SASL. SASL AUTHENTICATION ADDED ------------------------- The LOGIN and AUTHENTICATE commands are now implemented using the Cyrus Simple Authentication and Security Layer (SASL) library. See the IETF RFC 2222 for more background on SASL. The "Build-instructions" file has more information on obtaining the required SASL library. The benefit of this work is that there are very few dependencies on the particular authentication system (such as Kerberos) used at your site. Once you compile SASL and install a plug-in for each mechanism you want to support, the IMSP server should just work. There are other software packages that also rely on the Cyrus SASL libraries, so the tricky details of compiling authentication modules can be done just once for all those packages. IMPLEMENTATION STATUS --------------------- NOOP Done LOGIN/LOGOUT Done, with SASL sasl_checkpass function AUTHENTICATE Done, with SASL integration CAPABILITY Done GET/SET/UNSET Done CREATEADDRESSBOOK Done DELETEADDRESSBOOK Done RENAMEADDRESSBOOK Done SEARCHADDRESS Done FETCHADDRESS Done STOREADDRESS Done DELETEADDRESS Done LOCK/UNLOCK Done SETACL/DELETEACL Done GETACL/MYRIGHTS Done SUBSCRIBE/UNSUBSCRIBE Returns "not implemented" as of 1.6a1 CREATE Returns "not implemented" DELETE Returns "not implemented" RENAME Returns "not implemented" MOVE Returns "not implemented" LIST Returns "not implemented" This meets minimal protocol compliance standards outlined in the IMSP specification. ORGANIZATION OF THE CODE ------------------------ The lib/* sources are copied from the Cyrus IMAP mail server. We decided to copy the sources over rather than compiling against the "libcyrus" associated with the IMAP server. The rest of the sources are in the "imsp" directory. imsp_server.c -- parses IMSP protocol received from the client and responds appropriately. main.c -- main program which listens on a socket waiting for clients to connect. authize.c -- authorization module abook.c -- address book support abook_ldap.c -- read-only address books implemented via LDAP lookups option.c -- support for options database alock.c -- support for advisory locking database syncdb.c -- low level key-value database functions im_util.c -- IMSP/IMAP protocol utility functions dispatch.c -- File writing/reading with provisions for handling of multiple streams simultaneously. Also includes SASL-negotiated protection mechanism. adate.c -- RFC 822 date generator sasl_support.c - Support routines needed by the SASL library These files were removed during the SASL integration: login_krb.c -- kerberos authentication routines login_unix.c -- unix authentication routines These files were removed while pulling the IMAP integration features: bb.c -- support for global mailboxes imap_client.c -- IMSP's IMAP client routines for proxy connections proxy_krb.c -- kerberos proxy login proxy_unix.c -- unix proxy login