Dave Zarzycki's incomplete brain dump on system startup, mobility and merging similar daemons in to "the anti-kernel"

Mobility and performance are hot topic at Apple. Both mean many things to many people. Both ideas can be related at some level. I believe the important undercurrent of ideas is as follows:

The master tracking bug I filed for these issues is 3190635.


So as the owner of SystemStarter and as a natural consequence, responsibility for the care and feeding of our boot up proceedure, what do I see the problems and solutions as?

First and foremost, we must switch from a boot-up model where loginwindow launches "as soon as everybody should be able to login" to where loginwindow launches "as soon as anybody can login." Ideally, this is as soon as the boot volume becomes mounted read/write. This requires changes between loginwindow and it's dependencies.

  1. loginwindow needs to register for a call back for when "directory" infrastructure changes, comes or goes. This will partially decouple loginwindow from needing "the network" to be up before launching. See 3101359.
  2. The other half will involve cooperation from the file system team and loginwindow. Before logging in a user, loginwindow will need to verify that the user's home directory is mounted/mountable, and if not, let the user know that they cannot finish logging in until a disk is done being fscked or an NFS server/export comes online.

So the user-land boot-up model that we should be moving towards looks something like this:

  1. Run the anti-kernel
  2. get the disk fscked and mounted read/write
  3. Register all services (lazy or not) with the anti-kernel
  4. Launch the WindowServer and loginwindow
  5. Launch non servicy startup-items and other "at boot" items (if any) in the background


So what is this anti-kernel? At the end of this web page is a huge table of nearly all of the daemons on our system. I tried to summarize their mobility/lazy launch situation. What you'll see if you look closely at many of the daemons is that they support or could easily support launching lazily/on-demand. Each does it different daemons due to the different mechanisms that they use.

What has become really obvious to me is that init, mach_init, xinetd, cron, atd, configd, kextd, SystemStarter, watchdog and friends are very similar. They are top level daemons that take specific kinds of events and then schedule code or commands to run based on limted events/rules. I think a unifed tool/daemon is begging to be born. The anti-kernel I envision will be expected to source just about every event possible coming out of the kernel and provide a language for scheduling/triggering commands/jobs to run when a condition is satisfied. So for example, cron functionality could be used in combination with changing my configuration with configd style functionality. Or tools that listen on multiple event sources like mDNSResponder (mach and TCP/IP) can finally be launched lazily from what daemon. If this daemon is well written it could be used by users to lazily launch programs like the ssh-agent, SecurityAgent or iChatAgent.app. It could also schedule iCal to launch at certain times while the user is logged in. There are plenty of other examples where this could be useful to the user in addition to the system itself.

At the very least, I think Apple should experiment with what this kind of daemon might allow for us to do.


Interesting mobility/laziness support bugs are as follows:

Interesting "be lazy" bugs are as follows:

The anti-kernel per daemon investigation:

(We should start tracking mobility bugs against specific daemons and tools too.)

System Wide Services

Daemon:Short summary:The big picture summary:
ATSServerThe font server (xfs for Unix heads)This could be launched lazily via mach_init.
DirectoryServicelookupd rewrittenThis could be launched lazily via mach_init.
SecurityServerTrusted half of the security frameworkThis could be launched lazily via mach_init, but currently today, they manage /dev/random entropy, which is a layering violation.
AppleFileServerAFP serverThis could be launched lazily via xinetd, some changes to xinetd would be needed, but nothing major.
SharedIPkext to share an IP with ClassicThis is loaded at boot today. The kernel should request this get loaded on demand when somebody (Classic) requests an PF_NDRV socket.
IPFirewallipfw support codeIf we're always going to load it, why not compile it in? (it was more work to take it out of the kernel...)
OpenAFSAFS ServersThis needs more investigation, but in the short to mid term, this is probably best left as a startup item. There is little to be gained by making this lauch lazily.
OpenLDAPLDAP serversThis could run out of xinetd.
SystemStarterruns startup itemsSystemStarter does too much today. Once loginwindow gets mobility aware, SystemStarter can get out of the GUI business. Once more services lauch themselves lazily via various mechanisms like mach_init or xinetd, SystemStarter will slowly get off the critical path for boot-up. This kind of functionality should be merged in to the anti-kernel.
amdam-utils automounterThis needs more investigation, but in the short to mid term, this is probably best left as a startup item. There is little to be gained by making this lauch lazily.
atdbatch job schedulerThis should be merged with cron, init, mach_init and friends to form the super-job schedular.
cronperiodic job schedulerThis should be merged with atd, init, mach_init and friends to form the super-job schedular.
initreaps processes and launches ttysThis should be merged with cron, atd, mach_init and friends to form the super-job schedular.
mach_initmananges the Mach bootstrap namespaceIn addition to managing the Mach bootstrap name space, it launches mach based services lazily like [x]inetd. This should be merged with cron, atd, init and friends to form the super-job schedular.
autodiskmountFinds, fscks and mounts disksThis could be launched via mach_init (not lazily until we have a device based trigger mechanism that should evolve in the anti-kernel).
automountApple's NFS automounterThis needs more investigation, but in the short to mid term, this is probably best left as a startup item. There is little to be gained by making this lauch lazily.
configdAll things configHistorically called ipconfigd, these days it manages other configuration details than just networking. This is arguably one of the more modern daemons in the system, and the closest thing we have to the "anti-kernel" in the system. But I believe it is fair to say that configd wasn't designed with becoming the anti-kernel goal in mind.
bluedBluetoothThe bluetooth daemon wants is just the start of various to come that want to be launched on demand when hardware shows up. Today, blued launches on demand, but never exits after the hardware goes away. Anecdotal evidence suggests that the daemon is automatically launched after a reboot if a bluetooth device was ever attached.
bootparamdboot paramater serverThis should run out of xinetd.
bootpdbootpdThis should run out of xinetd.
coreservicesdRemote AppleEvent serverLaunches via xinetd in Panther.
coreservicesdShared Carbon app stateThis will launch lazily via mach_init in Panther.
cupsdThe CUPS Printing daemonThis could run out of xinetd, but some extra work to cupsd needs to run, and some extra support needs to be added to xinetd so that individual printers can advertised via Rendezvous rather than just the CUPS daemon. This problem is much like the Apache (httpd) problem.
fix_prebindingPrebinding fix-upsListens to dyld requests for prebinding fix-ups. This will/should be switching to launching lazily via mach_init in Panther.
distnotedCarbon/AppKit notification serverThis will launch lazily via mach_init in Panther.
dhcpdDHCP serverLaunches lazily via xinetd today.
fingerdfinger serverLaunches lazily via xinetd today.
ftpdFTP serverLaunches lazily via xinetd today.
httpdHTTP serverThe Apache web server. This is probably the complicated daemon to make launch via [x]inetd. It is complicated by the fact that many of the Apache developers do not want to support running out of an inetd. This problem is much like the cupsd problem.
dynamic_pagermanages swap filesthis could be merged in to the anti-kernel, but after talking with some of the Mach kernel engineers, it would add some restrictions to the anti-kernel design.
crashreporterdlistens for mach exceptions and logs crashesThis could be merged with the anti-kernel. Processing mach exceptions can be thought of like init(8) reaping processes via the wait family of system calls.
gettylogin tty helperlaunched via init(8) today, it isn't worth being launched lazily.
identdident serverLaunches lazily via xinetd today.
imapdIMAP serverLaunches lazily via xinetd today.
kadmindKerberos admin serverThis could launch lazily via xinetd.
kdcdKerberos Domain Controller server
kpropdKerberos ??? serverThis could probably launch lazily via xinetd.
kextdKernel extension daemonThis should be launched via mach_init, but since it cares about device event, it cannot be launched lazily at this time.
lookupdPre DirectoryService lookup cache serverThis should launch lazily via mach_init.
lsregister -loadLaunchServices db helperThis should ideally be done lazily at login time and with the help of modern software package management, at install time.
irpd (bind)lookupd by the people who brought you DNSNot much to say here, we don't have any interest in this at the moment.
lwresd (bind)lookupd for DNS by the people you brought you DNSThis could be handled by a xinetd/mach_init equivlent for AF_UNIX domain sockets.
mDNSResponderRendezvous supportThis could be launched lazily vi the anti-kernel, but it would require work, and given that it would need to run almost all the time, there would be little value in lazily launching it.
namedDNS serverThis could be launched lazily via xinetd.
hwmondhardware monitoring daemonMore investigation is needed, but one would hope the anti-kernel's trigger hooks would be flexable enough to launch this on demand.
hlfsdam-utils home-link file system daemonThis needs more investigation, but in the short to mid term, this is probably best left as a startup item. We don't know of anybody who uses this daemon.
loginwindowGUI login daemonloginwindow in theory is fine. It launches on demand from init(8). The only problem is that random groups at Apple have used loginwindow as a home for random hunks of code that have nothing to do with logging users in.
natdBSD NAT daemon"...it comes into play when a user turns on the Internet Sharing feature in the Sharing pref pane. Clicking "Start" launches a process called InternetSharing (built by the InternetSharingSupport project in cvs) and this is the policy program that starts natd, enables ip forwarding, starts the dhcp server, etc etc. The InternetSharing process actually continues to run as long as Internet Sharing is enabled so that it may listen for config change notifications from configd and take action if necessary." -- Matt Scopp
netinfod -s localThe local NetInfo domainThis could be launched on demand via xinetd.
mountd, nfsd, nfsiod, rpc.lockd, rpc.statdNFS serversThese are conditionally launched today if we export NFS volumes, but it would be nice if the NFS startup-item code was unified with some existing or new lazy launch mechanism.
portmapSun RPC name to port mapperThis, like mDNSResponder maps names to addresses/ports. xinetd can also map names to ports via the historic tcpmux port. It would be nice to unify these.
nibinddThe portmap for netinfodNetInfo is going away. I'm not sure how much we care about this in the long run.
nmbd (samba)Rendezvous/DynDNS for NetBIOSThis, like mDNSResponder maps names to addresses/ports. This is owned by the samba world. Not likely that we can make it launch lazily or be merged with some other daemon.
smbd (samba)SMB file serverThis should switch to launching lazily via xinetd.
ntpdNTP client/serverThis needs to be running all of the time to detect and fix clock drift in the machine that you're on. If we just care about fixing the time and we don't care about fixing drift, we can run the ntpdate command out of cron.
panicdumpRecords a panic dump to diskThis is not really a daemon, but something launched at boot. This information wants to be extracted as soon as possible and logged, but I'm not sure where a good home for it is.
pcscdPC/SC Smartcard daemonThis could be launched lazily, but it needs an AF_UNIX equivlent for xinetd or mach_init.
pop3dPOP3 serverThis launches lazily via xinetd today.
pppdPPP client/serverThis launches lazily today via a variety of mechanisms.
routedRIP routing daemonThis should probably be launched lazily via configd given the right networking configuration.
rpc.yppasswdd, ypbind, ypservNIS daemonsI'm not sure what to do about these and I'm not sure how much we care. NIS is dead. Long live LDAP. The servers can be launched out of xinetd. We'd need to figure out a way to trigger the client daemon to launch lazily.
rsyncdrsync daemonThis can lauch lazily via xinetd today.
rtadvdRouter advertisement daemonThis could be launched lazily via xinetd.
rtsoldRouter solicitation daemonThis should probably be a configd plug-in or launched lazily via configd.
saslauthdSASL daemon ("password server")This could be launched lazily, but it needs an AF_UNIX equivlent for xinetd or mach_init.
sendmailApple's MTA for todayLaunching this lazily will require some work due to the mail queue on disk might need flushing at boot.
syslogdsyslog serverLaunching this lazily will require some small amount of work do the the kernel's syslog interface.
sshdssh serverLaunched lazily via xinetd today.
talktalk serverLaunched lazily via xinetd today.
tftpdtftp serverLaunched lazily via xinetd today.
timlegacy cleartext password serverThis could be launched lazily via xinetd.
timedtime server daemonalready runs via xinetd today
updatewhile (1) {sync(); sleep(30);}This should just be absorbed in to the anti-kernel.
vsdbutilManages db of known disksConditionally run at startup. It would be nice to find a better home for this.
watchdogMac OS X Server SysV style init(8)This functionality should be merged with the anti-kernel.
xfsX11 font serverThis could be launched lazily via the anti-kernel (needs both TCP and AF_UNIX domain for lazily launching).
xinetdInternet super serverThis is our laziness enabling daemon for network servers.
servermgrdMac OS X Server Manager daemonThis could probably be launched lazily. More investigation is needed.
serversettingsdMac OS X Server ??? daemonThis could probably be launched lazily. More investigation is needed.
sambadmindMac OS X Server Samba admin daemonThis could probably be launched lazily. More investigation is needed.

Per User Services

CCacheServerThe Kerberos Credentials Cache Server.This could be launched lazily via mach_init (since the Kerberos team decided to use mach for the implementation), but it is currently launched lazily today when somebody first calls in to the Kerberos library
Dockhalf-daemon/half-appIt is worth considering launching it via a standardized FBE management daemon.
InternetReminder.appfrom Setup Assistant.appThis really wants to be launched on demand from the per user form of the anti-kernel.
SecurityAgenttrusted per-user program of the Security frameworkCould probably be lazily launched via mach_init. 3190357 will probably help.
SystemUIServeranother misc daemon as far as I can tellMore research is needed.
XquartzX11 serverThis could be launched lazily from the per user anti-kernel.
fetchmailUnix e-mail download agentThis should be launched lazily/managed via the anti-kernel in it's per user form.
gnuservemacs agentCould be launched lazily via the anti-kernel in it's per-user form.
hdi_agenthard disk image agentThis is launched on demand today when somebody requests that a disk image is to be mounted.
iChatAgent.appiChat agentThis should be launched lazily at login. 3190357 should help.
pbsPaste Board serverCould probably be lazily launched via mach_init. 3190357 will probably help.
ssh-agentssh "key chain"Listens on a AF_UNIX domain socket. Could be lazily launched.
screentty mux and session managerListens on a AF_UNIX domain socket. Could be lazily launched.
quartswmQuartz X11 window managerCould be launched via the anti-kernel in it's per-user form. It needs to be relaunched if it dies.

LoginWindow plug-ins

BezelServicesF1-F5 keysThis probably shouldn't live in loginwindow. More research needs to be done.
BootCacheFinal BootCache codeThis tells the BootCache to stop recording disk block access and lets the BootCache begin to clean up. This code really shouldn't live in loginwindow. With the right rules/triggers this could be run from the anti-kernel.
DisplayServicesSimilar to BezelServices. This probably shouldn't live in loginwindow. More research needs to be done. (talk to Chester Devine?)
KextHelperThis probably shouldn't live in loginwindow. More research needs to be done. (talk to nik gervae probably)
MCXThis for managed servers. If so it writes out certain prefercences. This probably shouldn't live in loginwindow. More research needs to be done. (talk to Rob Neville)
SystemSoundServerLike the window server but for simple sound clients. This probably shouldn't live in loginwindow. More research needs to be done.
URLMountUIProxyThis exists only so that it can get registered in the per-user mach sub bootstrap port. This shouldn't live in loginwindow. See 3190356.

configd plug-ins

ATconfigAppleTalk config mgmtAppleTalk config mgmt (appropriate use of configd)
Apple80211MonitorAirPort supportAirPort config mgmt (appropriate use of configd)
BluetoothBluetooth supportstarts blued once a bluetooth dongle is attached and keep it alive forever (inappropriate use of configd)
DynamicPowerStepCPU clock down supportThis will probably move to the kernel in the future from what I hear.
IPConfigurationDHCP and IP config mgmtappropriate use of configd
IPv6ConfigurationDHCPv6 and IPv6 config mgmtappropriate use of configd
IPMonitorcompanion to IPConfigurationmanages default route and resolver configuration, proxies etc. (appropriate use of configd)
InterfaceNamermaps Unix to IOKit devicesThis maps the Unix device names to IOKit devices (appropriate use of configd)
EAPOL802.1x supportAuthenticantion at the Ethernet layer (appropriate use of configd)
KernelEventMonitorWatches NKE event socket(appropriate use of configd)
KernelNotificationsKUNC supportKernel User Notification Center (half should move to kextd, half should move to SystemUIServer)
Kickerconfigd helperHelper for kicking components who are not SystemConfiguration aware.
LinkConfigurationnetwork device configurationManages "layer 2" device configuration (appropriate use of configd)
PPPControllerPPPManages running pppd with the right parameters (appropriate use of configd)
PowerManagementSome Power Management codeThis code could really live in about any daemon.
PreferencesMonitorwatches for pref file changesThis brings networking config files in to effect (appropriate use of configd)
PrinterNotificationsUSB printer supportLike blued (inappropriate use of configd)
USBUPSSupportWatches for USB UPS devicesLike blued (inappropriate use of configd)
$Id: daemon-investigation.html,v 1.1.1.1 2003/11/13 19:18:00 zarzycki Exp $