The fundamental purpose of an operating system (OS) is to enable a variety of programs to share a single computer efficiently and productively. This demands memory protection, preemptively scheduled timesharing, coordinated access to I/O peripherals, and other services. In addition, an OS can allow several users to share a computer. In this case, efficiency demands services that protect users from harming each other, enable them to share without prior arrangement, and mediate access to physical devices.
On today's computer systems, programmers usually implement these goals through a large program called the kernel. This kernel implements all functionality that requires special privileges to either complete it successfully or to allow users being restricted so they can't harm each other. An example of such a functionality is talking to a physical device or managing data stored on the disk. Such kernels are commonly refered to as "monolithic kernels" or "macrokernels".
Since the kernel must be accessible to all user programs, it is the natural place to add functionality to the system. Since the only model for process interaction is that of specific, individual services provided by the kernel, no one creates other places to add functionality. As time goes by, more and more is added to the kernel.
A traditional monolithic kernel system allows users to add components to a kernel only if they both understand most of it and have a privileged status within the system. Testing new components requires a much more painful edit-compile-debug cycle than testing other programs. It cannot be done while others are using the system. Bugs usually cause fatal system crashes, further disrupting others' use of the system. The entire kernel is usually non-pageable. (There are systems with pageable kernels, but deciding what can be paged is difficult and error prone. Usually the mechanisms are complex, making them difficult to use even when adding simple extensions.)
Because of these restrictions, functionality which properly belongs behind the wall of a traditional kernel is usually left out of systems unless it is absolutely mandatory. Many good ideas, best done with an open/read/write interface cannot be implemented because of the problems inherent in the monolithic nature of a traditional system. Further, even among those with the endurance to implement new ideas, only those who are privileged users of their computers can do so. The software copyright system darkens the mire by preventing unlicensed people from even reading the kernel source.
Some systems have tried to address these difficulties by tring to go without a kernel at all. Smalltalk-80 and the Lisp Machine both represented such a method of getting around the problem. System code is not distinguished from user code; all of the system is accessible to the user and can be changed as needed. Both systems were built around languages that facilitated such easy replacement and extension, and were moderately successful.
But they both were fairly poor at insulating users and programs from each other, failing one of the principal goals of OS design. There is no way to fix this flaw as there is no zone protected from the users where the user insulation management code could be placed. This makes the method suitable only for single user devices such as mobile phones or game consoles. Today this method is largerly abandoned.
Microkernel systems try to move as much code from the kernel as possible. The microkernel is only providing access to the physical devices and managing the processes and their communication. Everything other like filesystems is moved into "userspace".
Most microkernel projects like those that use the Mach 3.0 kernel, carry on the hard-to-change tradition of OS design. The internal structure is different, but the same heavy barrier between user and system remains. The single-servers, while fairly easy to construct, inherit all the deficiencies of the monolithic kernels.
A multi-server divides the kernel functionality up into logical blocks with well-defined interfaces. Properly done, it is easier to make changes and add functionality. So most multi-server projects do somewhat better. Much more of the system is pageable. You can debug the system more easily. You can test new system components without interfering with other users. But the wall between user and system remains; no user can cross it without special privilege.
There is one multiserver microkernel operating system, which substantially lowers the barrier between the users and the system. This operating system is called GNU Hurd and the revolutionary idea behind its design is that the area of the system code is kept as little as possible. Programs are required to communicate only with a few essential parts of the kernel; the rest of the system is replaceable dynamically. Users can use whatever parts of the remainder of the system they want, and can easily add components themselves for other users to take advantage of. No mutual trust need exist in advance for users to use each other's services, nor does the system become vulnerable by trusting the services of arbitrary users.
This has been done by identifying those system components which users must use in order to communicate with each other. One of these is responsible for identifying users' identities and is called the authentication server. In order to establish each other's identities, programs must communicate, each with an authentication server they trust. Another component establishes control over system components by the superuser, provides global bookkeeping operations, and is called the process server.
Not all user programs need to communicate with the process server; it is only necessary for programs which require its services. Likewise, the authentication server is only necessary for programs that wish to communicate their identity to another. None of the remaining services carry any special status; not the network implementation, the filesystems, the program execution mechanism (including setuid), or any others.
The OSHS goes in this effort beyond the GNU Hurd. The OSHS nanokernel interface was stripped to mere three kernel calls. The physical device handling is taken out of the kernel if possible (for example on the x86 architecture, where it is possible to allow a process to access a particular physical device without granting system level privileges to the process). Only the functionality that absolutely needs to be in the kernel is left in the kernel. Examples of functionality recognized as not needing a place in the kernel are a sophisticated process scheduler trying to enforce "fair use of the processor" (replaced by the scheduling server; only a very simple round-robin scheduler with realtime priorities and no "fair processor use enforcement", which is much easier to design and debug, is implemented in the kernel) or a virtual memory manager. This refining leads to a much smaller microkernel than is generally achievable using traditional methods of microkernel design, leading to even more stable microkernel because there is much less code to break than even before. Additionally this may lead to a kernel that, once matured, will not need any way to develop further. This is the reason why the OSHS is said to have a nanokernel instead of microkernel.
The three kernel calls of the OSHS kernel are the KCALL__SEND, the KCALL__RECV and the KCALL__CLOSE calls, which are used by the processes to communicate with each other and to close these communication channels. To make the other kernel-level functionality accessible the kernel masquerades as a process and provides a set of handles (called pipes) usable with these two kernel calls which can be used by the processes to request kernel-level functionality such as creating processes and connecting them together with pipes. The result is a system that appearch much more "kernelless" than the traditional microkernel systems. There is a kernel but the processes can't say, where.
Another source of microkernel bloat in traditional microkernel operating systems is the bunch of "must be present on the boot time" device drivers such as disk drivers, which create a chicken-and-egg problem: drivers cannot be loaded from the device without a driver that controls that device but that driver is (usually) stored at that device too. Traditional kernels (both, "macro" and "micro") solve this by placing these drivers into the kernel. By contrast, OSHS nanokernel circumvents this problem by utilizing the 'initrd' idea taken from GNU/Linux: the drivers are packed into a sort of ramdisk image, this ramdisk image is loaded by the bootloader among with the kernel and the kernel is provided with an information about where to search for it. The kernel packs a server program that is able to launch programs from this ramdisk and another initialization program which tells the kernel how to boot.
The net result of these aggresive optimizations of the OSHS nanokernel design is a much more scalable operating system, whose boot-time kernel (along with the boot-time drivers) can be made so tiny that it can be burned directly into the FLASH RAMs of the modern PCs instead of BIOS. Try this with Linux (1)!.