This README consists of LWP related parts taken from, RPC2 User Guide and Reference Manual M. Satyanarayanan (Editor) Richard Draves, James Kistler, Anders Klemets, Qi Lu, Lily Mummert, David Nichols, Larry Raper, Gowthami Rajendran, Jonathan Rosenberg, Ellen Siegel So occasional references to RPC2 can be safely ignored. Jan LWP, a coroutine-based lightweight process package. =================================================== RPC2 runs on LWP, a lightweight process package that allows multiple non-preemptive threads of control to coexist within one Unix process. RPC2 and LWP run entirely at user-level on the Unix 4.3BSD interface; no kernel changes are necessary. The first versions of LWP and RPC2 were operational in early 1984 and mid-1985 respectively. The design of LWP predates that of the Cthreads package in Mach. LWP and Cthreads are conceptually similar, but differ substantially in the details. We have successfully emulated all of LWP on top of the preemptive and nonpre- emptive versions of Cthreads, and a subset of the non-preemptive version of Cthreads on top of LWP. Both LWP and RPC2 have evolved over time, resulting in increased functionality and robustness. They have also been ported to a wide variety of machine architectures, such as IBM-RTs, MIPS, Sun2, Sun3, Sparc, and i386, as well as variants of the Unix operating systems such as Mach, SunOS and AIX. Whenever there has been choice between portability and machine-specific performance, we have always favored portability. Credits ------- The original design and implementation of LWP was done by Larry Raper. Its documentation descends from a manual by Jonathan Rosenberg and Larry Raper, later extended by David Nichols and M. Satyanarayanan. Richard Draves, Qi Lu, Anders Klemets and Gowthami Rajendran helped in revising and improving this document. The Basic LWP Package ===================== The LWP package implements primitive functions providing basic facilities that enable procedures written in C, to proceed in an unsynchronized fashion. These separate threads of control may effectively progress in parallel, and more or less independently of each other. This facility is meant to be general purpose with a heavy emphasis on simplicity. Interprocess communication facilities can be built on top of this basic mechanism, and, in fact, many different IPC mechanisms could be implemented. The RPC2 remote procedure call package (also described in this manual) is one such IPC mechanism. The LWP package makes the following key design choices: o The package should be small and fast; o All processes are assumed to be trustworthy -- processes are not protected from each others actions; o There is no time slicing or preemption -- the processor must be yielded explicitly. In order to set up the environment needed by the lightweight process support, a one-time invocation of the LWP_Init function must precede the use of the facilities described here. The initialization function carves an initial process out of the currently executing C procedure. The process id of this initial process is returned as the result of the LWP_Init function. For symmetry a LWP_TerminateProcessSupport function may be used explicitly to release any storage allocated by its initial counterpart. If used, it must be issued from the process created by the LWP_Init function. Upon completion of any of the lightweight process functions, an integer value is returned to indicate whether any error conditions were encountered. Macros, typedefs, and manifest constants for error codes needed by the lightweight process mechanism reside in the file <lwp.h>. A process is identified by an object of type PROCESS, which is defined in the include file. The process model supported by the operations described here is based on a non-preemptive priority dispatching scheme. (A priority is an integer in the range [0..LWP_MAX_PRIORITY], where 0 is the lowest priority.) Once a given lightweight process is selected and dispatched, it remains in control until it voluntarily relinquishes its claim on the CPU. Relinquishment may be either explicit (LWP_DispatchProcess) or implicit (through the use of certain other LWP operations). In general, all LWP operations that may cause a higher priority process to become ready for dispatching, preempt the process requesting the service. When this occurs, the priority dispatching mechanism takes over and dispatches the highest priority process automatically. Services in this category (where the scheduler is guaranteed to be invoked in the absence of errors) are o LWP_CreateProcess o LWP_WaitProcess o LWP_MwaitProcess o LWP_SignalProcess o LWP_DispatchProcess o LWP_DestroyProcess The following services are guaranteed not to cause preemption (and so may be issued with no fear of losing control to another lightweight process): o LWP_Init o LWP_NoYieldSignal o LWP_CurrentProcess o LWP_StackUsed o LWP_NewRock o LWP_GetRock The symbol LWP_NORMAL_PRIORITY provides a good default value to use for process priorities. A Simple Example --------------- #include <lwp.h> static void read_process (int *id) { LWP_DispatchProcess (); /* Just relinquish control for now */ for (;;) { /* Wait until there is something in the queue */ while (empty(q)) LWP_WaitProcess (q); /* Process queue entry */ LWP_DispatchProcess (); } } static void write_process (void) { ... /* Loop & write data to queue */ for (mesg=messages; *mesg!=0; mesg++) { insert (q, *mesg); LWP_SignalProcess (q); } } int main (int argc, char **argv) { PROCESS *id; LWP_Init (LWP_VERSION, 0, &id); /* Now create readers */ for (i=0; i < nreaders; i++) LWP_CreateProcess (read_process, STACK_SIZE, 0, i, "Reader", &readers[i]); LWP_CreateProcess (write_process, STACK_SIZE, 1, 0, "Writer", &writer); /* Wait for processes to terminate */ LWP_WaitProcess (&done); for (i=nreaders-1; i>=0; i--) LWP_DestroyProcess (readers[i]); } The Lock Package ================ The lock package contains a number of routines and macros that allow C programs that utilize the LWP abstraction to place read and write locks on data structures shared by several light-weight processes. Like the LWP package, the lock package was written with simplicity in mind -- there is no protection inherent in the model. In order to use the locking mechanism for an object, an object of type struct Lock must be associated with the object. After being initialized, with a call to Lock_Init, the lock is used in invocations of the macros ObtainReadLock, ObtainWriteLock, ReleaseReadLock and ReleaseWriteLock. The semantics of a lock is such that any number of readers may hold a lock. But only a single writer (and no readers) may hold the clock at any time. The lock package guarantees fairness: each reader and writer will eventually have a chance to obtain a given lock. However, this fairness is only guaranteed if the priorities of the competing processes are identical. Note that no ordering is guaranteed by the package. In addition, it is illegal for a process to request a particular lock more than once, without first releasing it. Failure to obey this restriction may cause deadlock. Key Design Choices ------------------ o The package must be simple and fast: in the case that a lock can be obtained immediately, it should require a minimum of instructions; o All the processes using a lock are trustworthy; o The lock routines ignore priorities; A Simple Example ---------------- #include "lock.h" struct Vnode { ... struct Lock lock; /* Used to lock this vnode */ ... }; #define READ 0 #define WRITE 1 struct Vnode *get_vnode (char *name, int how) { struct Vnode *v; v = lookup (name); if (how == READ) ObtainReadLock (&v->lock); else ObtainWriteLock (&v->lock); } The IOMGR Package ================= The IOMGR package allows light-weight processes to wait on various Unix events. IOMGR_Select allows a light-weight process to wait on the same set of events that the Unix select call waits on. The parameters to these routines are the same. IOMGR_Select puts the caller to sleep until no user processes are active. At this time the IOMGR process, which runs at the lowest priority, wakes up and coaleses all of the select request together. It then performs a single select and wakes up all processes affected by the result. The IOMGR_Signal call allows a light-weight process to wait on delivery of a Unix signal. The IOMGR installs a signal handler to catch all deliveries of the Unix signal. This signal handler posts information about the signal delivery to a global data structure. The next time that the IOMGR process runs, it delivers the signal to any waiting light-weight processes. Key Design Choices ------------------ o The meanings of the parameters to IOMGR_Select, both before and after the call, should be identical to those of the Unix select; o A blocking select should only be done if no other processes are runnable. A Simple Example ---------------- void rpc2_SocketListener () { int ReadfdMask, WritefdMask, ExceptfdMask, rc; struct timeval *tvp; while (TRUE) { . . . ExceptfdMask = ReadfdMask = (1 << rpc2_RequestSocket); WritefdMask = 0; rc = IOMGR_Select (8*sizeof(int), &ReadfdMask, &WritefdMask, &ExceptfdMask, tvp); switch (rc) { case 0: /* timeout */ continue; /* main while loop */ case -1: /* error */ SystemError ("IOMGR_Select"); exit (-1); case 1: /* packet on rpc2_RequestSocket */ . . . process packet . . . break; default: /* should never occur */ } } } The Timer Package ================= The timer package contains a number of routines that assist in manipulating lists of objects of type struct TM_Elem. TM_Elems (timers) are assigned a timeout value by the user and inserted in a package-maintained list. The time remaining to timeout for each timer is kept up to date by the package under user control. There are routines to remove a timer from its list, to return an expired timer from a list and to return the next timer to expire. This specialized package is currently used by the IOMGR package and by the implementation of RPC2. A timer is used commonly by inserting a field of type struct TM_Elem into a structure. After inserting the desired timeout value the structure is inserted into a list, by means of its timer field. A Simple Example ---------------- static struct TM_Elem *requests; ... TM_Init (&requests); /* Initialize timer list */ ... for (;;) { TM_Rescan (requests); /* Update the timers */ expired = TM_GetExpired (requests); if (expired == 0) break; ... process expired element ... } The Preemption Package ====================== The preemption package provides a mechanism by which control can pass between light-weight processes without the need for explicit calls to LWP_DispatchProcess. This effect is achieved by periodically interrupting the normal flow of control to check if other (higher priority) procesess are ready to run. The package makes use of the interval timer facilities provided by 4.2BSD, and so will cause programs that make their own use of these facilities to malfunction. In particular, use of alarm (3) or explicit handling of SIGALRM is disallowed. Also, calls to sleep (3) may return prematurely. Care should be taken that routines are re-entrant where necessary. In particular, note that stdio (3) is not re-entrant in general, and hence light-weight processes performing I/O on the same FILE structure may function incorrectly. Key Design Choices ------------------ o The package should not affect the nonpreemptive scheduling behaviour of processes which do not use it; o It must be simple and fast, with a minimum of extra system overhead; o It must support nested critical regions; o Processes using the package are assumed to be co-operating. A Simple Example ---------------- #include <sys/time.h> #include "preempt.h" ... struct timeval tv; LWP_Init (LWP_VERSION, ... ); tv.tv_sec = 10; tv.tv_usec = 0; PRE_InitPreempt (&tv); PRE_PreemptMe (); ... PRE_BeginCritical (); ... PRE_EndCritical (); ... PRE_EndPreempt (); The Fast Time Package ===================== The Fast Time package allows the caller to find out the current time of day without incurring the expense of a kernel call. It works by mapping the page of the kernel that has the kernels time-of-day variable and examining it directly. Currently, this package only works on Suns. You may call the routines on other machines, but they will run more slowly. The initialization routine for this package is fairly expensive since it does a lookup of a kernel symbol via nlist (). If you have a program which runs for only a short time, you may wish to call FT_Init with the notReally parameter true to prevent the lookup from taking place. This is useful if you are using another package that uses Fast Time (such as RPC2). Some design considerations for LWP programming ============================================== Clients and servers are each assumed to be Unix processes using the LWP package. RPC2 will not work independently of the LWP package. The LWP package makes it possible for a single Unix process to contain multiple threads of control (LWPs). An RPC call is synchronous with respect to an individual LWP, but it does not block the encapsulating Unix process. Although LWPs are non-preemptive, RPC2 internally yields control when an LWP is blocked awaiting a request or reply. Thus more than one LWP can be concurrently making RPC requests. There is no a priori binding of RPC connections to LWPs, or LWPs to subsystems within a client or server. Thus RPC connections, LWPs and subsystems are completely orthogonal concepts. Since LWPs are non-preemptive, a long-running computation by an LWP will prevent the RPC2 code from getting a chance to run. This will typically manifest itself as a timeout by the client on the RPC in progress. To avoid this, the long running computation should periodically invoke IOMGR_Poll followed by LWP_DispatchProcess. For similar reasons, Unix system calls such as read (2), sleep (3), and select (2) that may block for a long time should be avoided. Instead, use IOMGR_Select.