STREAMS Programming Guide

Chapter 12 Multithreaded STREAMS

This chapter describes how to write a multithreaded STREAMS driver or module. It covers the necessary conversion topics so that new and existing STREAMS modules and drivers run in the multithreaded kernel. It describes STREAMS-specific multithreading issues and techniques. For general information, see Writing Device Drivers.

This chapter contains the following information:

Multithreaded (MT) STREAMS Overview

The SunOS 5 operating system is fully multithreaded, which means that it can make effective use of the available parallelism of a symmetric shared-memory multiprocessor computer. All kernel subsystems are multithreaded: scheduler, virtual memory, file systems, block/character/STREAMS I/O, networking protocols, and device drivers.

MT STREAMS requires you to use some different concepts and terminology. These concepts apply not only to STREAMS drivers, but to all device drivers in the Solaris operating environment. For a more complete description of these terms, see Writing Device Drivers. Additionally, see Chapter 1, Overview of STREAMS of this guide for definitions and Chapter 8, STREAMS Kernel-Level Mechanisms for elements of MT drivers.

Some of the multithreaded terms and ideas are.

Thread: Sequence of instructions executed within context of a process.
Lock: Mechanism to restrict access to data structures.
Single Threaded: Restricting access to a single thread.
Multithreaded: Allowing two or more threads access to a data element.
Multiprocessing: Two or more CPUs concurrently executing the Operating System.
Concurrency: Simultaneous execution.
Preemption: Suspending execution for the next thread to run.
Monitor: Portion of code that is single-threaded.
Mutual Exclusion: Exclusive access to a data element by a single thread at one time.
Condition Variables: Kernel event synchronization primitives.
Counting Semaphores: Memory-based synchronization mechanism.
Readers/Writer Locks: Data lock allowing one writer or many readers at one time.
Callback: On occurrence of a specific event, call a module function.
Synchronous Access: Only one thread is allowed in the perimeter. Upon return from a call the action is complete; when the thread is done, the job is done.
Asynchronous Access: Multiple threads are allowed in the perimeter. Upon return form a call there is no guarantee that the job is complete.
Perimeter Claim: A thread has synchronous access in the perimeter. The claim prevents subsequent synchronous access until the claim is released.
Exclusive Access: Calling a synchronous entry point in the perimeter.
Writer: A thread that has exclusive access to a perimeter.

MT STREAMS Framework

The STREAMS framework consists of the stream head, documented STREAMS data structures (such as queue_t, mblk_t) and STREAMS utility routines including STREAMS facilities documented in the Device Driver Interface (DDI). The STREAMS framework enables multiple kernel threads to concurrently enter and execute code defined by each module, including the open, close, put, and service procedures of each queue within the system.

The first goal of the SunOS 5 system is to preserve the interface and flavor of STREAMS and to shield module code as much as possible from the impact of migrating to the multithreaded kernel. Most of the locking is hidden from the programmer and performed by the STREAMS kernel framework. As long as module code uses the standard, documented programmatic interfaces to shared kernel data structures (such as queue_t, mblk_t, and dblk_t), and adheres to the DDI/DKI, the user does not have to explicitly lock these framework data structures.

The second goal is to make writing MT SAFE modules simple. One of the ways that the framework accomplishes this is by using the MT STREAMS perimeter mechanisms for controlling and restricting concurrent access to a STREAMS module. STREAMS perimeters allow the module writer to select the level of concurrency that a module can tolerate.

STREAMS Framework Integrity

The STREAMS framework ensures the integrity of the STREAMS data structures, such as queue_t, mblk_t, and dblk_t as long as the module conforms to the DDI/DKI, and does not directly access global operating system data structures or facilities not described in the DDI/DKI.

The q_next fields of the queue_t structure are not modified by the framework while a thread is actively executing within a synchronous entry point. However the q_next field might change while a thread is executing within an asynchronous entry point.

The q_ptr field is considered private to the module and the framework will not manipulate its value. When making a module MT Safe, the integrity of the module-private data structures must be ensured by the module itself. This integrity can be guaranteed by creating private locks, or by the control of concurrency within the module by the use of STREAMS perimeters. Knowing what the framework supports is critical in deciding what the module writer must provide.

Note –

Hardening Information. As in previous Solaris operating environment releases, a module must not call another module's put or service procedures directly. The DDI/DKI routines putnext(9F), put(9F), and other routines in Section 9F must be used to pass a message to another queue. Calling another module's routines directly circumvents the design of the MT STREAMS framework and can yield unknown results.

Note –

Hardening Information. Once a message is passed using a putq, put, putnext, as well as the perimeter function qwriter, it cannot be accessed again because the use of this message has been given to the new routine. If a reference needs to be retained by the module, it should copy it by using copyb, copymsg, dupb, or dupmsg.

Message Ordering

The STREAMS framework guarantees the ordering of messages along a stream if all the modules in the stream preserve message ordering internally. This ordering guarantee only applies to messages that are sent along the same stream and produced by the same source.

The STREAMS framework does not guarantee that a message has been seen by the next put procedure when the call to putnext(9F) or qreply(9F) returns.

MT STREAMS Perimeters

Solaris STREAMS uses a facility known as perimeters for handling thread concurrency in STREAMS modules. Perimeters allow the module writer to select conditions that will result in exclusive access to a queue, a pair of queues, or all the queues in a module. This makes multithreading issues easier to work with in STREAMS.

Perimeters work somewhat like reader/writer locks, where there can be many readers, but only one writer. A synchronous access to the perimeter is similar to holding the writer lock, in that only one thread can be in the perimeter at a time. Synchronous entry will hold a perimeter exclusively until the thread eventually unwinds out of the perimeter (usually when it returns from a put/putnext or a qwriter call that initially invoked the synchronous behavior). While a thread has synchronous access to the perimeter, all other access (synchronous or asynchronous) will be deferred.

An asynchronous access is similar to the reader lock, where many threads can be in the perimeter at a time, including the possible recursive entry of a thread previously entering the perimeter. The asynchronous "claim" is not released until the thread winds out of the put/putnext. Because asynchronous access is similar to the reader lock, any synchronous access will be deferred or blocked until all asynchronous claims are released.

Caution –

Hardening Information. The perimeter data is private to the STREAMS framework, and should not be modified by the module itself.

STREAMS enables the definition of two synchronous perimeters. One is an inner perimeter, and is used to define synchronous entry points on a queue or a queue pair (the read and write queue's for a specific module instance). It also identifies an outer perimeter, which is made up of all the inner perimeters for all the queues for a specific module.

There is also a special inner perimeter, PERMOD, that is similar to the outer perimeter, but does not have the overhead of the outer perimeter. PERMOD identifies a single synchronous entry point for all queues for this module. Because PERMOD is like a hybrid of an inner perimeter and outer perimeter, the PERMOD perimeter cannot have an outer perimeter.

Inner Perimeters

For the most part, the module writer does not need to specify an inner perimeter, as the STREAMS framework automatically creates it for the module. What needs to be specified is the type of perimeter, and the concurrency of the perimeter.

Inner perimeters come in two types:

D_PERQ: Enables synchronous entry to be different between the read queue and the write queue. Therefore, if a synchronous putnext is occurring on the read queue, a synchronous or asynchronous putnext can occur on the write queue (or other (a)synchronous access on the write queue).
D_QPAIR: Protects both the read queue and the write queue, so synchronous access to one queue will prevent synchronous or asynchronous access to the other queue.

Another perimeter, D_PERMOD is slightly different, and is discussed in PERMOD Perimeter.

An inner perimeter becomes exclusive (writer) whenever an inner synchronous entry point is encountered. By default all the entry points are considered to be synchronous until enabled as "shared" entry points. As previously stated, synchronous entry points remain exclusive until the thread returns to the caller of the synchronous entry point. If the synchronous function calls putnext, the perimeter remains exclusive across the putnext, up till the synchronous function can return to its caller, and subsequent entries into the perimeter will be deferred.

Inner perimeters can specify additional concurrency on the STREAMS entry points for open/close, put/putnext, service, and callbacks as shown in Table 12–4.

Outer Perimeters

The module writer can also specify an outer perimeter. An outer perimeter is the linked list of all inner perimeters for all queues associated with the specified module. Entering the outer perimeter is equivalent to entering each of the inner perimeters. As this can also be an expensive operation, the outer perimeter is only entered synchronously, and upon successful completion of a qwriter(PERIM_OUTER) makes the outer perimeter exclusive. This also has the effect of making each of the inner perimeters exclusive.

Use of outer perimeters is reserved for module data that has an effect on all queue instances of the module, such as module state that might allow messages to pass between other instances of the module, information that allows a driver to configure shared hardware, or at open/close time when information is needed for all open instances for a module.

Outer perimeters, at this time, have only one concurrency modifier. This is D_MTOCEXCL, and instructs the framework to enter the outer perimeter on each open and close of queues for the module.

`PERMOD` Perimeter

The PERMOD perimeter is a hybrid of the inner and outer perimeter. It is implemented primarily for modules that might have a large number of queue instances, and cannot afford the latency for entering the outer perimeter. Because it is a hybrid, PERMOD perimeters cannot have an outer perimeter, and modules that have D_MTPERMOD and D_MTOUTPERIM defined will fail at open. As PERMOD perimeters are implemented as inner perimeters, they share all the concurrency states as the inner perimeter, see Table 12–2.

Hot Perimeters

All STREAMS modules and drivers in the Solaris operating environment must be D_MTSAFE, and must account for multithreading. Specifying an inner and/or outer perimeter will handle concurrency issues that the module writer may encounter while developing the module or driver. Experienced STREAMS programmers might decide that the perimeter should not have any synchronous entry points, and should run fully hot. To define a fully-hot perimeter, the module writer need only specify the D_MTSAFE flag without an inner perimeter type (D_MTPERQ, D_MTQPAIR, D_MTPERMOD) and without an outer perimeter (D_MTOUTPERIM).

Caution –

Hardening Information. All STREAMS entry points run concurrently, and in a multiprocessor environment, there can be a put procedure running simultaneously with a service procedure or even a close procedure. So the writer must take precautions against kernel panics by making sure that other concurrent threads will not reference data the current thread is trying to change or remove.

Defining Perimeter Types

To configure a module with perimeter types and concurrency types, use the f_flag field in fmodsw(9s) with D_MTSAFE or'd with the appropriate perimeter type flags. See Table 12–1.

The easiest method is to initially implement your module and configure it to be per-module single threaded, and increase the level of concurrency as needed. Sample Multithreaded Device Driver Using a Per Module Inner Perimeter provides a complete example of using a per-module perimeter, and Sample Multithreaded Module With Outer Perimeter provides a complete example with a higher level of concurrency.

To configure a driver with perimeter and concurrency types, put MT_SAFE and the appropriate perimeter flags in the cb_flag field of the cb_ops structure for the driver.

Choosing a Perimeter Type

Table 12–1 summarizes examples of when to use an inner perimeter, or both an inner and outer perimeter for a STREAMS module.

Table 12–1 Choosing a Perimeter Type


Module Description	Perimeter Type
A module where the `put` procedure reads as well as modifies module global data. Use a per-module inner perimeter on the single-threaded module.	inner
A module where all the module private data associated with a queue (or a read/write pair of queues) can be configured to be single-threaded. Use an inner perimeter for each corresponding queue (or queue pair) .	inner
A module where most of the module private data is associated with a queue (or a queue pair); but has some module global data that is mostly read. Use an inner perimeter for the queue (or queue pair) plus an outer perimeter for global data. Use `qwriter` to protect the sections where it modifies the module's global data (see `qwriter`).	inner and outer
A module that requires higher concurrency for certain message types while not requiring message ordering. Use an inner perimeter for shared access to the `put` procedures. Use an outer perimeter for `put` procedures that require exclusive access. Use `qwriter` when messages are handled in the `put` procedures that require exclusive access (see `qwriter`).	inner and outer
A hardware driver can use an appropriate set of inner and outer perimeters to restrict the concurrency in the `open`, `close`, `put`, and `service` procedures. With explicit synchronization primitives `(mutex`, condition variables, readers/writer, semaphore), these drivers restrict the concurrency when accessing the hardware registers in interrupt handlers. When designing such drivers, you need to be aware of the issues listed in MT SAFE Modules Using Explicit Locks.	inner and outer

Several flags specify the inner and outer perimeters (see Table 12–2 and Table 12–3). These flags fall into three categories:

Define the presence and scope of the inner perimeter
Define the presence of the outer perimeter (which can have only one scope)
Modify the default concurrency for the different entry points

You configure the inner perimeter by choosing one of the mutually exclusive flags shown in Table 12–2.

Table 12–2 Inner Perimeter Flags


Flag	Description
`D_MTPERMOD`	The module has an inner perimeter that encloses all the module's queues
`D_MTAPAIR`	The module has an inner perimeter around each read/write pair of queues
`D_MTPERQ`	The module has an inner perimeter around each queue
none	The module has no inner perimeter

Configure the outer perimeter using the flag shown in Table 12–3.

Table 12–3 Outer Perimeter Flag


Flag	Description
`D_MTOUTEPERIM`	In addition to an inner perimeter (or none), the module has an outer perimeter that encloses all the module's queues. This can be combined with all the inner perimeter options except `D_MTPERMOD`.

By default all synchronous entry points enter the inner perimeter exclusively and enter the outer perimeter shared. To modify this behavior use the flags shown in Table 12–4.

Table 12–4 Modify Exclusive/Shared Access Flags


Flag	Description
`D_MTOCEXCL`	`open`/`close` entry points are synchronous.
`D_MTPUTSHARED`	`put`/`putnext` entry points are asynchronous
`_D_MTOCSHARED`	`open`/`close` entry points are asynchronous (experimental)
`_D_MTCBSHARED`	`callback`s (via `qtimeout`, `qbufcall`) are asynchronous (experimental)
`_D_MTSVCSHARED`	`service` procedures are asynchronous (experimental)

Caution –

Hardening Information. Concurrency flags designated with a preceding underbar "_" are experimental, and their behavior might change in the future and should not be relied upon.

MT SAFE Modules and Drivers

A module or a driver can be either MT SAFE or MT UNSAFE. A module or driver is MT SAFE when its data values are correct regardless of the order that multiple threads access and modify the data. For MT SAFE mode, use MT STREAMS perimeters to restrict the concurrency in a module or driver to:

Per-module single threading
Per queue-pair single threading
Per queue single threading
Per queue or per queue-pair single threading of the put and service procedures with per module single threading of the open and close routines
Unrestricted concurrency in the put and service procedures with the ability to restrict the concurrency when handling messages that modify data
Completely unrestricted concurrency

MT SAFE Module

To configure a module as being MT SAFE, use the f_flag field in fmodsw(9S).

MT SAFE modules can use different MT STREAMS perimeters to restrict the concurrency in the module to a concurrency that is natural given the data structures that the module contains, thereby removing the need for module private locks (see MT STREAMS Perimeters for information on perimeters). A module that requires unrestricted concurrency can be configured to have no perimeters. Such modules have to use explicit locking primitives to protect their data structures. While such modules can exploit the maximum level of concurrency allowed by the underlying hardware platform, they are more complex to develop and support. See MT SAFE Modules Using Explicit Locks.

Independent of the perimeters, there will be at most one thread allowed within any given queue's service procedure.

Your MT SAFE modules should use perimeters and avoid using module private locks (mutex, condition variables, readers/writer, or semaphore). Should you opt to use module private locks, you need to read MT SAFE Modules Using Explicit Locks along with this section.

Note –

MT UNSAFE mode for STREAMS modules was temporarily supported as an aid in porting SVR4 modules; however, MT UNSAFE is not supported after SVR4. Beginning with the release of the Solaris 7 operating environment, no MT UNSAFE module or driver has been supported.

Note –

Upper and lower multiplexors share the same perimeter type and concurrency level.

MT SAFE Driver

To configure a driver as being MT SAFE, initialize the cb_ops(9S) and dev_ops(9S) data structures. This code must be in the header section of your module. For more information, see Example 12–1, and dev_ops(9S).

The driver is configured to be MT SAFE by setting the cb_flag to D_MP. It also specifies any MT STREAMS perimeters by setting flags in the cb_flag field. (See mt-streams(9F).)

Routines Used Inside a Perimeter

This section describes the routines and data fields used after you enter a perimeter.

`qprocson`/`qprocsoff`

The routines qprocson(9F) and qprocsoff(9F) respectively enable and disable the put and service procedures of the queue pair. Before calling qprocson(9F) and after calling qprocsoff(9F), the module's put and service procedures are disabled; messages flow around the module as if it were not present in the stream.

Call qprocson(9F) in the first open of a module, but only after allocating and initializing any module resources on which the put and service procedures depend. Call the qprocsoff routine in the close routine of the module before deallocating any resources on which the put and service procedures depend.

Note –

To avoid deadlocks, modules must not hold private locks across the calls to qprocson(9F) or qprocsoff(9F).

`qtimeout`/`qunbufcall`

The timeout(9F) and bufcall(9F) callbacks are asynchronous. For a module using MT STREAMS perimeters, the timeout(9F) and bufcall(9F) callback functions execute outside the scope of the perimeters. This makes synchronization of callbacks with the rest of the module complex.

To make timeout(9F) and bufcall(9F) functionality easier to use for modules with perimeters, there are additional interfaces that use synchronous callbacks. These routines are qtimeout(9F), quntimeout(9F), qbufcall(9F), and qunbufcall(9F). When using these routines, the callback functions are executed inside the perimeters, and hence have the same concurrency restrictions as the put and service procedures.

`qwriter`

Modules can use the qwriter(9F) function to upgrade from shared to exclusive access at a perimeter. For example, a module with an outer perimeter can use qwriter(9F) in the put procedure to upgrade to exclusive access at the outer perimeter. A module where the put procedure runs with shared access at the inner perimeter (D_MTPUTSHARED) can use qwriter(9F) in the put procedure to upgrade to exclusive access at the inner perimeter.

Returning from a qwriter call does not mean that the callback function has executed. If the framework can become exclusive in the qwriter call, it will enter the perimeter synchronously, and execute the callback. If it cannot, the callback will be deferred. It is a good idea for any caller of qwriter to immediately return to its caller as there is little that can be accomplished in this thread of execution.

Caution –

Hardening Information. Do not call qwriter with another queue, as qwriter assumes that the caller has already made a claim to the perimeter that the queue is associated with (asynchronous entry), and calling another perimeter will cause problems.

Note –

qwriter(9F) cannot be used in the open or close procedures. If a module needs exclusive access at the outer perimeter in the open and/or close procedures, it has to specify that the outer perimeter should always be entered exclusively for open and close (using D_MTOCEXCL).

The STREAMS framework guarantees that all deferred qwriter(9F) callbacks associated with a queue have executed before the module's close routine is called for that queue.

For an example of a driver using qwriter(9F) see Example 12–2.

`qwait`

A module that uses perimeters and must wait in its open or close procedure for a message from another STREAMS module has to wait outside the perimeters; otherwise, the message would never be allowed to enter its put and service procedures. This is accomplished by using the qwait(9F) interface. See qwriter(9F) man page for an example.

Asynchronous Callback Functions

Interrupt handlers and other asynchronous callback functions require special care by the module writer, because they can execute asynchronously to threads executing within the module open, close, put, and service procedures.

For modules using perimeters, use qtimeout(9F) and qbufcall(9F) instead of timeout(9F) and bufcall(9F). The qtimeout and qbufcall callbacks are synchronous and consequently introduce no special synchronization requirements.

Because a thread can enter the module at any time, you must ensure that the asynchronous callback function acquires the proper private locks before accessing private module data structures, and releases these locks before returning. You must cancel any outstanding registered callback routines before the data structures on which the callback routines depend are deallocated and the module closed.

For hardware device interrupts, this involves disabling the device interrupts.
Outstanding callbacks from timeout(9F) and bufcall(9F) must be canceled by calling untimeout(9F) and unbufcall(9F).

The module cannot hold certain private locks across calls to untimeout(9F) or unbufcall(9F). These locks are those that the module's timeout(9F) or bufcall(9F) callback functions acquire. See MT SAFE Modules Using Explicit Locks.
If outstanding callbacks from esballoc(9F) are associated with a particular stream, they must be allowed to complete before the module close routine deallocates the private data structures on which they depend.

`close()` Race Conditions

Because the callback functions are by nature asynchronous, they can be executing or about to execute at the time the module close routine is called. You must cancel all outstanding callback and interrupt conditions before deallocating those data structures or returning from the close routine.

The callback functions scheduled with timeout(9F) and bufcall(9F) are guaranteed to have been canceled by the time untimeout(9F) and unbufcall(9F) return. The same is true for qtimeout(9F) and qbufcall(9F) by the time quntimeout(9F) and qunbufcall(9F) return. You must also take responsibility for other asynchronous routines, including esballoc(9F) callbacks and hardware, as well as software interrupts.

Unloading a Module that Uses `esballoc`

The STREAMS framework prevents a module or driver text from being unloaded while there are open instances of the module or driver. If a module does not cancel all callbacks in the last close routine, it should not be allowed to be unloaded.

This is an issue mainly for modules and drivers using esballoc because esballoc callbacks cannot be canceled. Thus, modules and drivers using esballoc have to be prepared to handle calls to the esballoc callback free function after the last instance of the module or driver has been closed.

Modules and drivers can maintain a semaphore count of outstanding callbacks. They can deny an unload by making the _fini(9E) routine return EBUSY if there are outstanding callbacks.

Use of the `q_next` Field

The q_next field in the queue_t structure can be referenced in open, close, put, and service procedures as well as the synchronous callback procedures (scheduled with qtimeout(9F), qbufcall(9F), and qwriter(9F)). However, the value in the q_next field should not be trusted. It is relevant to the STREAMS framework, but may not be relevant to a specific module.

All other module code, such as interrupt routines, timeout(9F) and esballoc(9F) callback routines, cannot dereference q_next. Those routines have to use the “next” version of all functions. For instance, use canputnext(9F) instead of dereferencing q_next and using canput(9F).

MT SAFE Modules Using Explicit Locks

Although the result is not reliable, you can use explicit locks either instead of perimeters or to augment the concurrency restrictions provided by the perimeters.

Caution –

Explicit locks cannot be used to preserve message ordering in a module because of the risk of re-entering the module. Use MT STREAMS perimeters to preserve message ordering.

All four types of kernel synchronization primitives are available to the module writer: mutexes, readers/writer locks, semaphores, and condition variables. Because cv_wait implies a context switch, it can only be called from the module's open and close procedures, which are executed with valid process context. You must use the synchronization primitives to protect accesses and ensure the integrity of private module data structures.

Constraints When Using Locks

When adding locks in a module, observe these constraints:

Avoid holding module private locks across calls to putnext(9F). The module might be re-entered by the same thread that called putnext(9F), causing the module to try to acquire a lock that it already holds. This can cause kernel panic.
Do not hold module private locks, acquired in put or service procedures, across the calls to qprocson(9F) or qprocsoff(9F). Doing this causes deadlock, since qprocson(9F) and qprocsoff(9F) wait until all threads leave the inner perimeter.
Similarly, do not hold locks, acquired in the timeout(9F) and bufcall(9F) callback procedures, across the calls to untimeout(9F) or unbufcall(9F). Doing this causes deadlock, because untimeout(9F)and unbufcall(9F) wait until an already executing callback has completed.

The first restriction deters using module private locks to preserve message ordering. The preferred mechanism is to use MT STREAMS perimeters to preserve message ordering.

Preserving Message Ordering

Module private locks cannot be used to preserve message ordering because they cannot be held across calls to putnext(9F) and the other messages that pass routines to other modules. The alternatives for preserving message ordering are:

Use MT STREAMS perimeters.
Pass all messages through the service procedures. The service procedure can drop the locks before calling putnext(9F) or qreply(9F), without reordering messages, because the framework guarantees that at most, one thread will execute in the service procedure for a given queue.

Use perimeters to avoid the performance penalty for using service procedures.

Preparing to Port

When modifying a STREAMS driver to take advantage of the multithreaded kernel, a level of MT safety is selected according to:

The desired degree of concurrency
The natural concurrency of the underlying module
The amount of effort or complexity required

Much of the effort in conversion is simply determining the appropriate degree of data sharing and the corresponding granularity of locking (see Table 12–1). The actual time spent configuring perimeters and/or installing locks should be much smaller than the time spent in analysis.

To port your module, you must understand the data structures used within your module, as well as the accesses to those data structures. You must fully understand the relationship between all portions of the module and private data within that module, and to use the MT STREAMS perimeters (or the synchronization primitives available) to maintain the integrity of these private data structures.

You must explicitly restrict access to private module data structures as appropriate to ensure the integrity of these data structures. You must use the MT STREAMS perimeters to restrict the concurrency in the module so that the parts of the module that modify private data are single-threaded with respect to the parts of the module that read the same data. (For more information about perimeters, see MT STREAMS Perimeters.) Besides perimeters, you can use the synchronization primitives available (mutex, condition variables, readers/writer, semaphore) to explicitly restrict access to module private data appropriate for the operations within the module on that data.

The first step in multithreading a module or driver is to analyze the module, breaking the entire module up into a list of individual operations and the private data structures referenced in each operation. Part of this first step is deciding upon a level of concurrency for the module. Ask yourself which of these operations can be multithreaded and which must be single-threaded. Try to find a level of concurrency that is “natural” for the module and matches one of the available perimeters (or, alternatively, requires the minimal number of locks) , and has a simple and straightforward implementation. Avoid additional unnecessary complexity.

Typical questions to ask are:

What data structures are maintained within the module?
What types of accesses are made to each field of these data structures?
When is each data structure accessed destructively (written) and when is it accessed non-destructively (read)?
Which operations within the module should be allowed to execute concurrently?
Is per module single-threading appropriate for the module?
Is per queue-pair or per queue single-threading appropriate?
What are the message ordering requirements?

Porting to the SunOS 5 System

When porting a STREAMS module or driver from the SunOS 4 system to the SunOS 5 system, the module should be examined with respect to the following areas:

The SunOS 5 Device Driver Interface (DDI/DKI)
The SunOS 5 MT design

For portability and correct operation, each module must adhere to the SunOS DDI/DKI. Several facilities available in previous releases of the SunOS system have changed and can take different arguments, or produce different side effects, or no longer exist in the SunOS 5 system. The module writer should carefully review the module with respect to the DDI/DKI.

Each module that accesses underlying Sun-specific features included in the SunOS 5 system should conform to the Device Driver Interface. The SunOS 5 DDI defines the interface used by the device driver to register device hardware interrupts, access device node properties, map device slave memory, and establish and synchronize memory mappings for DVMA (Direct Virtual Memory Access). These areas are primarily applicable to hardware device drivers. Refer to the Device Driver Interface Specification within the Writing Device Drivers for details on the SunOS 5 DDI and DVMA.

The kernel networking subsystem in the SunOS 5 system is based on STREAMS. Datalink drivers that used the ifnet interface in the SunOS 4 system must be converted to use DLPI for the SunOS 5 system. Refer to the Data Link Provider Interface, Revision 2 specification.

After reviewing the module for conformance to the SunOS 5 DKI and DDI specifications, you should be able to consider the impact of multithreading on the module.

Sample Multithreaded Device Driver Using a Per Module Inner Perimeter

Example 12–1 is a sample multithreaded, loadable, STREAMS pseudo-driver. The driver MT design is the simplest possible based on using a per module inner perimeter. Thus, only one thread can execute in the driver at any time. In addition, a quntimeout(9F) synchronous callback routine is used. The driver cancels an outstanding qtimeout(9F) by calling quntimeout(9F) in the close routine. See close() Race Conditions.

Example 12–1 Multithreaded, Loadable, STREAMS Pseudo-Driver

/*
 * Example SunOS 5 multithreaded STREAMS pseudo device driver.
 * Using a D_MTPERMOD inner perimeter.
 */

#include		<sys/types.h>
#include		<sys/errno.h>
#include		<sys/stropts.h>
#include		<sys/stream.h>
#include		<sys/strlog.h>
#include		<sys/cmn_err.h>
#include		<sys/modctl.h>
#include		<sys/kmem.h>
#include		<sys/conf.h>
#include		<sys/ksynch.h>
#include		<sys/stat.h>
#include		<sys/ddi.h>
#include		<sys/sunddi.h>

/*
 * Function prototypes.
 */
static			int xxidentify(dev_info_t *);
static			int xxattach(dev_info_t *, ddi_attach_cmd_t);
static			int xxdetach(dev_info_t *, ddi_detach_cmd_t);
static			int xxgetinfo(dev_info_t *,ddi_info_cmd_t,void *,void**);
static			int xxopen(queue_t *, dev_t *, int, int, cred_t *);
static			int xxclose(queue_t *, int, cred_t *);
static			int xxwput(queue_t *, mblk_t *);
static			int xxwsrv(queue_t *);
static			void xxtick(caddr_t);

/*
 * Streams Declarations
 */
static struct module_info xxm_info = {
   99,            /* mi_idnum */
   "xx",          /* mi_idname */
   0,             /* mi_minpsz */
   INFPSZ,        /* mi_maxpsz */
   0,             /* mi_hiwat */
   0              /* mi_lowat */
};

static struct qinit xxrinit = {
		NULL,           /* qi_putp */
		NULL,           /* qi_srvp */
		xxopen,         /* qi_qopen */
		xxclose,        /* qi_qclose */
		NULL,           /* qi_qadmin */
		&xxm_info,      /* qi_minfo */
		NULL            /* qi_mstat */
};

static struct qinit xxwinit = {
		xxwput,         /* qi_putp */
		xxwsrv,         /* qi_srvp */
		NULL,           /* qi_qopen */
		NULL,           /* qi_qclose */
		NULL,           /* qi_qadmin */
		&xxm_info,      /* qi_minfo */
		NULL            /* qi_mstat */
};

static struct streamtab xxstrtab = {
		&xxrinit,       /* st_rdinit */
		&xxwinit,       /* st_wrinit */
		NULL,           /* st_muxrinit */
		NULL            /* st_muxwrinit */
};

/*
 * define the xx_ops structure.
 */

static 				struct cb_ops cb_xx_ops = {
		nodev,            /* cb_open */
		nodev,            /* cb_close */
		nodev,            /* cb_strategy */
		nodev,            /* cb_print */
		nodev,            /* cb_dump */
		nodev,            /* cb_read */
		nodev,            /* cb_write */
		nodev,            /* cb_ioctl */
		nodev,            /* cb_devmap */
		nodev,            /* cb_mmap */
		nodev,            /* cb_segmap */
		nochpoll,         /* cb_chpoll */
		ddi_prop_op,      /* cb_prop_op */
		&xxstrtab,        /* cb_stream */
		(D_NEW|D_MP|D_MTPERMOD) /* cb_flag */
};

static struct dev_ops xx_ops = {
		DEVO_REV,         /* devo_rev */
		0,                /* devo_refcnt */
		xxgetinfo,        /* devo_getinfo */
		xxidentify,       /* devo_identify */
		nodev,            /* devo_probe */
		xxattach,         /* devo_attach */
		xxdetach,         /* devo_detach */
		nodev,            /* devo_reset */
		&cb_xx_ops,       /* devo_cb_ops */
		(struct bus_ops *)NULL /* devo_bus_ops */
};


/*
 * Module linkage information for the kernel.
 */
static struct modldrv modldrv = {
		&mod_driverops,   /* Type of module. This one is a driver */
		"xx",             /* Driver name */
		&xx_ops,          /* driver ops */
};

static struct modlinkage modlinkage = {
		MODREV_1,
		&modldrv,
		NULL
};

/*
 * Driver private data structure. One is allocated per Stream.
 */
struct xxstr {
		struct		xxstr *xx_next;	/* pointer to next in list */
		queue_t		*xx_rq;				/* read side queue pointer */
		minor_t		xx_minor;			/* minor device # (for clone) */
		int			xx_timeoutid;		/* id returned from timeout() */
};

/*
 * Linked list of opened Stream xxstr structures.
 * No need for locks protecting it since the whole module is
 * single threaded using the D_MTPERMOD perimeter.
 */
static struct xxstr						*xxup = NULL;


/*
 * Module Config entry points
 */

_init(void)
{
	  return (mod_install(&modlinkage));
}

_fini(void)
{
	  return (mod_remove(&modlinkage));
}

_info(struct modinfo *modinfop)
{
	  return (mod_info(&modlinkage, modinfop));
}

/*
 * Auto Configuration entry points
 */

/* Identify device. */
static int
xxidentify(dev_info_t *dip)
{
	  if (strcmp(ddi_get_name(dip), "xx") == 0)
			return (DDI_IDENTIFIED);
	  else
			return (DDI_NOT_IDENTIFIED);
}

/* Attach device. */
static int
xxattach(dev_info_t *dip, ddi_attach_cmd_t cmd)
{
	  /* This creates the device node. */
	  if (ddi_create_minor_node(dip, "xx", S_IFCHR, ddi_get_instance(dip), 
				DDI_PSEUDO, CLONE_DEV) == DDI_FAILURE) {
			return (DDI_FAILURE);
	  }
	  ddi_report_dev(dip);
	  return (DDI_SUCCESS);
}

/* Detach device. */
static int
xxdetach(dev_info_t *dip, ddi_detach_cmd_t cmd)
{
	  ddi_remove_minor_node(dip, NULL);
	  return (DDI_SUCCESS);
}

/* ARGSUSED */
static int
xxgetinfo(dev_info_t *dip, ddi_info_cmd_t infocmd, void *arg,	 
				void **resultp)
{
	  dev_t dev = (dev_t) arg;
	  int instance, ret = DDI_FAILURE;

	  devstate_t *sp;
	  state *statep;
	  instance = getminor(dev);

	  switch (infocmd) {
			case DDI_INFO_DEVT2DEVINFO:
				if ((sp = ddi_get_soft_state(statep, 
						getminor((dev_t) arg))) != NULL) {
					*resultp = sp->devi;
					ret = DDI_SUCCESS;
				} else
					*result = NULL;
				break;

			case DDI_INFO_DEVT2INSTANCE:
				*resultp = (void *)instance;
				ret = DDI_SUCCESS;
				break;

			default:
				break;
	  }
	  return (ret);
}

static
xxopen(rq, devp, flag, sflag, credp)
	  queue_t			*rq;
	  dev_t				*devp;
	  int				flag;
	  int				sflag;
	  cred_t			*credp;
{
	  struct xxstr *xxp;
	  struct xxstr **prevxxp;
	  minor_t 			minordev;

	  /* If this stream already open - we're done. */
	  if (rq->q_ptr)
			return (0);

    /* Determine minor device number. */
	  prevxxp = & xxup;
	  if (sflag == CLONEOPEN) {
			minordev = 0;
			while ((xxp = *prevxxp) != NULL) {
				if (minordev < xxp->xx_minor)
					break;
				minordev++;
				prevxxp = &xxp->xx_next;
			}
			*devp = makedevice(getmajor(*devp), minordev)
	  } else
			minordev = getminor(*devp);

	  /* Allocate our private per-Stream data structure. */
	  if ((xxp = kmem_alloc(sizeof (struct xxstr), KM_SLEEP)) == NULL)
			return (ENOMEM);

	  /* Point q_ptr at it. */
	  rq->q_ptr = WR(rq)->q_ptr = (char *) xxp;

	  /* Initialize it. */
	  xxp->xx_minor = minordev;
	  xxp->xx_timeoutid = 0;
	  xxp->xx_rq = rq;

	  /* Link new entry into the list of active entries. */
	  xxp->xx_next = *prevxxp;
	  *prevxxp = xxp;

	  /* Enable xxput() and xxsrv() procedures on this queue. */
	  qprocson(rq);

	  return (0);
}

static
xxclose(rq, flag, credp)
	  queue_t			*rq;
	  int				flag;
	  cred_t			*credp;

{
	  struct		xxstr		*xxp;
	  struct		xxstr		**prevxxp;

	  /* Disable xxput() and xxsrv() procedures on this queue. */
	  qprocsoff(rq);
	  /* Cancel any pending timeout. */
		 xxp = (struct xxstr *) rq->q_ptr;
		 if (xxp->xx_timeoutid != 0) {
	 		 (void) quntimeout(rq, xxp->xx_timeoutid);
	 		 xxp->xx_timeoutid = 0;
		 }
	  /* Unlink per-stream entry from the active list and free it. */
	  for (prevxxp = &xxup; (xxp = *prevxxp) != NULL; 
				prevxxp = &xxp->xx_next)
			if (xxp == (struct xxstr *) rq->q_ptr)
				break;
	  *prevxxp = xxp->xx_next;
	  kmem_free (xxp, sizeof (struct xxstr));

	  rq->q_ptr = WR(rq)->q_ptr = NULL;

	  return (0);
}

static
xxwput(wq, mp)
	  queue_t		*wq;
	  mblk_t		*mp;
{
	  struct xxstr	*xxp = (struct xxstr *)wq->q_ptr;

		/* write your code here */
     /* *** Sacha's Comments *** broken */
 		freemsg(mp);
		mp = NULL;

		if (mp != NULL)
			putnext(wq, mp);
}

static
xxwsrv(wq)
		queue_t		*wq;
{
		mblk_t		*mp;
		struct xxstr	*xxp;

		xxp = (struct xxstr *) wq->q_ptr;

		while (mp = getq(wq)) {
			/* write your code here */
			freemsg(mp);

			/* for example, start a timeout */
			if (xxp->xx_timeoutid != 0) {
				/* cancel running timeout */
				(void) quntimeout(wq, xxp->xx_timeoutid);
			}
			xxp->xx_timeoutid = qtimeout(wq, xxtick, (char *)xxp, 10);
		}
}

static void
xxtick(arg)
		caddr_t arg;
{
		struct xxstr *xxp = (struct xxstr *)arg;

		xxp->xx_timeoutid = 0;      /* timeout has run */
		/* write your code here */

}

Sample Multithreaded Module With Outer Perimeter

Example 12–2 is a sample multithreaded, loadable STREAMS module. The module MT design is a relatively simple one, based on a per queue-pair inner perimeter plus an outer perimeter. The inner perimeter protects per-instance data structure (accessed through the q_ptr field) and the module global data is protected by the outer perimeter. The outer perimeter is configured so that the open and close routines have exclusive access to the outer perimeter. This is necessary because they both modify the global-linked list of instances. Other routines that modify global data are run as qwriter(9F) callbacks, giving them exclusive access to the whole module.

Example 12–2 Multithread Module with Outer Perimeter

/*
 * Example SunOS 5 multi-threaded STREAMS module.
 * Using a per-queue-pair inner perimeter plus an outer perimeter.
 */

#include		<sys/types.h>
#include		<sys/errno.h>
#include		<sys/stropts.h>
#include		<sys/stream.h>
#include		<sys/strlog.h>
#include		<sys/cmn_err.h>
#include		<sys/kmem.h>
#include		<sys/conf.h>
#include		<sys/ksynch.h>
#include		<sys/modctl.h>
#include		<sys/stat.h>
#include		<sys/ddi.h>
#include		<sys/sunddi.h>

/*
 * Function prototypes.
 */
static			int xxopen(queue_t *, dev_t *, int, int, cred_t *);
static			int xxclose(queue_t *, int, cred_t *);
static			int xxwput(queue_t *, mblk_t *);
static			int xxwsrv(queue_t *);
static			void xxwput_ioctl(queue_t *, mblk_t *);
static			int xxrput(queue_t *, mblk_t *);
static			void xxtick(caddr_t);

/*
 * Streams Declarations
 */
static struct module_info xxm_info = {
   99,					/* mi_idnum */
   “xx”,			/* mi_idname */
   0,					/* mi_minpsz */
   INFPSZ,			/* mi_maxpsz */
   0,					/* mi_hiwat */
   0					/* mi_lowat */
};
/*
 * Define the read-side qinit structure
 */
static struct qinit xxrinit = {
		xxrput,         /* qi_putp */
		NULL,           /* qi_srvp */
		xxopen,         /* qi_qopen */
		xxclose,        /* qi_qclose */
		NULL,           /* qi_qadmin */
		&xxm_info,      /* qi_minfo */
		NULL            /* qi_mstat */
};
/*
 * Define the write-side qinit structure
 */
static struct qinit xxwinit = {
		xxwput,         /* qi_putp */
		xxwsr,          /* qi_srvp */
		NULL,           /* qi_qopen */
		NULL,           /* qi_qclose */
		NULL,           /* qi_qadmin */
		&xxm_info,      /* qi_minfo */
		NULL            /* qi_mstat */
};

static struct streamtab xxstrtab = {
		&xxrini,        /* st_rdinit */
		&xxwini,        /* st_wrinit */
		NULL,           /* st_muxrinit */
		NULL            /* st_muxwrinit */
};

/*
 * define the fmodsw structure.
 */

static struct fmodsw xx_fsw = {
		“xx”,         /* f_name */
		&xxstrtab,      /* f_str */
		(D_NEW|D_MP|D_MTQPAIR|D_MTOUTPERIM|D_MTOCEXCL) /* f_flag */
};

/*
 * Module linkage information for the kernel.
 */
static struct modlstrmod modlstrmod = {
		&mod_strmodops,	/* Type of module; a STREAMS module */
		“xx module”,		/* Module name */
		&xx_fsw,				/* fmodsw */
};

static struct modlinkage modlinkage = {
		MODREV_1,
		&modlstrmod,
		NULL
};

/*
 * Module private data structure. One is allocated per stream.
 */
struct xxstr {
		struct		xxstr *xx_next;	/* pointer to next in list */
		queue_t		*xx_rq;				/* read side queue pointer */
		int			xx_timeoutid;		/* id returned from timeout() */
};

/*
 * Linked list of opened stream xxstr structures and other module
 * global data. Protected by the outer perimeter.
 */
static struct xxstr						*xxup = NULL;
static int some_module_global_data;


/*
 * Module Config entry points
 */
int
_init(void)
{
		return (mod_install(&modlinkage));
}
int
_fini(void)
{
		return (mod_remove(&modlinkage));
}
int
_info(struct modinfo *modinfop)
{
		return (mod_info(&modlinkage, modinfop));
}


static int
xxopen(queue_t *rq,dev_t *devp,int flag,int sflag, cred_t *credp)
{
		struct xxstr *xxp;
		/* If this stream already open - we're done. */
		if (rq->q_ptr)
			return (0);
		/* We must be a module */
		if (sflag != MODOPEN)
			return (EINVAL);

		/*
		 * The perimeter flag D_MTOCEXCL implies that the open and
		 * close routines have exclusive access to the module global
		 * data structures.
		 *
		 * Allocate our private per-stream data structure.
		 */
	 	xxp = kmem_alloc(sizeof (struct xxstr),KM_SLEEP);

		/* Point q_ptr at it. */
		rq->q_ptr = WR(rq)->q_ptr = (char *) xxp;

		/* Initialize it. */
		xxp->xx_rq = rq;
		xxp->xx_timeoutid = 0;

		/* Link new entry into the list of active entries. */
		xxp->xx_next = xxup;
		xxup = xxp;

		/* Enable xxput() and xxsrv() procedures on this queue. */
		qprocson(rq);
		/* Return success */
		return (0);
}

static int
xxclose(queue_t,*rq, int flag,cred_t *credp)
{
		struct			xxstr				*xxp;
		struct			xxstr				**prevxxp;

		/* Disable xxput() and xxsrv() procedures on this queue. */
		qprocsoff(rq);
		/* Cancel any pending timeout. */
	 	xxp = (struct xxstr *) rq->q_ptr;
	 	if (xxp->xx_timeoutid != 0) {
	 		(void) quntimeout(WR(rq), xxp->xx_timeoutid);
	 	 	xxp->xx_timeoutid = 0;
	 	}
		/*
		 * D_MTOCEXCL implies that the open and close routines have
		 * exclusive access to the module global data structures.
		 *
		 * Unlink per-stream entry from the active list and free it.
		 */
		for (prevxxp = &xxup; (xxp = *prevxxp) != NULL; 
				prevxxp = &xxp->xx_next) {
			if (xxp == (struct xxstr *) rq->q_ptr)
				break;
		}
		*prevxxp = xxp->xx_next;
		kmem_free (xxp, sizeof (struct xxstr));
		rq->q_ptr = WR(rq)->q_ptr = NULL;
		return (0);
}

static int
xxrput(queue_t, *wq, mblk_t *mp)
{
		struct xxstr	*xxp = (struct xxstr *)wq->q_ptr;
	
		/*
		 * Write your code here. Can read “some_module_global_data”
		 * since we have shared access at the outer perimeter.
		 */
		putnext(wq, mp);
}

/* qwriter callback function for handling M_IOCTL messages */
static void
xxwput_ioctl(queue_t, *wq, mblk_t *mp)
{
		struct xxstr				*xxp = (struct xxstr *)wq->q_ptr;

		/*
		 * Write your code here. Can modify “some_module_global_data”
		 * since we have exclusive access at the outer perimeter.
		 */
		mp->b_datap->db_type = M_IOCNAK;
		qreply(wq, mp);
}

static
xxwput(queue_t *wq, mblk_t *mp)
{
		struct xxstr				*xxp = (struct xxstr *)wq->q_ptr;

		if (mp->b_datap->db_type == M_IOCTL) {
			/* M_IOCTL will modify the module global data */
			qwriter(wq, mp, xxwput_ioctl, PERIM_OUTER);
			return;
		}
		/*
		 * Write your code here. Can read “some_module_global_data”
		 * since we have exclusive access at the outer perimeter.
		 */
		putnext(wq, mp);
}

static
xxwsrv(queue_t wq)
{
		mblk_t			*mp;
		struct xxstr	*xxp= (struct xxstr *) wq->q_ptr;

		while (mp = getq(wq)) {
		/*
		 * Write your code here. Can read “some_module_global_data”
		 * since we have exclusive access at the outer perimeter.
		 */
			freemsg(mp);

			/* for example, start a timeout */
			if (xxp->xx_timeoutid != 0) {
				/* cancel running timeout */
				(void) quntimeout(wq, xxp->xx_timeoutid);
			}
			xxp->xx_timeoutid = qtimeout(wq, xxtick, (char *)xxp, 10);
		}
}

static void
xxtick(arg)
		caddr_t arg;
{
		struct xxstr *xxp = (struct xxstr *)arg;

		xxp->xx_timeoutid = 0;      /* timeout has run */
		/*
		 * Write your code here. Can read “some_module_global_data”
		 * since we have shared access at the outer perimeter.
		 */
}

Chapter 12 Multithreaded STREAMS

Multithreaded (MT) STREAMS Overview

MT STREAMS Framework

STREAMS Framework Integrity

Message Ordering

MT STREAMS Perimeters

Inner Perimeters

Outer Perimeters

PERMOD Perimeter

Hot Perimeters

Defining Perimeter Types

Choosing a Perimeter Type

MT SAFE Modules and Drivers

MT SAFE Module

MT SAFE Driver

Routines Used Inside a Perimeter

qprocson/qprocsoff

qtimeout/qunbufcall

qwriter

qwait

Asynchronous Callback Functions

close() Race Conditions

Unloading a Module that Uses esballoc

Use of the q_next Field

MT SAFE Modules Using Explicit Locks

Constraints When Using Locks

Preserving Message Ordering

Preparing to Port

Porting to the SunOS 5 System

Sample Multithreaded Device Driver Using a Per Module Inner Perimeter

Example 12–1 Multithreaded, Loadable, STREAMS Pseudo-Driver

Sample Multithreaded Module With Outer Perimeter

Example 12–2 Multithread Module with Outer Perimeter

`PERMOD` Perimeter

`qprocson`/`qprocsoff`

`qtimeout`/`qunbufcall`

`qwriter`

`qwait`

`close()` Race Conditions

Unloading a Module that Uses `esballoc`

Use of the `q_next` Field