STREAMS uses a facility known as perimeters for handling thread concurrency in STREAMS modules. Perimeters allow the module writer to select conditions that will result in exclusive access to a queue, a pair of queues, or all the queues in a module. This makes multithreading issues easier to work with in STREAMS.
Perimeters work similar to reader or writer locks, where there can be many readers, but only one writer. A synchronous access to the perimeter is similar to holding the writer lock, in that only one thread can be in the perimeter at a time. Synchronous entry holds a perimeter exclusively until the thread eventually unwinds out of the perimeter, usually when it returns from a put() or putnext() or a qwriter() call that initially invoked the synchronous behavior. While a thread has synchronous access to the perimeter, all other access, synchronous or asynchronous, will be deferred.
An asynchronous access is similar to the reader lock, where many threads can be in the perimeter at a time, including the possible recursive entry of a thread previously entering the perimeter. The asynchronous "claim" is not released until the thread winds out of the put() or putnext(). Because asynchronous access is similar to the reader lock, any synchronous access is deferred or blocked until all asynchronous claims are released.
Caution - Hardening Information. The perimeter data is private to the STREAMS framework, and should not be modified by the module itself. |
STREAMS enables the definition of two synchronous perimeters. One is an inner perimeter, and is used to define synchronous entry points on a queue or a queue pair (the read and write queue's for a specific module instance). It also identifies an outer perimeter, which is made up of all the inner perimeters for all the queues for a specific module.
There is also a special inner perimeter, PERMOD, that is similar to the outer perimeter, but does not have the overhead of the outer perimeter. PERMOD identifies a single synchronous entry point for all queues for this module. Because PERMOD is like a hybrid of an inner perimeter and outer perimeter, the PERMOD perimeter cannot have an outer perimeter.
The module writer does not need to specify an inner perimeter, as the STREAMS framework automatically creates it for the module. The module writer needs to specify the type of perimeter and the concurrency of the perimeter.
Inner perimeters come in two types:
Enables synchronous entry to be different between the read queue and the write queue. Therefore, if a synchronous putnext() is occurring on the read queue, a synchronous or asynchronous putnext() can occur on the write queue or other synchronous or asynchronous access on the write queue.
Protects both the read queue and the write queue, so synchronous access to one queue prevents synchronous or asynchronous access to the other queue.
Another perimeter, D_PERMOD is slightly different. For more information, see PERMOD Perimeter.
An inner perimeter becomes exclusive writer whenever an inner synchronous entry point is encountered. By default all the entry points are considered to be synchronous until enabled as "shared" entry points. Synchronous entry points remain exclusive until the thread returns to the caller of the synchronous entry point. If the synchronous function calls putnext(), the perimeter remains exclusive across the putnext(), up till the synchronous function can return to its caller, and subsequent entries into the perimeter will be deferred.
Inner perimeters can specify additional concurrency on the STREAMS entry points for open or close, put or putnext, service, and callback calls.
The module writer can also specify an outer perimeter. An outer perimeter is the linked list of all inner perimeters for all queues associated with the specified module. Entering the outer perimeter is equivalent to entering each of the inner perimeters. As this can also be an expensive operation, the outer perimeter is only entered synchronously. The outer perimeter becomes exclusive only upon successful completion of a qwriter(PERIM_OUTER). This also has the effect of making each of the inner perimeters exclusive.
Use of outer perimeters is reserved for module data that has an effect on all queue instances of the module, such as module state that might allow messages to pass between other instances of the module, information that allows a driver to configure shared hardware, or at open/close time when information is needed for all open instances for a module.
Outer perimeters, at this time, have only one concurrency modifier. This is D_MTOCEXCL, and instructs the framework to enter the outer perimeter on each open and close of queues for the module.
The PERMOD perimeter is a hybrid of the inner and outer perimeter. It is implemented primarily for modules that might have a large number of queue instances, and cannot afford the latency for entering the outer perimeter. Because it is a hybrid, PERMOD perimeters cannot have an outer perimeter, and modules that have D_MTPERMOD and D_MTOUTPERIM defined fails at open. As PERMOD perimeters are implemented as inner perimeters, they share all the concurrency states as the inner perimeter, see Inner Perimeters.
All STREAMS modules and drivers in the Oracle Solaris operating environment must be D_MTSAFE, and must account for multithreading. Specifying an inner and/or outer perimeter will handle concurrency issues that the module writer may encounter while developing the module or driver. Experienced STREAMS programmers might decide that the perimeter should not have any synchronous entry points, and should run fully hot. To define a fully-hot perimeter, the module writer need only specify the D_MTSAFE flag without an inner perimeter type (D_MTPERQ, D_MTQPAIR, D_MTPERMOD) and without an outer perimeter (D_MTOUTPERIM).
Caution - Hardening Information. All STREAMS entry points run concurrently, and in a multiprocessor environment, there can be a put procedure running simultaneously with a service procedure or even a close procedure. So the writer must take precautions against kernel panics by making sure that other concurrent threads will not reference data the current thread is trying to change or remove. |
To configure a module with perimeter types and concurrency types, use the f_flag field in fmodsw with D_MTSAFE or'd with the appropriate perimeter type flags. See Choosing a Perimeter Type.
The easiest method is to initially implement your module and configure it to be per-module single threaded, and increase the level of concurrency as needed. Sample Multithreaded Device Driver Using a Per Module Inner Perimeter provides a complete example of using a per-module perimeter, and Sample Multithreaded Module With Outer Perimeter provides a complete example with a higher level of concurrency.
To configure a driver with perimeter and concurrency types, put MT_SAFE and the appropriate perimeter flags in the cb_flag field of the cb_ops structure for the driver.
Use the following summaries to determine when to use an inner perimeter, or both an inner and outer perimeter for a STREAMS module.
inner – A module where the put procedure reads as well as modifies module global data. Use a per-module inner perimeter on the single-threaded module.
A module where all the module private data associated with a queue (or a read/write pair of queues) can be configured to be single-threaded. Use an inner perimeter for each corresponding queue or queue pair .
inner and outer – A module where most of the module private data is associated with a queue (or a queue pair); but has some module global data that is mostly read. Use an inner perimeter for the queue or queue pair and an outer perimeter for global data. Use qwriter to protect the sections where it modifies the module's global data. For more information, see theqwriter Function man page.
A module that requires higher concurrency for certain message types while not requiring message ordering. Use an inner perimeter for shared access to the put procedures. Use an outer perimeter for put procedures that require exclusive access. Use qwriter when messages are handled in the put procedures that require exclusive access.
A hardware driver can use an appropriate set of inner and outer perimeters to restrict the concurrency in the open, close, put, and service procedures. With explicit synchronization primitives mutex, condition variables, readers or writer, semaphore, these drivers restrict the concurrency when accessing the hardware registers in interrupt handlers. When designing such drivers, you need to be aware of the issues listed in MT SAFE Modules Using Explicit Locks.
Several flags specify the inner and outer perimeters, as shown in Inner Perimeters and Outer Perimeters. These flags fall into three categories:
Define the presence and scope of the inner perimeter
Define the presence of the outer perimeter, which can have only one scope
Modify the default concurrency for the different entry points
You configure the inner perimeter by choosing one of the following mutually exclusive flags:
D_MTPERMOD – The module has an inner perimeter that encloses all the module's queues
D_MTAPAIR – The module has an inner perimeter around each read/write pair of queues
D_MTPERQ – The module has an inner perimeter around each queue
none – The module has no inner perimeter
Configure the outer perimeter using the following flag:
D_MTOUTEPERIM – In addition to an inner perimeter (or none), the module has an outer perimeter that encloses all the module's queues. This can be combined with all the inner perimeter options except D_MTPERMOD.
By default all synchronous entry points enter the inner perimeter exclusively and enter the outer perimeter shared. To modify this behavior use the following flags:
D_MTOCEXCL – open or close entry points are synchronous
D_MTPUTSHARED – put or putnext entry points are asynchronous
_D_MTOCSHARED – open or close entry points are asynchronous (experimental)
_D_MTCBSHARED – callbacks (by using qtimeout, qbufcall) are asynchronous
_D_MTSVCSHARED – service procedures are asynchronous (experimental)
Caution - Hardening Information. Concurrency flags designated with a preceding underbar "_" are experimental, and their behavior might change in the future and should not be relied upon. |