This section explains the logging features of the kernel memory allocator and how you can employ them to debug system crashes.
As explained earlier, the second half of each buftag contains extra information about the corresponding buffer. Some of this data is debugging information, and some is data private to the allocator. While this auxiliary data can take several different forms, it is collectively known as "Buffer Control" or bufctl data.
However, the allocator needs to know whether a buffer's bufctl pointer is valid, since this pointer might also have been corrupted by malfunctioning code. The allocator confirms the integrity of its auxiliary pointer by storing the pointer and an encoded version of that pointer, and then cross-checking the two versions.
As shown in Extra Debugging Data in the Buftag, these pointers are the bcp (buffer control pointer) and bxstat (buffer control XOR status). The allocator arranges bcp and bxstat so that the expression bcp XOR bxstat equals a well-known value.
Figure 7 Extra Debugging Data in the Buftag
In the event that one or both of these pointers becomes corrupted, the allocator can easily detect such corruption and panic the system. When a buffer is allocated, bcp XOR bxstat = 0xa110c8ed ("allocated"). When a buffer is free, bcp XOR bxstat = 0xf4eef4ee ("freefree").
In the event that the allocator finds a corrupt buftag, it panics the system and produces a message similar to the following:
kernel memory allocator: boundary tag corrupted bcp ^ bxstat = 0xffeef4ee, should be f4eef4ee
Remember, if bcp is corrupt, it is still possible to retrieve its value by taking the value of bxstat XOR 0xf4eef4ee or bxstat XOR 0xa110c8ed, depending on whether the buffer is allocated or free.
The buffer control (bufctl) pointer contained in the buftag region can have different meanings, depending on the cache's kmem_flags. The behavior toggled by the KMF_AUDIT flag is of particular interest: when the KMF_AUDIT flag is not set, the kernel memory allocator allocates a kmem_bufctl_t structure for each buffer. This structure contains some minimal accounting information about each buffer. When the KMF_AUDIT flag is set, the allocator instead allocates a kmem_bufctl_audit_t, an extended version of the kmem_bufctl_t.
This section presumes the KMF_AUDIT flag is set. For caches that do not have this bit set, the amount of available debugging information is reduced.
The kmem_bufctl_audit_t (bufctl_audit for short) contains additional information about the last transaction that occurred on this buffer. The following example shows how to apply the bufctl_audit macro to examine an audit record. The buffer shown is the example buffer used in Detecting Memory Corruption:
> 0x70a9ae00,5/KKn 0x70a9ae00: 5 4ef83 0 0 1 bbddcafe feedface 139d 70ae3200 d1befaed
Using the techniques presented above, it is easy to see that 0x70ae3200 points to the bufctl_audit record: it is the first pointer following the redzone. To examine the bufctl_audit record it points to, apply the bufctl_audit macro:
> 0x70ae3200$<bufctl_audit 0x70ae3200: next addr slab 70378000 70a9ae00 707c86a0 0x70ae320c: cache timestamp thread 70039928 e1bd0e26afe 70aac4e0 0x70ae321c: lastlog contents stackdepth 7011c7c0 7018a0b0 4 0x70ae3228: kmem_zalloc+0x30 pid_assign+8 getproc+0x68 cfork+0x60
The 'addr' field is the address of the buffer corresponding to this bufctl_audit record. This is the original address: 0x70a9ae00. The 'cache' field points at the kmem_cache that allocated this buffer. You can use the ::kmem_cache dcmd to examine it as follows:
> 0x70039928::kmem_cache ADDR NAME FLAG CFLAG BUFSIZE BUFTOTL 70039928 kmem_alloc_24 020f 000000 24 612
The 'timestamp' field represents the time this transaction occurred. This time is expressed in the same manner as gethrtime(3C).
'thread' is a pointer to the thread that performed the last transaction on this buffer. The 'lastlog' and 'contents' pointers point to locations in the allocator's transaction logs. These logs are discussed in detail in Allocator Logging Facility.
Typically, the most useful piece of information provided by bufctl_audit is the stack trace recorded at the point at which the transaction took place. In this case, the transaction was an allocation called as part of executing fork(2).