Preventing Executable Files From Compromising Security

A number of security bugs are related to default executable stacks when their permissions are set to read, write, and execute. While stacks with execute permissions are allowed, most programs can function correctly without using executable stacks.

The noexec_user_stack variable enables you to specify whether stack mappings are executable. The variable is available as of the Solaris 2.6 release. By default, this variable is set to zero, except on 64-bit applications, which provides ABI-compliant behavior. If the variable is set to a non-zero value, the system marks the stack of every process in the system as readable and writable, but not executable.

Once this variable is set, programs that attempt to execute code on their stack are sent a SIGSEGV signal. This signal usually results in the program terminating with a core dump. Such programs also generate a warning message that includes the name of the offending program, the process ID, and the real UID of the user who ran the program. For example:

a.out[347] attempt to execute code on stack by uid 555 

The message is logged by the syslog daemon when the syslog kern facility is set to notice level. This logging is set by default in the syslog.conf file, which means that the message is sent to both the console and the /var/adm/messages file. For more information, see the syslogd(1M) and syslog.conf(4) man pages.

The syslog message is useful for observing potential security problems. The message also identifies valid programs that depend upon executable stacks that have been prevented from correct operation by setting this variable. If you do not want any messages logged, then set the noexec_user_stack_log variable to zero in the /etc/system file. Even though messages are not being logged, the SIGSEGV signal can continue to cause the executing program to terminate with a core dump.

You can use the mprotect() function if you want programs to explicitly mark their stack as executable. For more information, see the mprotect(2) man page.

Because of hardware limitations, the capability of catching and reporting executable stack problems is not available on most x86-based systems. Systems in the AMD64 product family can catch and report executable stack problems.