Solaris 7 64-bit Developer's Guide

Chapter 6 Advanced Topics

This chapter presents a collection of miscellaneous system programming topics for systems programmers who want to understand more about the 64-bit Solaris operating environment.

What's New for Applications

Most of the new features of the 64-bit environment are simply extensions of generic 32-bit interfaces, though there are several new features that are unique to SPARC V9.

The SPARC Compliance Definition, Version 2.4 contains details of the SPARC V9 ABI. It describes the 32-bit, SPARC V8 ABI and the 64-bit, SPARC V9 ABI. You can obtain this document from SPARC International at www.sparc.com.

Generic 64-bit ABI Features

64-bit applications are described using Executable and Linking Format (ELF64), which allows large applications and large address spaces to be described completely.

SPARC V9 ABI Features

Below is a list of the SPARC V9 ABI Features.

The SPARC V9 ABI allows all 64-bit SPARC instructions and 64-bit wide registers to be used to their full effect. Many of the new relevant instructions are extensions of the existing V8 instruction set. See The SPARC Architecture Manual, Version 9.
The basic calling convention is the same. The first six arguments of the caller are placed in the out registers %o0-%o5. The SPARC V9 ABI still uses a register window on a larger register file to make calling a function a cheap operation. Results are returned in %o0. Because all registers are now treated as 64-bit quantities, 64-bit values can now be passed in a single register, rather than a register pair.
The layout of the stack is different. Apart from the increase in the basic cell size from 32-bit to 64-bit, various "hidden" parameter words have been removed. The return address is still %o7 + 8.
%o6 is still referred to as the stack pointer register %sp, and %i6 is the frame pointer register %fp. However, the %sp and %fp registers are offset by a constant, known as the stack bias, from the actual memory location of the stack. The size of the stack bias is 2047 bits.
Instruction sizes are still 32 bits. Address constant generation therefore takes more instructions. The call instruction can no longer be used to branch anywhere in the address space, since it can only reach within plus or minus 2Gbytes of %pc.
Integer multiply and divide functions are now implemented completely in hardware.
Structure passing and return are accomplished differently. Small data structures and some floating point arguments are now passed directly in registers.
User traps allow certain traps from non-privileged code to be handled by a user trap handler (instead of delivering a signal).
All data types are now aligned to their size.
Many basic derived types are larger. Thus many system call interface data structures are now different sizes.
Two different sets of libraries exist on the system: those for 32-bit SPARC applications and those for 64-bit SPARC applications.

Address Space Layout

For 64-bit applications, the layout of the address space is closely related to that of 32-bit applications, though the starting address and addressing limits are radically different. Like SPARC V8, the stack grows down from the top of the address space, while the heap extends the data segment from the bottom.

The diagram below shows the default address space provided to a 64-bit application. The regions of the address space marked as reserved might not be mapped by applications. These restrictions might be relaxed on future systems.

The actual addresses above describe a particular implementation on a particular machine, and are given for illustrative purposes only.

Placement of Text and Data

By default, 64-bit programs are linked with a starting address of 0x100000000. The whole program will be above 4 gigabytes, including its text, data, heap, stack, and shared libraries. This helps insure that 64-bit programs are correct by making it so the program will fault in the least significant 4 gigabytes of its address space if it truncates any of its pointers.

While 64-bit programs are linked above 4 gigabytes, it is still possible to link them below 4 gigabytes by using a linker mapfile and the -M option to the compiler or linker. A linker mapfile for linking a 64-bit SPARC program below 4 gigabytes is provided in /usr/lib/ld/sparcv9/map.below4G. See the ld(1) linker man page for more information.

Interprocess Communication

The following interprocess communication (IPC) primitives continue to work between 64-bit and 32-bit processes:

The System V IPC primitives, such as shmop(2), semop(2), msgsnd(2)
mmap(2) on shared files
pipe(2) between processes
door_call(3X) between processes
rpc(3N) between processes on the same or different machines using the external data representation described in xdr(3N)

Although all these primitives allow interprocess communication between 32-bit and 64-bit processes, you might need to take explicit steps to ensure that data being exchanged between processes is correctly interpreted by all of them. For example, two processes sharing data described by a C data structure containing variables of type long cannot do so without understanding that a 32-bit process views this variable as a 4-byte quantity, while a 64-bit process views this variable as an 8-byte quantity.

One way to handle this difference is to ensure that the data has exactly the same size and meaning in both processes. Ensure that the data structures are built using fixed-width types, such as int32_t and int64_t.

A family of derived types that mirrors the system derived types is available in <sys/types32.h>. These types possess the same sign and sizes as the fundamental types of the 32-bit system but are defined in such a way that the sizes are invariant between the ILP32 and LP64 compilation environments.

Sharing pointers between 32-bit and 64-bit processes is substantially more difficult. Obviously, pointer sizes are different, but more importantly, while there is a 64-bit integer quantity (long long) in existing C usage, there is no equivalent to a 64-bit pointer in a 32-bit environment. In order for a 64-bit process to share data with a 32-bit process, you must remember that the 32-bit process can only "see" up to 4 Gbytes of that shared data at a time.

The XDR routine xdr_long(3N) might seem to be a problem; however, it is still handled as a 32-bit quantity over the wire to be compatible with existing protocols. If the 64-bit version of the routine is asked to encode a long value that does not fit into a 32-bit quantity, the encode operation fails.

ELF and System Generation Tools

64-bit binaries are stored in files in ELF64 format, which is a direct analog of the ELF32 format, except that most fields have grown to accommodate full 64-bit applications. ELF64 files can be read using elf(3E) APIs; for example, elf64_getehdr(3E).

Both 32-bit and 64-bit versions of the ELF library, libelf(4), support both ELF32 and ELF64 formats and their corresponding APIs. This allows applications to build, read, or modify both file formats from either a 32-bit or a 64-bit system (though a 64-bit system is still required to execute a 64-bit program).

In addition, Solaris provides a set of GELF (Generic ELF) interfaces that allow the programmer to manipulate both formats using a single, common API. See elf(3E).

All of the system ELF utilities, including ar(1), nm(1), ld(1) and dump(1), have been updated to accept both ELF formats.

`/proc`

The /proc interfaces are available to both 32-bit and 64-bit applications. 32-bit applications can examine and control the state of other 32-bit applications. Thus, an existing 32-bit debugger can be used to debug a 32-bit application.

64-bit applications can examine and control 32-bit or 64-bit applications. However, 32-bit applications are unable to control 64-bit applications, since the 32-bit APIs do not allow the full state of 64-bit processes to be described. Thus, a 64-bit debugger is required to debug a 64-bit application.

`libkvm` and `/dev/ksyms`

The 64-bit version of the Solaris system is implemented using a 64-bit kernel. Applications needing to examine or modify the contents of the kernel directly must be converted to 64-bit applications and linked with the 64-bit version of libkvm(4).

Before doing this conversion and cleanup work, you should examine why the application needs to look directly at kernel data structures in the first place. It is possible that in the time since the program was first ported or created, additional interfaces have been made available on the Solaris platform, to extract the needed data with system calls. See sysinfo(2), kstat(3K), sysconf(3C), and proc(4) as the most common alternative APIs. If these interfaces can be used instead of libkvm(4), use them and leave the application as 32-bit for maximum portability. As a further benefit, most of these APIs are probably faster and might not require the same security privileges needed to access kernel memory.

The 32-bit version of libkvm(4) returns a failure from any attempt to use kvm_open(3K) on a 64-bit kernel or crash dump. Similarly, the 64-bit version of libkvm(4) returns failure from any attempt to use kvm_open(3K) on a 32-bit kernel crash dump.

Because the kernel is a 64-bit program, applications that open /dev/ksyms to examine the kernel symbol table directly need to be enhanced to understand ELF64 format.

The ambiguity over whether the address argument to kvm_read(3K) or kvm_write(3K) is supposed to be a kernel address or a user address is even worse for 64-bit applications and kernel. All applications using libkvm(4) that are still using kvm_read(3K) and kvm_read(3K) should transition to use the appropriate kvm_kread(3K), kvm_kwrite(3K), kvm_uread(3K) and kvm_uwrite(3K) routines. These routines were first made available in Solaris 2.5.

Applications that read /dev/kmem or /dev/mem directly can still run, though any attempt they make to interpret data they read from those devices might be wrong; data structure offsets and sizes are almost certainly different between 32-bit and 64-bit kernels.

`libkstat`

The size of many kernel statistics are completely independent of whether the kernel is a 64-bit or 32-bit program. The data types exported by named kstats (see kstat(3K)) are self-describing, and export signed or unsigned, 32-bit or 64-bit counter data, appropriately tagged. Thus, applications using libkstat need not be made into 64-bit applications to work successfully with the 64-bit kernel.

Note -

If you are modifying a device driver that creates and maintains named kstats, you should try to keep the size of the statistics you export invariant between 32-bit and 64-bit kernels by using the fixed-width statistic types.

Changes to `stdio`

In the 64-bit environment, the stdio facility has been extended to allow more than 256 streams to be open simultaneously. The 32-bit stdio facility continues to have a 256 streams limit.

64-bit applications should not rely on having access to the members of the FILE data structure. Attempts to directly access private implementation-specific structure members can result in compilation errors. Existing 32-bit applications are unaffected by this change, but it is recommended that any direct usage of these structure members be removed from all code.

The FILE structure has a long history, and a few applications have looked inside the structure to glean additional information about the state of the stream. Since the 64-bit version of the structure is now opaque, a new family of routines has been added to both 32-bit libc and 64-bit libc to allow the same state to be examined without depending on implementation internals. See, for example, __fbufsize(3S).

Performance Issues

The following sections discuss advantages and disadvantages of 64-bit performance.

64-bit Application Advantages

Arithmetic and logical operations on 64-bit quantities are more efficient.
Operations use full-register widths, the full-register set, and new instructions.
Parameter passing of 64-bit quantities is more efficient.
Parameter passing of small data structures and floating point quantities is more efficient.

64-bit Application Disadvantages

64-bit applications require more stack space to hold the larger registers.
Applications contain a bigger cache footprint from larger pointers.

System Call Issues

The following sections discuss system call issues.

What does `EOVERFLOW` mean?

The EOVERFLOW return value is returned from a system call whenever one or more fields of the data structure used to pass information out of the kernel is too small to hold the value.

A number of 32-bit system calls now return EOVERFLOW when faced with large objects on the 64-bit kernel. While this was already true when dealing with large files, the fact that daddr_t, dev_t, time_t, and its derivative types struct timeval and timespec_t now contain 64-bit quantities might allow more EOVERFLOW return values to be observed by 32-bit applications.

Beware ioctl(2)

Some ioctl(2) calls have been rather poorly specified in the past. Unfortunately, ioctl(2) is completely insulated from compile-time type checking; therefore it can be a source of bugs that are difficult to track down.

Consider two ioctl(2) calls - one that manipulates a pointer to 32-bit quantity (IOP32), the other that manipulates a pointer to a long quantity (IOPLONG).

The following code sample works as part of a 32-bit application:

Example 6-1

	int a, d;
	long b;
	...
	if (ioctl(d, IOP32, ) == -1)
			return (errno);
	if (ioctl(d, IOPLONG, ) == -1)
			return (errno);

Both ioctl(2) calls work correctly when this code fragment is compiled and run as part of a 32-bit application.

Both ioctl(2) calls also return success when this code fragment is compiled and run as a 64-bit application. However, neither ioctl(2) works correctly. The first ioctl(2) passes a container that is too big, and on a big-endian implementation, the kernel will copy in or copy out from the wrong part of the 64-bit word. Even on a little-endian implementation, the container probably contains stack garbage in the upper 32-bits. The second ioctl(2) will copy in or copy out too much, either reading an incorrect value, or corrupting adjacent variables on the user stack.